[HN Gopher] Leaving LinkedIn
___________________________________________________________________
Leaving LinkedIn
Author : wyclif
Score : 433 points
Date : 2024-03-06 04:49 UTC (18 hours ago)
(HTM) web link (corecursive.com)
(TXT) w3m dump (corecursive.com)
| willcipriano wrote:
| > My previous company had what I thought at the time was a pretty
| decent sized app, like 150,000 lines of code. And the LinkedIn
| front end, when I got there had 2 million, it's just like, I'm,
| did you say million? Uh, that's, that's a lot.
|
| I'd love to see what makes it so large. That's like 1/15th the
| size of Linux.
| apimade wrote:
| Including boilerplate code? ASP and JS is far more verbose in
| those areas.
| PreachSoup wrote:
| Think js+html+css, that's a lot
| dilyevsky wrote:
| Have you ever worked on projects (particularly java suffers
| from this) where there is generational bloat because most
| people have very narrow understanding of the codebase so they
| can only add things and not remove/refactor without breaking
| it? That's how
| CapsAdmin wrote:
| Usually if you peek at the first few commits it all seemed
| nice and clean. The turning point is usually as soon as the
| original author/'s leave and new people come in.
| Cthulhu_ wrote:
| That's not likely to be the issue here tbh, or, that's
| focusing on low-hanging fruit. You get large codebases from
| having many features. No amount of coding and refactoring
| will bring a codebase's size down in a significant and
| measurable way unless features are ruthlessly removed.
| dilyevsky wrote:
| I mean sure, I could understand Figma's frontend or
| something having millions of lines but what does linkedin
| frontend do that would reasonably require this?
| crooked-v wrote:
| My guess is that it's because there are 800 different
| versions of everything because none of the company's
| 13,000 UI/UX designers are willing to use the same
| component libraries.
| superhuzza wrote:
| Blaming 2,000,000 lines of code on UX designers, and not
| the developers, is certainly a new one.
| bluetomcat wrote:
| These high-profile apps are elaborate spying devices.
| Besides the UI code, they probably have thousands of
| classes, some of which abandoned as dead code, with names
| like ProfileViewActivityLocationTracker...
| realusername wrote:
| Orgs never value engineers that improve the overall codebase
| so I can't blame them. You're only getting promoted and
| compensated by delivering the next big feature which is part
| of the owners vision and being visible.
| The_Colonel wrote:
| > particularly java suffers from this
|
| Correlation, not causation. Java is often used on large
| projects, it doesn't itself make things brittle.
|
| It's like the bomber survivorship fallacy - the fact you see
| such messed up projects mostly in Java is that they would not
| survive in languages like Python, where even a simple method
| rename can easily lead to big regressions.
| tayo42 wrote:
| That quote is why you can't take anything seriously on the
| internet about running software or software engineering
| seriously.
|
| There are to many people that just don't know what they don't
| know and act they have things solved.
|
| I think the software world has like two bubbles of people
| around the scale they work with and they don't understand each
| other or run into each other, except in interviews and on the
| internet.
| esprehn wrote:
| It's funny you say that because as someone in the large scale
| bubble when I read 2 million lines that didn't seem that
| large. Other companies of similar size are dealing with 2-10x
| larger web apps.
|
| Ex. https://stripe.com/blog/migrating-to-typescript
|
| Which talks about 3.7m lines at Stripe.
| chriskrycho wrote:
| Yeah, and I have it on good authority that there are JS/TS
| apps more than 10 times larger than that at Google. Scale
| is relative.
| diarrhea wrote:
| The site makes me sad whenever I visit.
|
| On my beefy desktop, load times for the home page are at around
| three seconds. On my feeble (but still enormously expensive: a
| Microsoft Surface) laptop, it's well in excess of 10 seconds.
| Both tests run on the very same network, so my conclusion is
| that load times are CPU bound, aka there's huge amounts of
| JavaScript running, for one reason or another.
| bobterryo wrote:
| I was really impressed with how open Chris Krycho was about his
| struggles without playing the blame game. CoRecursive is one of
| my favorite podcasts because it explores the complex context
| behind the code.
| Lyngbakr wrote:
| Agreed. And I also think Adam is a great host. He asks good
| questions and lets the guest talk.
| yellow_lead wrote:
| He seems like the kind of guy I'd like to work with,
| personally.
| LightFog wrote:
| Oh boy, been there. This is Conway's law in action - they are
| about to go and create the same code soup again - because the org
| won't have changed. Having been in the same boat my lesson is
| positive engineering initiatives can only come top down - through
| some very senior champion. You can't change an organisation from
| the bottom up - and it's the organisation that produces the code
| base.
| zer00eyz wrote:
| >> This is Conway's law in action
|
| Not just one layer, look at the stack: Type script is MS's
| Conways law. API/microsevice everything is more of a google
| thing (vs FB monolith)... So you have two other major product
| of Conways Law that your trying to jam into your own org.
|
| On top of that a 5 year plan is never going to digestible by
| (almost) any company. How many people stay in one place for
| that long? How much has the front end changed in 5 years? Is
| any choice you make in the JS world relevant in 5 years? Though
| sell there too!
| antupis wrote:
| I think it is more nuanced, you can also change the
| organization bottom up, but it needs to be new stuff that does
| not exist or has it infancy. Changing existing stuff is very
| hard even top-down.
| LightFog wrote:
| Indeed, and greenfield projects are a great way to move
| upward in the org - as Jim in the story recognises.
| chii wrote:
| > new stuff that does not exist or has it infancy.
|
| If it was new stuff, then the people doing it must not be
| very low at the bottom (by virtue of the org being quite
| shallow - such as a startup).
|
| Even new stuff in a big company with lots of layers of
| management cannot make new stuff with excellence. Eventually,
| probably sooner rather than later, it gets polluted.
|
| Not to mention that in a big org, the people at the bottom do
| not get rewarded for making excellence in engineering, if the
| top doesn't appreciate it. SO why go that extra mile, when a
| mediocre job will earn you the same salary? You'd be more
| incentivized to move jobs to grow your pay instead, and use
| this as a stepping stone.
| klyrs wrote:
| As somebody who has only ever worked at places small enough
| for individuals to all roughly understand eachother's
| impact, could I even get a job at a big org like that? I
| don't even understand the mindset
| nercury wrote:
| Yes of course. You have to demonstrate that you are very
| skilled at building with the particular bricks they use.
| Don't mention anything else to show your commitment to
| particular brick usage.
| klyrs wrote:
| Yikes! I didn't want to understand the mindset after all!
| Thanks :)
| hef19898 wrote:
| Get one? Sure. Be happy with it? Propably not.
|
| I learned that the hard way, only the other way around.
| klyrs wrote:
| Sometimes, I daydream about being a nameless drone in a
| cubefarm, taking home a nice paycheck and leaving my work
| at work. But, yeah, the dream passes quickly because I
| know I'd hate it with a burning passion.
| devjab wrote:
| Have you changed an organisation from the bottom up? In two
| decades I've never seen it happen in organisations larger
| than 50 people.
|
| You can change something, like how an engineering team works
| and how an organisation does DevOps and other things that
| management doesn't really know anything about and trust their
| employees on. But moving an organisation into something like
| team topologies which is a more modern expansion on Conway,
| is virtually impossible from the bottom up in my experience.
| Change management in general is very hard both directions,
| but it's much harder going upwards because going upwards
| means you don't really have the support of management above
| you. Maybe they'll humour you but to make an actual impact
| you're going to need them onboard. You'll even have to reach
| pretty far up top if you want to inflict lasting culture
| changes as things carried by a single (or few) managers often
| dies with them.
|
| My career has sort of put me in a role where I help non-tech
| startups transition from that to medium or enterprise size. I
| solely focus on the tech journey though, as I know I won't
| have too much impact on culture or work processes on the
| grander scale. Often this leads to a semi-team topologies
| approach as the tech teams divide systems into independent
| business related services, but as a whole I don't expect any
| changes to reach outside of the development departments.
| _puk wrote:
| I'm not disagreeing on your overall point, but it can be
| done (I've done it with 130+, so there's a lot of incumbent
| process and numerous senior managers to align).
|
| The key is having a champion who can tie change in with
| higher level desires.
|
| In these orgs there's often an external (as in stakeholders
| external to prod eng) perception that "engineering is slow,
| product doesn't deliver, they're preventing us delivering
| our OKRs", and that can be used as leverage for change.
|
| You can't go dark and stop delivery, but you can usually
| carve out a 10 - 20% allowance for change (that doesn't
| sound much, but is 1 - 2 days every 2 weeks, which quickly
| adds up). Start small, show success in impacting metrics
| that external stakeholders care about, then next quarter
| push for more.
|
| I've focused myself in a similar area as you, but actually
| lean into processes and people, whilst still guiding tech -
| maybe we should chat!
| krig wrote:
| If you want to dig into Conway's law and its implications, I
| can't recommend this video essay by Casey Muratori enough:
| https://youtu.be/5IUj1EZwpJY?si=dPxsXieBwZsP0PPP
|
| Fair warning though, you'll lose all hope that companies like
| Microsoft will ever manage to produce anything that they don't
| then ruin.
| Xelbair wrote:
| >Fair warning though, you'll lose all hope that companies
| like Microsoft will ever manage to produce anything that they
| don't then ruin.
|
| you are implying that we still had any hope left in such
| endeavors.
| jiggawatts wrote:
| Oh. My. God.
|
| This video explains _everything_ I 've seen in the enterprise
| IT of a huge government org that was the result of a long
| series of back-to-back mergers and splits.
|
| It's _exactly_ this!
|
| Conway's law, but _over time_.
|
| Dysfunction not _just_ because of the large hierarchy, but
| also because of the built up layers of history. Some elements
| still limping along, some essentially amputated and mere
| phantom limbs of the org tree now, some outright dead and
| spreading miasma and corruption from their decay.
|
| I knew or suspected most of this, but to hear it said out
| loud so clearly has crystallised my understanding of it.
| krig wrote:
| Yeah once it hits, it really explains so much of the issues
| in large organizations (and explains a lot about small
| organizations / indie development as well). For example,
| why is it that mergers and buyouts rarely work out?
| Conway's law largely explains it! When a product is bought
| by an organization, the result is an integration over time
| of both organization org-charts combined. If the companies
| are both large, this immediately becomes an intractable
| mess. If the original company was small and the new company
| is large, it _also_ becomes intractable because now
| different teams will have to communicate to manage or split
| components that don't fit the communication patterns of the
| new organization, and will lead to large refactors and
| rewrites and in the end the essential qualities of the
| original software will inevitably change.
|
| Even if you know this, there is nothing you can do - you
| have to organize yourself somehow and how you do that can't
| be left up to just mirror whatever this piece of software
| that was brought in happens to be organized - because _that
| too_ doesn't reflect an actual org chart but is an
| integration over time of all org charts that have touched
| it.
|
| I don't even know how to think about the implications this
| has for the use of open source software and how open source
| is developed, like... I think there are huge opportunities
| for research into Conway's law and how it relates to
| software development practices, software quality, etc.
| jiggawatts wrote:
| I have this naively optimistic hope that AIs will allow
| orgs to scale units past Dunbar's number.
|
| We humans can't effectively communicate with groups
| larger than about 7 people, but AIs have no such limits.
| We could all simply talk to _one_ manager that can
| integrate everything and everyone into a unified whole.
|
| It's like the ship Minds in the Culture series having
| hundreds or even thousands of conversations at once.
| krig wrote:
| That's an interesting thought, yeah... technology and
| communication are definitely interwoven, things are
| possible today that were impossible before computers
| simply due to communication constraints. One could
| imagine a large number of practically autonomous small
| teams organized "automatically", more mirroring a neural
| network than a hierarchy.
| pixelfarmer wrote:
| The problem is always communication because it is the
| means to cooperate. The root of many issues in software
| development is the simple fact that instead of letting
| the required communication pathways define the
| organization, it is the organization which defines the
| pathways and through that causes communication
| obstructions.
|
| "Not Just Bikes" has a good few videos, including
| https://www.youtube.com/watch?v=n94-_yE4IeU and a couple
| more that talk about problems that larger roads
| effectively cause more traffic ("induced demand",
| "traffic generation"). Organizational structures are like
| roads, and like roads they can get overloaded, which in
| turn means traffic needs to be reduced. There is even
| communication jam, and to combat that something like
| enforced communication reduction (lower information
| throughput), etc. to keep this manageable. That also
| causes decision making being done with less and less
| information the more steps are included in a
| communication chain (like upwards/downwards in a
| hierarchy), which in turn means the quality of decision
| making is severely hampered by it.
|
| This whole mess is also the reason why the agile
| manifesto puts humans before processes and other such
| things, in fact it implies you change even the
| organizational setup to fit the project, not the other
| way around. But in the space of "managerial feudalism"
| (David Graeber) this is pretty much impossible to pull
| off.
| krig wrote:
| The tragedy of agile is that the practices that are
| labelled agile in practice tend to exemplify the exact
| opposite of everything that the manifesto advocates..
| Nathanba wrote:
| interestingly positive, I think I might agree with you.
| It seems the nr.1 problem of good leaders in
| organizations is that they can't be everywhere at once.
| But an AI can be.
| dsr_ wrote:
| You might be correct, but the AI minds that you are
| contemplating don't exist yet, and there is no reason to
| think that they will be developed from current LLMs.
|
| Once again, seizing the term AI to mean LLMs and other
| current generative techniques has poisoned clear
| thinking. When we think "AI", we are thinking about HAL
| and Cortana and C3PO, which is not what we actually have.
| navane wrote:
| The more the initial fade of AI assisted work sets in,
| and given the inherent vagueness and unpredictability of
| managing, I'm eagerly awaiting not my job, but my bosses
| job being replaced by AI. There's no need for exactness,
| but superficial clarity, decisiveness and seeming
| coherence.
| ConcernedFish wrote:
| I've watched a lot of Casey Muratori's other presentations
| but just happened to find this one last week and I
| wholeheartedly agree. Like many people I'd heard of
| Conway's Law but always imagined it as a pithy truism and
| hadn't thought that the effects could run so deep.
|
| Casey's example is focused on a (frankly, ridiculously)
| large organisation but I've seen the same thing in numerous
| small companies with just 20-30 developers, and it's hard
| not to imagine that this is universal, which is a pretty
| depressing thought.
|
| Recently I've been involved in a new project where teams
| were set up to be 'domain-specific' with the idea that this
| somehow avoided the issues of Conway's Law, but this just
| feels like exactly the same problem because team structures
| and the software that they end up producing is siloed in
| exactly the same way as the organisation structures that
| existed at the time.
|
| Casey's point that the final design is inherently limited
| by the lack of high-bandwidth communication between the
| groups of people that need it most is also fairly
| demotivating, personally.
|
| Tangentially, having been writing more Golang recently this
| made me think of Golang's mantra (or was it Rob Pike's
| quote) of "Don't communicate by sharing memory; share
| memory by communicating.". Go/CSPs concurrency and sharing
| model is interesting, but I wonder if it's a fair analogue
| to compare sharing memory by communicating as low-bandwidth
| communication, and communication by sharing memory as the
| high-bandwidth communication that seems to be needed for
| effective designs.
| paulsutter wrote:
| Microsoft has moved very quickly to add LLM functionality to
| multiple products
|
| Literally the opposite of LinkedIn. I'm approximately user
| 5000 on linkedin, and since I joined almost nothing useful
| has been added, despite a huge increase in complexity
| krig wrote:
| LinkedIn and Microsoft are the same, that said I personally
| think the rush to push LLMs into everything is entirely
| consistent with my prediction and with the internal
| structure of Microsoft (current and historical).
| baobabKoodaa wrote:
| Come on now, that's not true. Every week LinkedIn adds a
| new email marketing category which you haven't opted-out of
| yet, because it didn't exist before.
| badpun wrote:
| Exactly. Remember that we are not customers, we are the
| product. Presumably, LinkedIn is adding a lot of useful
| features for their actual customers, they're just not
| visible to us.
| Temporary_31337 wrote:
| What's broken with current office 365? It's a good product
| and service.
| AdrianB1 wrote:
| This is what you see from the outside. LinkedIn also looks
| good from outside. This is the entire point of the story.
| rajamaka wrote:
| If the user experience is good, then the product is good.
| Users don't care about build time or internal dependency
| management.
| __s wrote:
| Build time & internal dependency management are just a
| few markers used to predict where user experience will go
| in the future
| throwaway14356 wrote:
| Making something look good is very popular but entirely
| different from actual quality.
| CoastalCoder wrote:
| > LinkedIn also looks good from outside.
|
| I partially disagree. IIRC their search is broken, and
| their job alerts are pretty lousy at de-duplicating
| advertised positions.
|
| And IIRC they hijack the back button, which I find pretty
| offensive.
| speed_spread wrote:
| The existence of the product is broken. Office suites
| should not be web applications.
| chasd00 wrote:
| To be fair, it's only a web application if you want it to
| be. I have all the usual office products installed
| locally. When I get a link that opens a document in the
| browser I click open in app and off it goes.
| Xcelerate wrote:
| Does Conway's law imply that an organization of size 1
| produces a system with components that are optimally
| integrated?
|
| I sometimes wonder what would happen if a company hired 100
| software engineers, gave them all the same task to program
| _the entire thing_ independently on their own, and then paid
| them as a monotonic function of the ranking of their output
| against their peers. (Obviously, there are limits on what
| type of product could be developed here, but I think this
| still encompasses far more company products than you would
| suspect.)
| krig wrote:
| An organization of size 1 has other constraints, like
| time.. But also, Conway's law applies outside the
| organization as well, communication constraints shape
| society at large and the way a society is organized limits
| the communication channels available.
|
| On the topic of Microsoft/Linkedin, aren't they the ones
| who used to or maybe still assign the same task to multiple
| teams and let the teams compete for who can deliver? That
| does sound vaguely like what you propose.
| throwaway14356 wrote:
| Run all 100 in parallel in production and monitor for
| inconsistencies.
| Xcelerate wrote:
| I remember one time we had to provide some important
| numbers to auditors. A data analyst on the team wrote
| some queries to collect this information, but my manager
| decided to write his own queries independently as a
| consistency check. The numbers didn't match at all.
|
| So he asked me to write my own queries as well to see who
| was right. Lo and behold, my numbers didn't match either
| of theirs at all.
| JackMorgan wrote:
| This feels like something that is just crazy enough to
| work.
|
| Most teams I've been on or managed were at their optimal
| efficiency with three people. Just enough to keep any one
| person from making too many poor decisions, but not to many
| to spend half a day in meetings.
| klabb3 wrote:
| > Does Conway's law imply that an organization of size 1
| produces a system with components that are optimally
| integrated?
|
| It's creative work, like anything else. 1 is not
| necessarily always the right size, but 100 certainly isn't.
| If you look at musical bands usually a sweet spot is
| between 1-6. You can add more in eg an orchestra or choir
| but everyone's not involved in the creative work.
|
| Then it depends on how well you vibe with people. If you
| have extremely unique ideas and thoughts, you may be better
| off alone. If you find peers with similar values,
| preferences and experiences, you might be fine with 5
| people.
| datadrivenangel wrote:
| The best way to de-risk a project is to run three at once.
| Just don't let the teams find out about the other ones.
| whiterknight wrote:
| Fred brooks believes the Max is 1 or 2 (if they have
| experience together), and explains how to scale an org
| around those people.
| TremendousJudge wrote:
| Does he say what to do when they die, quit, or retire?
| whiterknight wrote:
| So clever! The brief answer is you have many staff who
| have knowledge but not creative authority. If you think
| about it for a few minutes I think you'll find some
| options.
| cbm-vic-20 wrote:
| I've witnessed what Casey calls "Conway's Nightmare" (at
| 32:04 in his video) firsthand- it is definitely real.
| Basically, a software system isn't necessarily just a
| reflection of an organization, it's an amalgamation of all of
| the organizations that worked on the software system in the
| past.
| dasil003 wrote:
| Top down comes with its own risks. The bottom line is big ideas
| and big change can only happen with both the champion at the
| top, as well as good understanding at the bottom, and a
| critical mass of good alignment and competence throughout the
| middle layers. Though Conway's Law is immutable, it doesn't
| necessarily depend on a formal org chart, it can be worked with
| by simply creating ad-hoc communication structures between the
| right tech leads and competent managers. A handful of
| technically weak or empire-building managers in the middle can
| fuck the whole thing pretty easily though, and depending on the
| lifecycle of your company it may be a lost cause due to
| Pournelle's Iron Law of Bureaucracy.
| LightFog wrote:
| Although it's not always necessary to go through the middle -
| you can do the old switcheroo and build a parallel org in
| your image, emerge from the shadows and duke it out with the
| old structure.
| chasd00 wrote:
| If you come at the king you best not miss. Fail at taking
| down the old structure and those relationships are so
| destroyed they will come after you. I've seen groups of
| consultants become unemployable across the whole industry
| by failing to take down the old guard. Word spreads.
| szundi wrote:
| Very good comment
| nonrandomstring wrote:
| > as well as good understanding at the bottom, and a critical
| mass of good alignment and competence throughout the middle
| layers.
|
| How do you hold the middle together? Especially under such
| turbulence? "Strong" leadership isn't enough.
|
| Reflective practice [0] is one way to build that. It's common
| in medical and some safety-critical civil/military spheres
| but strangely missing from many corporate structures. I talk
| about it here with regard to a more mature cybersecurity
| stance [1]. It intersects with Agile ideas but doesn't make
| much noise in software engineering project management as much
| as it should? Anyone have this in their org?
|
| [0] https://en.wikipedia.org/wiki/Reflective_practice
|
| [1] https://cybershow.uk/blog/posts/reflective
| dartos wrote:
| Have fewer middle managers?
| rpastuszak wrote:
| I'm adding "Conway's Soup" to my vocab as a name for the result
| of such process.
| julik wrote:
| And even then - the senior champion might as well burn out on
| it, even when they do manage to pass the initiative. Been there
| too.
| austin-cheney wrote:
| That highlights the greatest repeated tragedy to befall
| software: poor leadership. For some bizarre reason developers
| almost always think they can fix people problems with better
| tools. For example if all your developers suck give them a
| popular framework. It's just an excuse to avoid dealing with
| people that provides license for the children to run the
| daycare.
|
| If you want excellence set high standards defined by rules that
| to hold people accountable and impose ownership
| (liability/reward). It's not complicated but it requires not
| being terrified of assertiveness and confrontation from the top
| down.
| bsenftner wrote:
| > It's not complicated but it requires not being terrified of
| assertiveness and confrontation from the top down.
|
| It requires the professional communication skills that are
| not taught in a STEM education. That's why this is all so
| difficult: we are not taught how to communicate effectively
| with one another, and the entire tech landscape has poor
| communicators misinforming and misleading one another.
| geraldhh wrote:
| Statements of poor communication always make me wonder if
| the shirky principle might be part of the problem
| bsenftner wrote:
| I do not believe so. This lack of communication skills is
| also due to a lack of recognition of the value good
| communications engenders. An undergraduate business
| school education pragmatically touches on professional
| communications, but only as far as how to motivate and
| control people, not how to effectively communicate - as
| in a shared synchronization of understanding, which is
| what professional communications is all about. Colleges
| of communications teach professional communications as
| their foundation, before the various media majors one
| graduates within. The more I look at the levels of
| communications in general within society, our
| civilization has not yet recognized how critical
| communications are to every aspect of life, not just
| business and careers. Good communications are like fire,
| capable of destroying and capable of being the foundation
| greater things are built, and the lack of this
| recognition amazes me.
| Aeolun wrote:
| I think well communicated terrible ideas are still
| terrible.
| BeetleB wrote:
| Good communication doesn't consist of stating
| tautologies.
| deckard1 wrote:
| yes, but the Senior Principle Staff Engineer that has two
| whole years of experience doesn't know the difference.
| austin-cheney wrote:
| That cannot be over-stated enough. I learned my
| communication skills in the military while building my
| software career. Those worlds are so incredibly different
| that I have spent most of my professional life lost.
|
| On one hand my expectations for excellence in software are
| alien to my peers who just want to complete a JIRA task by
| any means possible. Yes, it is possible to deliver superior
| results for an entire day's effort in about 2 hours
| provided you aren't framed by all kinds of institutional
| roadblocks and unnecessary abstractions. I am scared to
| death to actually communicate my opinions regarding
| software because people tend to whine about how hard life
| is. For this reason I have abandoned my JavaScript career.
|
| Then on the military side I have to unlearn the high
| agreeableness that is so necessary for not getting fired in
| software. By high agreeableness I mean keeping your
| opinions to yourself, not voicing concerns, treating
| everything with kid gloves, and making everything as easy
| as possible. The military side is a much safer environment
| that expects brutal honesty. Nobody there believes
| themselves to be god's gift to the universe and they are
| always in a learning mode, which means they are coach-able
| and will not whine about corrections and mentorship.
| bsenftner wrote:
| Although I do not have military experience, I've spent
| some time writing software deployed to military security
| clients, and my interactions with technical armed forces
| staff has had remarkably good communications. To the
| degree it changed my attitude about the armed forces in a
| significant positive direction, and not just the tech
| side.
| javcasas wrote:
| From the point of view of upper management, the squeaky wheel
| gets replaced.
|
| On the other hand, the non-squeacky wheel gets to collect
| their salary for a long long time.
| gilbetron wrote:
| Conway's Law isn't a "law" - it is an interesting observation
| and there are definitely interactions (both ways) between
| organizational structure and software "design". But people
| weigh it too heavily. There are plenty of counterexamples of,
| for instances, fine-grained teams in one company that build a
| monolith, and the same type of teams in another company
| building a microservices architecture. Complex software is
| complex, and once your require more than a handful of people,
| that complexity requires a complex organization structure to
| handle the challenges of communication and responsibility. But
| it isn't a purely deterministic relationship between
| organizational structure and software structure. More that
| certain organizational structures are really bad at building
| complex software, and some are better.
|
| It is interesting to think about, but there is this tendency to
| invoke "Conway's Law" as some simple method of fixing the
| extraordinary complex problem of developing and evolving
| complex software over time.
|
| Really what it is is that organizations from the 70s, 80s, and
| 90s were built around building physical things, for the most
| part. Manufacturing and construction, mostly. And taking those
| organizations and applying them to software development
| fundamentally was a terrible match. We had to find new
| structures that were better suited, and we have gotten a lot
| better, but we are still working on it. Management was taught
| by those that managed manual laborers, who's output was easy to
| measure, and whom you could force to work a certain amount
| without a significant drop in their output. The same isn't true
| for knowledge workers. And so new management had to be created,
| a process we still are working on as well, but many managers
| these days are way better than managers from 30 years ago.
|
| Read Conway's original paper, it is interesting, but it isn't a
| physical Law that cannot be violated!
| lolinder wrote:
| > But people weigh it too heavily. There are plenty of
| counterexamples of, for instances, fine-grained teams in one
| company that build a monolith, and the same type of teams in
| another company building a microservices architecture.
|
| These aren't counterexamples because Conway's law makes no
| comment on how the teams will choose to deploy what they
| build. It talks about the overall design of the system, the
| solution space that was explored. Windows is a monolithic
| application if the alternative is microservices, but anyone
| who gives it more than a cursory glance can see the org chart
| (both past and present!) reflected in the disjointed UX.
|
| > And so new management had to be created, a process we still
| are working on as well, but many managers these days are way
| better than managers from 30 years ago.
|
| I'm not sure why this makes Conway's law not applicable
| anymore--what you're describing is that new management does a
| better job of creating communication structures that lend
| themselves well to creating software. That may well be
| correct, but the resulting improvements _validate_ Conway 's
| law! If we're getting better at software, it may well be
| _because_ people are talking about and accounting for the
| impact that communication structures have on the output.
| adql wrote:
| > There are plenty of counterexamples of, for instances,
| fine-grained teams in one company that build a monolith, and
| the same type of teams in another company building a
| microservices architecture.
|
| ...like ?
| saltminer wrote:
| Oracle certainly has. Several GBUs realized they were
| working towards the same goal of a microservices base layer
| for future service hosting (for handling auth, logging,
| databases, etc.) and combined their efforts, and I happened
| to work on one of the resulting teams.
|
| I'm not sure how public they've been about all this, but
| you certainly can teach an old dog new tricks, provided
| there is management buy-in (which we certainly had - at one
| point, our teams were moved between GBUs to save us from
| budget cuts).
| krig wrote:
| It's not a physical law - it relates to people - "law" has a
| wider meaning outside the field of physics.
|
| I recommend watching the talk! He specifically addresses the
| very points you bring up. Of course reality is messy and
| people especially so, but Conway's law is one of very few
| theories within CS that has held up in replication studies.
| snotrockets wrote:
| That's the thing that stumped me here: how does a _Senior Staff
| Engineer_ doesn't understand their job isn't to build software,
| but to enable the rest of the organization to build software?
| (Paraphrasing on that old Toyota principle)
| joshhart wrote:
| I spent 12 years at LinkedIn. Sadly, it's not even close to the
| engineering org it used to be. The era where Kevin Scott led
| engineering was a really good one in comparison.
| hasty_pudding wrote:
| Ryan Rolansky said LinkedIn is essentially feature complete.
| Redoubts wrote:
| Isn't it?
| pcloadletter_ wrote:
| They seem to be trying to jam AI into a bunch of things
| now.
|
| As an aside, I work on a product that should be considered
| feature complete, but PMs keep dreaming up features so they
| can get promoted. I wish we could just take some time to
| clean up the horrendous tech debt accrued over the past 10
| years of feature proliferation.
| gardenhedge wrote:
| If you consider lots of features that don't work "complete"
| then I guess
| MisterPea wrote:
| Every engineer of a company of this size says the same exact
| thing.
|
| IMO more about growth of the engineering teams that lead to any
| specific culture deteriorating.
| mkl95 wrote:
| > That's 20 times almost the size of my, how, how do you even
| build that? How long to build? Oh, 17 minutes for a new build.
| That's, that's not bad. We should probably make that better, but
| that's not bad.
|
| Am I spoiled for thinking 17 minutes is too much, even if it's 2
| million LOC? I guess LinkedIn can afford it
| esprehn wrote:
| Yeah, they could definitely optimize it quite a lot further.
| With investment in code structure and caching I bet they could
| get below 5min at that scale. But also for edit/refresh that
| should be on the order of a second or two, never minutes.
| liampulles wrote:
| It is not too much or too little. The real question is whether
| the value of X% speedup worth Y hours of developer time to make
| the improvement. Relative to other work that could be done.
|
| Answer could well be yes since build time impacts everyone, but
| also have to factor in complexity of the CI system they use.
| bluepizza wrote:
| No, that's not the real question. Decisions have more factors
| than just what cash will they bring in.
|
| Long build times are an operational risk - it's very hard to
| apply patches to solve incidents with a 20 minute build time.
| liampulles wrote:
| To clarify: when I say value I don't just mean monetary
| value - though of course everything in a business
| ultimately does boil down to that, including the total
| expectation of risks.
|
| I agree though, definitely a factor
| izacus wrote:
| So reducing this would be a multi-year project of a
| sizeable team (let's say 5-10 people). How do you get
| someone to sign off for this kind of expediture and not lay
| everyone off when the next economic fart comes?
|
| Because THAT'S the hard problem to solve in any bigger
| organization. Not hacking the code.
| pcloadletter_ wrote:
| >So reducing this would be a multi-year project of a
| sizeable team
|
| Maybe, or just a knowledgable person to dig in for a bit
| and solve it. On one of my teams when I was back at
| Microsoft was working on a similarly huge codebase with
| build times twice as long. We had a senior engineer who
| was really into build optimizations from a previous job
| who wrote a tech design, implemented better project
| splitting in our monorepo, better local caching, and
| finally cloud caching, all within 3 months. When I left,
| our builds ranged from 30 seconds to 3 minutes.
|
| It's not guaranteed that it'll be this simple, but I also
| don't think it's guaranteed it'll be 3 years and several
| people.
| legulere wrote:
| It's not as easy as it seems to answer that question. You'll
| have rebound effects meaning developers will probably
| recompile more often.
| ryanjshaw wrote:
| It depends on what those LOC are. Is it HTML? Is it JS? It it a
| compiled language? How much of it is generated?
|
| Aside: I don't understand the obsession people have with
| throwing out LOC. It's a completely meaningless number unless
| you're familiar with the codebase, in which case you don't need
| to be told the stats.
| ZaoLahma wrote:
| > Aside: I don't understand the obsession people have with
| throwing out LOC.
|
| Agreed. A better metric is how much of it you need to
| interact with on a daily basis in order to perform any
| meaningful work.
|
| If the code is sane, you'd only ever need to interact with
| and understand a fraction (a subset of the public APIs) of
| the entire codebase. You could have millions of lines of code
| and be productive when knowing only a couple of hundred of
| those.
|
| If the code is insane, the risk is that you'd need to
| interact with most of it when introducing any non-trivial
| functionality. Worst case scenario there isn't any clear
| rules of what is public and supposed to be used, and what is
| private and isn't supposed to be used, and people have used
| "whatever from where ever" to solve their problems.
|
| The second one is always bad, but also significantly worse
| the more lines of code there are. Perhaps those who throw
| LOCs around also implicitly tell us that they are dealing
| with the second variety of code.
| blitzar wrote:
| > I don't understand the obsession people have with throwing
| out LOC
|
| LOC is the metric I get to remain employed or fired on.
| rkangel wrote:
| The more important number for a codebase that size is almost
| always "what's the incremental build time for whatever I work
| on". It's _nice_ to have a "build from scratch" be fast, but
| almost always you can leave it for two hours to do the first
| build while you're in a meeting or something. More important is
| that when you make a change, the incremental build is fast.
|
| This is why build systems like buck/pants/bazel are good in
| larger codebases. You can _trust_ your incremental builds in a
| way that you can 't in any build system that isn't hermetic.
| pcloadletter_ wrote:
| And we have cloud caching for the "build from scratch" case
| so even that can be pretty quick. This all just takes a
| little time investment.
| throwawaaarrgh wrote:
| Poor kid, learning how an enterprise works the hard way. If he
| stays in software eng long enough he'll be back here eventually,
| with a little less gumption, but a little more tactics and grit
| to fight the battles he wants to fight.
| cybrox wrote:
| Are we belitteling people now because they have intrinsic
| standards and pride in the quality of their work instead of
| blindly chasing whatever is best for their replaceable career
| in a hypercapitalistic environment?
| yard2010 wrote:
| No. We're just being condescending pricks on the internet
| JohnL4 wrote:
| Huh, self-aware HN. Don't see that very often.
| doublerabbit wrote:
| Hopefully they'll learn that such standards don't exist and
| that if they do will deteriorate by the second year.
|
| If it wasn't for it being some big enterprise corporate
| company no one would give a stuff about the blog post.
|
| Do I care someone left company X, fired from company Y? No.
|
| Do I acknowledge it's upsetting? Yes, Pack your stuff and try
| again.
|
| welcome to life.
| CoastalCoder wrote:
| Aside from the GP comment calling the guy "kid", it doesn't
| seem especially condescending to me.
|
| I read it more like "Yeah, we've all been there. It's a sucky
| lesson to learn, but in the end he's likely to be wiser
| (albeit more jaded) to these things."
| throwawaaarrgh wrote:
| Not really bro, but you stay angry
| ztratar wrote:
| Sounds to me like he made a couple unfortunate moves:
|
| - Propose a 5 year plan (lol) - Don't lead the incident with
| leadership, but with blame - Speaking about problems more than
| solving problems (hard to do) - Lack of relationship building
|
| On one hand, I empathize with Chris. On the other hand, this
| sounds to me like he just didn't know how to perform in this
| environment. And that's totally fine! Not everything should learn
| how to perform in the bureaucratic knots -- startups are simpler
| in this way. And there's a reason the big guys lose their edge
| over time, and then some exec in a board room is faced with -10%
| YoY loss without any truthful VPs around the table.
| mjburgess wrote:
| I go back and forth on how to interpret people like Chris (and
| likewise, myself).
|
| I've been in this situation, and it's psychologically
| disorienting: I seem to be right, but am I? Are the people
| around me really as incompetent as they seem, as totally
| disinterested in learning from their colleagues, etc. etc.
|
| Then a few years after leaving, you look back and: oh, all
| those people have been fired or left; they still cannot do X as
| an org; the agility and competence of teams I then joined
| _really_ does exist etc.
|
| So, on the one hand, it's true that there's a sort of
| arrogance, political incompetence, and inability to cope with
| environments with a pathological practice culture --- but, _on
| the other hand_ , maybe that's the right reaction?
|
| Maybe the institution is undergoing a pathological cultural
| period, and maybe talented, considered, passionate people
| _should_ be driven mad by it. Maybe the people who arent driven
| mad by it are either irrelevant to productivity /growth, or
| worse, net deadweights.
|
| Who knows? That's what makes being in this environment such a
| psychodrama -- is the situation really this bad, or am I over-
| reacting? "Everyone tells me...."
| yard2010 wrote:
| Don't gaslight yourself. Believe! Remember that shit floats,
| and you have to pick your battles.
| dade_ wrote:
| Scum rises to the top.
| pm90 wrote:
| Its very hard to affect real change without power. The people
| who are genuinely interested in solving problems generally
| don't want to deal with the political intrigue that comes
| with power. So we're left with a situation where the leaders
| suck for the most part.
| J_Shelby_J wrote:
| Playing politics is hard. It's literally the job of
| business management and the only way they get a paycheck.
| So they're well practiced and beating them at that game is
| difficult.
|
| If you're technical, you can get a paycheck just by doing
| the work. Sure, you can play politics and tech, but is it
| worth it? Unless you are rewarded as a stakeholder, I'd say
| it's not worth it.
| camgunz wrote:
| A friend said something to me years ago that I find more and
| more wise as time goes on: almost everything is about
| aesthetics. Any given group of people (a company, a band, an
| advocacy group, a family) is pursuing its own sense of
| aesthetics. What does this look like in practice? Let's say
| using Elasticsearch would save $1M of engineering time and
| infra costs over using PostgreSQL for search but the people
| in charge of search have a vague aesthetic belief that
| Elasticsearch is garbage and they're willing to spend $1M of
| the company's money to not work with it. Is that nuts? Is
| that rational? Is that good engineering? Is that judicious
| use of company resources? Their bosses are the ones who make
| those decisions.
|
| More frustratingly, this works the other way. You might
| skunkworks Elasticsearch into your org. You might have cost-
| saving metrics up the wazoo; you might have multiple engineer
| testimonials; you might have satisfied customers or growing
| sales; you might have better velocity metrics; you might have
| built profitable services on top of it. An Elasticsearch-
| hating engineer can convince your boss that it's a ticking
| time bomb--with _no evidence whatsoever_ --and your boss can
| say, "hey I found out you're using Elasticsearch for all
| this, which is a big no-no. I love what you're doing here,
| but you need to migrate to Postgres".
|
| (I know Elasticsearch flunked Jepsen; it's just an example)
|
| You might think a few things:
|
| - can't I show all my successes and change hearts and minds?
|
| - isn't this how we evolve as a company? we skunkworks
| things, see if it works, build more on top of it, repeat?
|
| - if we don't take risks, won't another company out-compete
| us at some point?
|
| - won't some manager take notice of the inefficiency and do
| something about it?
|
| Sometimes the answer is yes! But it's almost never yes for
| the right reasons; it's merely the case that you found a
| manager who hates the PostgreSQL team, or hates one
| particular PostgreSQL advocate, or finds it advantageous to
| be seen as a manager who's spearheading new initiatives, or
| thinks that a homogeneous tech stack is a vulnerable tech
| stack, etc.
|
| This is the way we all work. We all have our irrational
| heuristics built up of lifetimes of experience that we hate
| interrogating; we naturally gravitate to others with the same
| irrational heuristics because they spare us that
| interrogation; we assault others who force that interrogation
| upon us because it's painful. You'll see this dynamic in
| political tribes, in technology choices at work, on reality
| TV, everywhere. You can watch us tie ourselves into knots in
| real time, selectively forget events or facts, or wildly
| misinterpret things, all to defend our aesthetic preferences.
| This is the psychological disorientation you're experiencing.
| It's not common to find people who are aware of these
| tendencies and--successfully--employ strategies to
| compensate, and it's even less common to find groups of
| people who do that. The probability is inverse to the number
| of people.
|
| Like you (and I guess OP), I've fallen victim to this over
| and over again on both sides. There is no justice, because
| there is no God and nothing even approaching perfect
| competition. Some of those groups are doing great, others are
| deceased. I've only relatively recently figured this out, so
| I've now added some things to my interview questions like
| "tell me about a time you were very wrong" and "how do you
| reevaluate decisions or longstanding beliefs you have". I
| also try to check myself whenever I react a little _too_
| positively or negatively to something, because that 's
| usually an indication that it's aligned or across my
| aesthetics and something I need to dig into.
|
| The good news is I've had success instilling this into people
| I've worked with. It's sort of "a minute to learn; a lifetime
| to master", but it starts creating a culture of psychological
| safety where everyone relies on the others to help them be
| the best they can be.
| vhiremath4 wrote:
| Strong +1. Also hi Zach!
| chriskrycho wrote:
| A bit of clarification from some of the nuance that
| (reasonably) had to get cut from the episode for time:
|
| - Propose a 5-year-plan: yeah, "lol" is about right. It was
| never going to win any hearts or minds. It was also _our_ least
| favorite plan. But it was also the only one we felt we could
| actually bring to our executive leadership given what they were
| telling us prior to that, which was basically "Even for a
| migration we are _asking_ you for, do not slow down product
| iteration velocity _at all_." How do you do that? Welllll...
|
| - I'm not really sure what you're referring to about leading
| the incident with blame instead of leadership. It's possible
| something got lost in translation with the amount we cut, but I
| actually aimed very much to do the opposite. We didn't blame
| the people who lowered the memory thresholds, the people who
| typo'd a bad value in YAML, or anyone else. I just insisted
| that we actually solve the root issues instead of leaving them
| to fester for the _next_ poor person who happened to be around
| when it inevitably blew up again.
|
| - "Speaking about problems more than solving problems": really
| not sure what you mean here. I didn't spend a ton of time
| bragging about what I had pulled off on the show because wow
| would that ever have been in poor taste, but... I did pretty
| well in the problems I solved there.
|
| - Lack of relationship building: yeah, I called that out on the
| episode! It was my weakest area. I had a really good rapport
| with engineers, and failed pretty miserably at building cachet
| with management, _especially_ with management above me.
|
| I wouldn't say in the end that it was just down to not knowing
| how to perform in the environment, though. Some of it was also
| a choice, in the end, _not_ to perform in ways that I could see
| would be successful _in the environment_ but which I simply did
| not believe in. I think a lot of the engineers I respect most
| are like this: can and will do the political dance _for
| something they believe in_ ... but not for things they _don't_
| believe in.
| lloydatkinson wrote:
| Chris: There we go. Okay, those are running now. Crap. Did I
| screw up my ability to hear you?
|
| Adam: Hello, hello, hello?
|
| Chris: I did. Dang it. Hang on.
|
| Adam: Hello, hello?
|
| Chris: Yeah, yeah, we're good now. Okay. Let's try that again.
|
| Truly fascinating dialog
| Raed667 wrote:
| It is a fairly common podcast trope to leave some bloopers at
| the start to humanise the people there.
| frou_dh wrote:
| They know you need something to complain about, so are giving
| it to you upfront.
| patja wrote:
| Can I complain about the podcast website that truncated the
| right side of the content on my phone and won't let me scale
| it to correct?
| gandutraveler wrote:
| It's a thing. Like how GenZs start their tiktok videos showing
| fake urgency.
| willvarfar wrote:
| Its interesting because, although I don't know the linkedin
| codebases nor worked at linkedin, I've seen a few codebases that
| sound scarily similar and am familiar with a few companies that
| sound organisationally and politically scarily similar.
|
| And I generally advocate 'finger gun' approaches.
|
| Finger gun rewrites can be implemented in a good way. Often when
| you have n clients trying to do the same thing, one of those can
| be used as a basis for serving the other platforms. Or if you
| start over, you can do it in a good clean quick concise way.
|
| The general success ingredient is putting a small team of
| veterans on making the new system. Success always comes from a
| small team of engineers who are both domain experts and technical
| experts rolled into one. (Controversially, I think that
| generalises; all success comes from this, even for mundane keep-
| the-lights-on problems. And everyone else is just slowing them
| down.)
|
| The big problem most tech management everywhere keeps repeating
| is that the next big system will be built the least-experienced.
|
| Would be really nice to hear the symmetrical interview from a
| finger-gunner.
| shp0ngle wrote:
| The article is not very specific so I can't really tell one way
| or other
|
| 5 year plan is sounding really bad, though
| willvarfar wrote:
| Yeap! We can't predict the future but we can all be very
| surprised if whatever tech is picked today is still a good
| fit, or even still exists, in 5 years. Especially in the
| javascript space.
| jameshart wrote:
| This is becoming less true than it once was. React has been
| 'this month's JS framework' for 10 years now.
| CuriouslyC wrote:
| As a sibling mentioned, this isn't really true in the JS
| space anymore. The JS frameworks and react/vue have pretty
| much won, so now you one of those UI kits and whatever
| build system/ancillary tooling your framework says you use.
| There might be churn under the covers, but the framework
| abstracts that away for you (though occasional migration
| issues are bound to come up with big changes).
| willvarfar wrote:
| it's always quiet before a storm :) It'll be fun to fast
| forward five years and see whether there wasn't some big
| disruptive change and everyone is now using rust++ for
| client dev or something? :D
| chriskrycho wrote:
| This is a reasonable take, and I mostly share it. None of us
| were especially happy about that plan! However, and I alluded
| to this on the episode, it wasn't actually a five year plan
| to get to a very specific endpoint. It was, instead a
| 3-5-year plan to make steady incremental changes to our
| entire system in ways that meant that if we realized our end
| destination needed to change (e.g. from React to Svelte or
| something not even built yet) 2 years along, that was
| _totally fine_.
| elbear wrote:
| It's not clear to me what ended up happening. Did they
| start a rewrite from scratch in React?
| chriskrycho wrote:
| They started a rewrite-from-scratch of _all_ the apps,
| including mobile, using a custom in-house server-driven
| UI framework with a Kotlin DSL backing it, without so
| much as doing a proof of concept to find the tricky parts
| first. And by "started" I mean "committed to it" before
| anyone had so much as written a line of code to spike it
| out. Which, YMMV, but to me that seemed then, and still
| seems now, just _wild_.
| elbear wrote:
| Yeah. I hope they're at least doing it gradually. As in,
| rewrite a small component and have it run side by side
| with the rest of the original app, then rewrite another
| one and so on.
| begueradj wrote:
| Most of the time, you have to deal with codebase. And when you
| finally have chance to work on a greenfield project, you will
| have to compromise between your opinions and wishes with those
| of everyone else who is involved.
| blitzar wrote:
| > when you finally have chance to work on a greenfield
| project
|
| the person that comes along next is going to moan about how
| much of a mess of compromises and hacks your, now, legacy
| codebase is
| EchoChamberMan wrote:
| I have never heard of a finger gun rewrite - what is that?
| jameshart wrote:
| From the podcast:
|
| > It was completely what another colleague of mine once
| described as being in finger guns mode, meaning like, yeah,
| yeah, yeah, this is going to be awesome, man, kind of finger
| gunning at each other without answering any of the kinds of
| questions about what does it look to like to operate this
| when we're trying to support hundreds and hundreds of
| engineers...
| EchoChamberMan wrote:
| Gotcha thanks.
| randysalami wrote:
| Tbh this was kind of my conclusion as someone new to the field
| (23). In technical consulting, I feel like this phenomenon is
| even more pronounced since A. most technical consultants don't
| get the time to become domain experts in their client's
| business, B. most technical consultants aren't entirely
| technical experts (consultants first, developers second), and
| C. consulting management has a vested interested in keeping
| things this way because they don't have the tools to make
| things different. Maybe this doesn't happen as much in boutique
| consulting firms but I wouldn't know (hire me if that's the
| case).
|
| It's given me an idea for a two-way portal where on one side,
| technical developers join the platform, self-organize into
| independent, transparent teams (not just developers but any
| role in development e.g. UX designer, project manager, etc) and
| on the other, companies join, post their potentially cross-
| functional work requests which are then distributed to the
| relevant teams. Another important concept is that these
| instances can be federated(?) so existing consulting firms can
| integrate this new kind of work allocation into their pipelines
| (only their consultants/their invited companies). There are a
| bunch other concepts I've thought through but that's where I'm
| at now.
|
| I feel that a system like this provides better value to a
| company vs. traditional consulting. First, the responsibility
| of team formation, networking, and allocation is moved to the
| service. These three things are hard for humans to do
| themselves and are responsible for so much consulting admin
| bloat/inefficiency and companies have no choice but to play.
| Second, it in theory improves the quality of work teams. With
| this focus on smaller, self-organized teams and a UX that
| allows for transparency I.e. companies can view teams, their
| members, and track record, companies know what they'll get and
| not need to roll the dice to see what available resource gets
| assigned to them. Moreover, I've read that consultancies have a
| tendency to throw certain resources on a large work item where
| they exist to just to cash in budget and write bad code. This
| problem is completely mitigated with this approach. Finally,
| it's a decentralized work environment (I think). Besides OSS, I
| don't know what other platform allows for developers to have so
| much autonomy and self-determination. You could argue sites
| like Upwork or Freelancer but these platforms seem more focused
| on small, individual jobs rather than larger work items which
| justify having larger, more competent teams.
| NamTaf wrote:
| Without negating your idea, I'd add that as a (non-IT)
| engineer of some 15 years, I've found that my first 8-10
| years was building my technical capabilities, however beyond
| that it was really about learning how to navigate (at
| increasingly more complex levels) the business systems and
| fit the technical to the business.
|
| Being a principal engineer, my major value is not in my
| technical capacity, but rather my ability to shape the
| technical solution to the best fit for the business - be it
| to do with existing tooling, processes, systems, etc. Though
| I can do the technical solutions (probably more efficiently,
| too), my value is much stronger when I steer multiple junior
| engineers doing the technical work whilst setting the
| guideposts so they align the solution to what best fits the
| business.
|
| I would therefore suggest that one major hurdle would be that
| these self-organising teams might not be able to bring to
| bear their full capacity insofar as they are not familiar
| with the businesses to which they pair. Though you can
| sometimes see parallels between company processes,
| understanding the nuance and detail that makes them different
| is where so many consultancy projects fall over. Think about
| the canonical meme of a team of consultants coming in and
| spending millions developing a solution that turns out to be
| a lemon. It's not for their lack of capability, it's that
| it's devilishly hard to really, truly fully understand a
| business's processes, systems and cruft in a sufficient level
| of detail to truly shape a solution such that it remains
| maintainable and fit for purpose in a long-term sense.
|
| That's the real reason why veteran employees are worth their
| weight in gold. It's that they are the custodians of the
| institutional memory and can bring that forward when it is
| needed.
|
| This of course has the important counterpoint of the
| necessity of seeding new ideas into a business, and that's an
| entirely different essay in itself, however suffice it to say
| that new ideas need to be integrated in a sustainable and
| controlled fashion - protected and nurtured from being driven
| out by the existing ways, but also moulded to integration
| where it's necessary. Failure to do either is a speedrun to
| pain.
|
| Edit to add: If you think that IT is somehow immune from this
| relative to traditional meatspace engineering disciplines,
| I'd say two things: 1) tech still goes through this, just
| sometimes operates on different timescales, and 2) go see if
| a COBOL greybeard collecting FU consultant wages to maintain
| old OT systems agrees with you.
| jameshart wrote:
| There's definitely a middle ground between 'we need to know how
| this plan will overcome every obstacle we will encounter before
| we even begin' and 'there won't be any obstacles what do you
| mean?'
|
| I always want to have a sense that the plan gives us _realistic
| options_ for dealing with the obstacles. And that doesn't just
| mean technical options - we'll need to have the people to
| tackle it too. Eg it's not great if the plan involves a bunch
| of teams running servers, and while _technically_ there are
| obviously ways for all those teams to operate and scale those
| servers, if they don't have the time or skills to do so then
| that option of having the teams run stuff is not really
| realistic.
|
| But on the other hand planning a careful course that navigates
| round every obstacle is just as bad because
|
| 1) the obstacles will likely have moved by the time you get
| there (eg we spend ages figuring out how teams on legacy JS
| will be affected by the project only to find by the time we get
| there everyone has already moved to typescript and all that
| prep was YAGNI).
|
| And 2) there are probably obstacles right in the path you've
| carefully chosen that you don't know about yet and when you hit
| them your plan will grind to a halt because there was only one
| way planned and this was it.
|
| We're reading a cartoon podcast description of a complex
| architectural debate here though, so who knows which of these
| straw men was actually close to what was happening in LinkedIn.
| chriskrycho wrote:
| This is a great summary. Obviously I bring my own perspective
| to this, because, well... it happened to me, and I "lost".
| But my read was that the folks I disagreed with and
| ultimately parted ways with were falling into the bucket you
| label 'there won't be any obstacles what do you mean?'.
|
| The tradeoff around "do we migrate to TS?" is a _great_
| example here, by the way. A huge part of our decision-making
| last year around the big app at LinkedIn was on exactly that
| question: If you are going to stop and throw the entire thing
| away, _you should waste exactly zero time migrating any of
| it_. On the other hand, if the thing is going to be migrated
| incrementally, you should _accelerate_ the TS effort because
| it will make it so much _easier_ to code-mod the code base.
|
| (I don't agree that it's a "cartoon" podcast description, but
| it is definitely extremely compressed and Adam and I chose to
| focus on the personal dynamics over the technical details,
| because diving into the latter would have made for a 5-hour
| episode. Also: it's really tricky to do any of this kind of
| public discussion in a way that doesn't just end up reading
| as a one-sided self-defense!)
| test1235 wrote:
| >The big problem most tech management everywhere keeps
| repeating is that the next big system will be built the least-
| experienced.
|
| Most places I've worked at, this ends up being the case because
| the most experienced are needed to fire fight and to keep
| things running.
| cogman10 wrote:
| Stop describing my workplace
| samstave wrote:
| Stop being the least experienced!!! (by tenure alone)
| foofie wrote:
| > (...) because the most experienced are needed to fire fight
| and to keep things running.
|
| By that account the most experienced started out as the least
| experienced who built a mess, accrued more technical debt,
| and built up their experience keeping their ball of mud from
| crumbling.
|
| They are also more resistent to change because this
| underlines all the crap they did. It's their baby, and their
| reputation is pinned on how successful it is.
|
| New is not necessarily better, but old and stable also
| doesn't mean fine engineering.
| samstave wrote:
| > _started out as the least experienced who built a mess,
| accrued more technical debt_
|
| THIS
|
| "experienced doesnt always mean good. This is why you build
| teams.
|
| The CFO is your enemy as an engineer.
| sidewndr46 wrote:
| > putting a small team of veterans on making the new system
|
| I talked to someone who was leaving his job because he did this
| in order to meet a critical date for a product launch. The fact
| that his group of 3-4 people did what the company could not in
| a year ruffled so many feathers he said the board told the
| entirety of engineering that under no circumstance would anyone
| be allowed to dictate any detail of future projects.
| swozey wrote:
| Yeah, I've been these "cool Skunkworks" teams and I literally
| made no more money, maybe got a $5k spot bonus (super rare),
| and burned myself out over a 6mo launch. Spent my entire
| thanksgiving week on-call putting out fires, literally every
| hour, while everyone else at the 800 person company was on
| vacation. Not a single executive or product manager involved
| in launching was there or checked in on things. Some moron (I
| hope he reads this, my nicks not anon) let us launch without
| connecting our credit card fraud system so we (ops, me) got
| slammed for an entire week with fraud server launches. Jason,
| you're a moron, and that's an understatement.
|
| It's like being paid in exposure. We'll pay you in cool tech
| and learning!
|
| Then you see the 20 go devs around you making the same money
| writing grpc services 8 hours a day home at 5 relaxing.
| sidewndr46 wrote:
| I'm not really sure how this relates. The story I was told
| had the guy working as the lead with several less
| experienced but still competent programmers for around 2
| months. There was no crunch, burnout, or anything of the
| like.
| Solvency wrote:
| You're being downvoted because most devs are blue collar
| and hate the idea of passionate talented devs doing
| something amazing quickly and effectively. Any anecdotes
| of small elite team success must be crushed.
| Aeolun wrote:
| If you are working more than 8 hours a day you have nobody
| to blame but yourself I think. Especially in a situation
| like that.
| duxup wrote:
| To me it seems crazy that a board would be getting their
| hands dirty on engineering "specifics". I would think you
| would have engineering level leadership making those calls
| and the board would either trust that person or should move
| on.
|
| Anyway:
|
| I will see that I always worry about super teams because
| super teams tend to get free rein, so of course they can be
| faster and more successful. But sometimes that's about the
| bureaucracy and not even the team members.
|
| I've started projects with a team and due to time factors
| decided "I'll just take all of this." and it worked out
| great. That's not because I'm amazing... it's just the old
| adage about "What one programmer can do in one month, two
| programmers can do in two months".
|
| Unfortunately, I've seen places where these super teams /
| people are setup and endlessly praised and it goes on and on
| and it becomes very clear it isn't them, it's the structure.
| sidewndr46 wrote:
| let me tell you a simple story
|
| 1. team A developed some really nice looking graphs from
| customer data we already collected
|
| 2. a member of team A pointed out he is colorblind and the
| palette did not work for him
|
| 3. product manager took the time to research this and
| figure out how we might make the palette configurable for
| those who are colorblind
|
| 4. CEO got wind of this and called everyone involved into a
| meeting. The decision of the CEO was no one would ever use
| colors for any data presentation. The only allowable color
| was a shade of gray
|
| I'd like to point out that using only a shade of gray for
| "colors" effectively makes the entire planet colorblind
| bluGill wrote:
| As someone who is colorblind I find shades of gray better
| than the colors I often see. OTOH, I can see most colors
| just fine, it is just that some colors most people think
| are very different look the same to me. There are a
| handful of people on earth who cannot see colors at all.
| duxup wrote:
| >The general success ingredient is putting a small team of
| veterans on making the new system.
|
| I don't disagree with this approach. But one thing I find is:
|
| Sometimes those veteran teams are heavily invested in existing
| concepts, tools, and approaches. It makes sense, they have seen
| these things work and they're proven to some extent.
|
| But it's always not the best choice.... and while the veteran
| teams often start these projects they often aren't the people
| to "finish" them as they get pulled away. And it is oh so easy
| to start projects and make decisions if you never see the
| outcome / have to deal with the fallout.
| HumblyTossed wrote:
| > But it's always not the best choice....
|
| I think it comes down to the personality of the veteran. If
| they're the "i've seen some shit, so I'll 'architect' the
| fuck out of this to handle any type of change" type veterans,
| then it's likely to fail. But if they take their knowledge of
| how the system has grown over time and architect to handle
| the types of changes they've seen, then there is likely a
| greater chance of success.
|
| Personally, I think a having some junior and mid-level
| engineers on the rewrite is good because it's a good way to
| see how the system is all put together (as opposed to just
| one area) and they'll be able to have conversations with the
| veterans about how the system has evolved as it grew and that
| is valuable information.
| liquidpele wrote:
| Meh. I've met brilliant juniors and worthless veterans. I
| think it's more important to recognize brilliance in
| people... That spark of hard work combined with vision and
| creativity.
| willvarfar wrote:
| Yeap, good point, tenure != expertise.
|
| If I could edit the post I'd insert a qualifying "expert"
| before veteran, where the expert veteran meets my definition
| of the 10x developer :D
|
| Any veterans who aren't gonna get the project done quickly
| _and_ properly aren't the kind of veterans I have in mind.
| mjr00 wrote:
| > Sometimes those veteran teams are heavily invested in
| existing concepts, tools, and approaches. It makes sense,
| they have seen these things work and they're proven to some
| extent.
|
| This is very closely related to why these veteran teams end
| up successful. Existing concepts, tools and approaches
| _should_ be the default; if your company uses Python with
| MySQL you better have a really, _really_ good reason for your
| new project to use Rust and CockroachDB.
|
| > while the veteran teams often start these projects they
| often aren't the people to "finish" them as they get pulled
| away.
|
| I disagree with this. The veterans are veterans because
| they've already spent time maintaining the existing systems.
| They've handled oncall, major bugs, and seen all kinds of
| stupid shit that happens with software. They'll get pulled
| onto other projects eventually, sure, but they have a higher
| chance of getting a system into production in front of users,
| IMO.
| duxup wrote:
| I disagree. There's no magic to being on call that means
| you understand something well enough. That's simply not
| automatic.
|
| Following through with something from start to "finish" is
| far far more valuable than fixing one off bugs and then
| doing something else, and then something else.
|
| If anything I think it can lead to a lot of poor
| assumptions based on very skewed experience.
| Aeolun wrote:
| I think the implication is they've seen all problems
| before and can engineer their new system to solve them.
| mjr00 wrote:
| Depends on the nature of your oncall. If your oncall is
| just following documented restart procedures when your
| app gets stuck, you won't learn much. If your oncall
| involves deep investigation into your system and how it
| interacts with other components (services, databases) etc
| you'll learn a _lot_.
|
| per some sibling comments though, years of experience !=
| veteran, to be clear. There's certainly people who have
| been working for 10+ years, sometimes even at the same
| company, and don't really understand what they're working
| on.
| edgyquant wrote:
| Most software is never finished so I have no idea what
| arbitrary point you're deciding they need to see it
| through before qualifying here.
| bluGill wrote:
| Some changes are better than others. If your product works
| in python/MySQL then rust probably isn't a good fit. Maybe
| Ruby or PostgresSQL would be better fits.
|
| If your main product is C or C++ then you should be
| learning about things like rust. Maybe you decide modern
| C++ is good enough, but if so you should be working with
| the C++ committee to push safe C++ (as a C++ developer I'm
| hoping this comes in C++26... I'm looking at how I can put
| rust into my code for now).
|
| The point is you should know what the good next steps to
| investigate are and what are not. Rust and Python are both
| great tools, but it is never okay to choose between the two
| on any project - the choice is obvious before you get to
| choosing. Both tools have other competitors that you
| perhaps should be looking at though (Rust - ada, C++.
| Python: ruby, go)
| mjr00 wrote:
| > Some changes are better than others. If your product
| works in python/MySQL then rust probably isn't a good
| fit. Maybe Ruby or PostgresSQL would be better fits.
| [...] Rust and Python are both great tools, but it is
| never okay to choose between the two on any project - the
| choice is obvious before you get to choosing.
|
| I actually really disagree with this. Python and Ruby are
| extremely similar in the problem they solve. If you're
| using MySQL as a basic OLTP database, and you need a
| basic OLTP database, you shouldn't use Postgres.
|
| I'm talking organizationally here, not just on a single
| project. You gain nothing by introducing Ruby to your
| company that's filled with Python experts. Instead, you
| introduce a bunch of friction and overhead as developers
| have to relearn things like syntax, how to set up their
| development environment, how to do dependency management,
| testing frameworks, etc...
|
| The time to introduce new tech is when it _is_
| fundamentally different enough that it could matter.
| Maybe you have a bunch of Python code, but there 's a
| certain area that's very performance sensitive: you could
| look at using Rust, Go or C++. Maybe you have MySQL, but
| you need a database that's used for complex OLAP queries
| rather than low-latency OLTP work. There you could look
| at Redshift, Snowflake or the various Postgres OLAP
| configurations.
|
| Where experience comes in is knowing when these new tools
| are worth the overhead they cause. Less experienced
| engineers vastly underestimate the overhead that comes
| with introducing new tooling. This even applies
| internally; for example, if you've used Flask for every
| project you've done at your company for the past 5 years
| is it really worth switching to Django?
| DiggyJohnson wrote:
| > The time to introduce new tech is when it is
| fundamentally different enough that it could matter.
|
| I think you agree more than you realize. GP is
| essentially saying there are few situations where you
| will be deciding between Python and Rust on account of
| organization and technical reasons. However, sometimes
| you should consider introducing new technologies, but
| only when you identify that the problem being addressed
| is much better solved by a new technology, fully aware
| that this will cause various sorts of friction within
| your organization.
| AndrewKemendo wrote:
| >The general success ingredient is putting a small team of
| veterans on making the new system. Success always comes from a
| small team of engineers who are both domain experts and
| technical experts rolled into one. (Controversially, I think
| that generalises; all success comes from this, even for mundane
| keep-the-lights-on problems. And everyone else is just slowing
| them down.)
|
| Fully agree with this for all scopes of projects. Larger
| projects are compositions of this principle with functional
| communications between focus areas.
|
| There's no financial incentives for Maintenance though, which
| means "leaders" have less and less incentive to fix what they
| have versus building from scratch. In my experience in these
| cases it comes down to (like most things) whomever the least
| flexible person in a power position decides to do.
|
| Fundamentally the problem with all large organizations is that
| desires and incentives become diffuse the more people are a
| part of it and the "alignment" of what should happen drifts,
| until eventually nobody cares to maintain the original purpose.
|
| You can argue this is good or bad or inevitable, but either way
| it's a structural issue between the broader world changing, and
| a organization that seeks consistency
| camgunz wrote:
| I'm also generally a finger-gunner. I don't identify with the
| stuff OP was criticizing though; in particular I want people to
| poke holes in the plan: those are usually requirements we need
| to meet or at least talk about why we don't need to meet them.
| But yeah, other than that, I generally think it's easier to
| rewrite things than it is to understand the intricacies of a
| gazillion-line software project. That probably strikes a lot of
| HNers as problematic (this is why we keep getting new JS
| frameworks every day! this is why we keep reinventing email!
| this is exactly why we keep building gazillion-line monsters!),
| but I don't really mean it like that, I mean it more as a
| manifestation of Gall's Law [0].
|
| To me, it's all about energy. If the energy is behind "fuck it,
| rewrite it", then that's what you should do. If the energy is
| behind, "preserve the valuable IP in the existing code" then
| that's what you should do. Actually I don't think it's even
| "should"; I think it's "will".
|
| > Would be really nice to hear the symmetrical interview from a
| finger-gunner.
|
| I found the initial framing of like "expediency vs. the right
| thing" not quite right. My read was that Chris just found fault
| with the finger-gunners' bad engineering. Here's a really good
| excerpt:
|
| ---
|
| Why doesn't Code Review just solve this? That was a direct
| quote from Jim. And my answer was, well it didn't, and it won't
| again in the future because people are going to make mistakes
| and just be better. Again, it's not an answer, because what
| happens when it's some junior on the rotation who thinks, yeah,
| that seems reasonable, and this PR was made by a very senior
| SRE?
|
| Why am I going to question whether that value is reasonable?
| Like, yeah, there's a bunch of boxes. That seems fine. Those
| kinds of things happen. And to that question of engineering
| systems, one of the things I think about a lot is, does our
| system only work or does this process only work? Or does this
| tool only succeed?
|
| If I'm acting like a senior engineer, on his or her very best
| day or, or does it work if I'm a super junior engineer who's
| having a bad day and we really want our systems to be workable
| for the latter case. And that helps all of us because
| sometimes, even though I'm a very senior engineer, sometimes I
| have bad days.
|
| Sometimes my brain doesn't feel like it's working at all. Does
| the system still support me on those days or does it punish me?
| I really would like to not be punished.
|
| ---
|
| I'm gonna try to be polite here, but Jim's "why doesn't Code
| Review just solve this" take here is shockingly naive to me. If
| he doesn't know that defects slip by even the best engineers
| with the best code review processes, I have to question his
| experience and judgement. Chris' response of "well it didn't"
| is all that needs to be said: we empirically can't rely on this
| system. Failing to recognize that is malpractice. The length
| Chris goes to to explain why it's malpractice speaks to his
| professionalism and patience, but yeah it's clear he was in a
| situation where someone pretty incompetent was now calling the
| shots.
|
| [0]:
| https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
| sharkbot wrote:
| This might be No True Scotsman, but it doesn't sound like you
| are a "finger gunner" after all.
|
| After (fairly) long time in the industry, it seems to come
| down to:
|
| - understanding what one's clients need (not what they ask
| for)
|
| - understanding what's in the realm of the possible
| (technically, politically, organizationally)
|
| - understanding what are the possible futures (good, great,
| and abject disaster)
|
| If, after answering all those questions, one still advocating
| for a rewrite, you've shown the maturity needed to undertake
| the effort.
|
| "Finger-gunning" is usually related to skipping those steps,
| not the ultimate decision.
|
| (edit for formatting)
| camgunz wrote:
| Fair! Yeah I mean, I will say I don't need a complete plan,
| but unknowns make me nervous, and maybe a true finger-
| gunner never doubts their ability to figure it out.
| ants_everywhere wrote:
| > the next big system will be built the least-experienced.
|
| This sounds plausible and is consistent with my experience, but
| I'm curious if you have a reason for why this is? Is it just
| that the experienced folks are already working on the
| established systems, and we don't know what the next big one
| will be? Or is management doing something intentionally that
| causes this? Or something else?
| sanitycheck wrote:
| Perhaps the experienced folk are "difficult" and have
| "opinions" about the best way to write the new system which
| is somewhat at odds with what management think they they
| need. Difficult opinions may include "no, it doesn't need to
| be 'AI powered' or include a blockchain or be a combinatorial
| explosion of configurability".
|
| The inexperienced folk will be more "cool yeah let's do it".
| You can't blame them, it's how to become experienced.
| chriskrycho wrote:
| For what it's worth I don't think what you're describing here
| is the same thing I meant by "finger guns" at all! What you're
| describing looks to me like good incremental development.
|
| With my "fingers guns mode" phrase, I really had in mind a
| blithe disregard for any of the real-world engineering problems
| and constraints that we knew we would hit (because they just
| inhered in what we knew the solution needed to do). You don't
| have to solve those right up front, but you do need to (a)
| acknowledge that they exist and (b) have some notion of what
| your approach to them will be.
|
| I'm _extremely_ in favor of taking a small team of veterans
| (who are open to doing something novel!) on a new system, as
| you say--let them leverage all their expertise, while being
| careful to avoid Second System Syndrome, to build something by
| _using_ all of the knowledge about those pitfalls
| /problems/etc. to avoid them. (The risk there is that you can
| end up overfitting on previous experience! That's real!)
| cbb330 wrote:
| I work at LinkedIn now. Chris's role and the podcast describes
| ember and front end web dev. I think the LoC and build he's
| referring to could be voyager-web. Our monolith flagship web app.
| And, there's more systems at LinkedIn with MM lines of code and
| long builds example is the mid tiers, the offline data stacks,
| the metrics systems, KafkaKafkaKafka.
|
| Unfortunately, 17min to build is pretty good. 17min without a
| transient infra failure is very good.
|
| Anyways, AMA
| hasty_pudding wrote:
| I worked at LinkedIn in infra. Their internal tools are a
| nightmare. The true definition of Jugaad.
|
| The entire company has zero concept of testing. No QA at all.
| Engineers push out half baked initiatives to add to their
| promotion packet then move on to the next thing.
|
| I trouble shot so many issues just in day to day usage of the
| internal tooling like for some reason, engineers just trying to
| do their jobs, are QA.
| cbb330 wrote:
| TIL jugaad haha.
|
| Yea, give everything 3x estimate because everything that can
| break will break along the way.
|
| Fresh clone of master won't build, the local build command is
| broken, gradle, remote build, GitHub, staging is an inside
| joke, prod host os upgrades, dependencies bumped in repo,
| http dependency's network route changed, etc. etc. etc.
| makingstuffs wrote:
| Seeing you use 'Jugaad' make me unreasonably happy.
|
| I feel like this sort of flinging shit over the wall
| mentality is very much becoming de-facto nowadays. Very often
| I have been required to just 'get shit done quick and dirty'
| over spending time to come up with a permanent solution.
|
| Quick and dirty is never the short term fix it is meant to
| be. It always ends up being left in place until it inevitably
| breaks down the line.
| rrr_oh_man wrote:
| But are things ever permanent? Isn't it to be expected that
| stuff can change in the future?
| hasty_pudding wrote:
| LinkedIn thinks so. Their tech stack is so over
| engineered and spaghetti that they spent a hundred
| million and years trying to move into Azure and were
| beaten by how bloated and unmovable the tech stack is.
|
| Theyre still using python 2 and centOS in tons of
| systems.
|
| They just started using github last year.
|
| Their tech stack is easily over a decade old.
| Domenic_S wrote:
| > _Theyre still using python 2 and centOS in tons of
| systems._
|
| There's 1 service on py2, with a plan to migrate _because
| of_ the move off of centos. centos deprecation is in
| progress.
| hasty_pudding wrote:
| > There's 1 service on py2
|
| Are you sure about that?
|
| CentOs migration has been in progress for years lol
| dustingetz wrote:
| jugaad: https://en.m.wikipedia.org/wiki/Jugaad
| yellow_lead wrote:
| In my first SWE job, I was perplexed, I never felt much
| impact for solving complex bugs, even if they had side
| effects like reducing latency for everyone in our infra.
| Anyways, I realized it's like you say - you get rewarded for
| pushing features and shipping stuff. This incentive can
| easily become a 'tragedy of the commons' situation.
| pm90 wrote:
| Why did the codebase end up like this? Is it a lack of hiring
| for platform or developer tooling teams?
| lvspiff wrote:
| https://xkcd.com/927/
| z3dd wrote:
| I bet in most places it all starts with a new innovative
| system of (self) assessment that relies on number of
| features/VI/KPI tied to promotions.
| j-krieger wrote:
| It's because LinkedIn is _old_ in terms of web-age. Really
| old. They, along with thousands of other companies, had the
| need for a reactive web front-end when JavaScript had not yet
| matured to where it is now.
| cbb330 wrote:
| this plus lack of performance eval/culture/strategy to grow
| responsibly and enable future success
|
| its a great learning experience though despite what people
| say about the inability to learn the new hot hot tech. the
| nuance of software development is the decisions that other
| people make, that is inescapable and is a skill worth
| developing. i'm not buying the "just do a startup" because
| i think its a cop out.
| marcinzm wrote:
| In my experience, a good platform team will increase velocity
| a bit while a bad platform team will tank it. Perpetual
| migrations and re-writes, "simplified" platforms that fail to
| meet future developer needs, half-abandoned NIH systems as
| people move teams, ownership moats if product teams try to
| touch platform code, etc, etc. So if a company has been
| burned a couple times they may decide to just not bother.
| pcloadletter_ wrote:
| This sounds like my exact experience at another big tech
| company
| Aeolun wrote:
| > Unfortunately, 17min to build is pretty good.
|
| My biggest pet peeve with this is that a lot of people see
| these values as immutable, and because building/testing/running
| every single push takes too long, we should not do that.
|
| As opposed to making builds faster, or build infra
| faster/cheaper.
| zaphar wrote:
| There are mostly only a few ways to make builds faster.
|
| * Ship less code (Very hard to get a large org to do)
|
| * Rebuild the same things less often. Requires using build
| tooling that supports this org wide, Still hard to do but not
| as hard as shipping less code.
|
| * Build more pieces in parallel. Also requires using build
| tooling that supports it org wide as well as structuring the
| code in such a way that it is amenable to parallel building.
|
| These are all expensive investments that will pay off huge if
| you do them but can be quite difficult to justify until you
| reach a certain size.
| Aeolun wrote:
| These kind of speedups at LinkedIn scale all fall off the
| chart at the bottom left of this scale:
|
| https://xkcd.com/1205/
|
| Nonetheless, leadership will never give it priority.
| recroad wrote:
| Can you expand on "ship less code"? I think I'm agreeing
| with you but I'm taking it to mean ship smaller pieces of
| code, not overall LoC.
| zaphar wrote:
| I mean both. Smaller pieces but also less of those
| pieces. It is my experience that many systems code size
| is less purely a function of what they need to do than a
| complex set of layered components and systems in the name
| of ease and fear of NIH syndrome.
|
| Sometimes doing the same thing with simpler components
| and fewer components is just better.
| cbb330 wrote:
| The top 1/3rd of Eng don't see it as immutable. But the stack
| can consist of 10 other teams tools, and you're not rewarded
| for the amount of time you're spending to fix other people's
| stuff. So it goes on
| AndrewKemendo wrote:
| >Anyways, AMA
|
| What precisely is LinkedIn trying to be?
|
| Seems like it's turning into 2007 Facebook. Is that
| intentional?
| duxup wrote:
| I wonder, is "being facebook" is that an organic goal
| LinkedIn came up with, or did they pick that up from their
| users?
|
| I suspect the user base has largely driven it. From the
| beginning it seemed that regardless of the stated purpose
| users wanted to use LinkedIn as a "different facebook".
| Personally I hate that, but a lot of people have been doing
| it for many years.
| disgruntledphd2 wrote:
| Nah, basically every other social type thing got screwed by
| FB convincing everyone that MAU's (i.e. the metric FB
| looked best on) was the right metric, when it clearly
| wasn't.
|
| Like, LinkedIn only needs MAUs who are trying to sell
| something or looking for a job, they 100% don't need people
| to log in every day (as their ads business is only a small
| proportion of revenue).
| duxup wrote:
| I don't buy entirely into "FB did it" kinda thing. They
| found it ... but users LOVE it. That's a choice in the
| last part.
| digging wrote:
| Some users love it, the type who want to sell their every
| thought online to anyone who can tolerate them.
|
| Other users (almost everyone I know) absolutely loathe
| it. It is hands-down the worst service I have an account
| with, but it's also practically required to get a job if
| you don't have lucky personal connections. I was hoping
| TFA was actually about leaving as a user, that it might
| be inspiration for me to free myself.
| disgruntledphd2 wrote:
| They made loads of product changes to get people to
| engage in that way. As an example, showing who viewed
| your profile in the email vs needing to log in.
| bombcar wrote:
| I've always thought LinkedIn was just Facebook for
| Business(tm) - has it ever been anything else?
| duxup wrote:
| It was always to some extent, but the feed was a lot more
| formal IMO at the beginning. Content was more like actual
| articles and links.
|
| But I think users just love / gravitate towards a more
| facebook style feed.
| bombcar wrote:
| Part of that might be the "reward for spamming" - any
| "social media" site is always amazing at the beginning,
| because only the die-hards know about it and the spammers
| have no return on investment.
|
| LinkedIn has slightly different style of spammer, but
| it's basically the end game for anything.
| digging wrote:
| > I suspect the user base has largely driven it.
|
| In what way? Users/customers do not generally drive
| development. Indirect measurements of users do, such as
| measuring "engagement".
|
| At _best_ , what likely happened was A/B testing showed
| "what users want", which rarely and only by chance
| intersected with what users actually want, and instead
| showed over and over that socials media patterns (light and
| dark) hijacking attention drives engagement.
| duxup wrote:
| Oh I disagree greatly.
|
| If people didn't want to use twitter, they wouldn't and
| it would be gone. But users do use it, and even the folks
| who tell me they're upset ABOUT Twitter, most choose to
| be there.
|
| Whatever the reason they make that choice, at some point
| that's on them.
| digging wrote:
| If the choice is "use the platform everyone else is on"
| vs "don't exist", many people will choose the former.
|
| That's not the same thing as wanting to use the platform.
| duxup wrote:
| If you feel like not being on twitter means "don't exist"
| as an individual... I feel like that also is a choice /
| lifestyle you chose.
|
| I can't imagine letting a service like that define things
| for me like that.
| digging wrote:
| > I can't imagine letting a service like that define
| things for me like that.
|
| Yes, that's very clear. Everyone is super impressed with
| how free your thinking is.
|
| However, you completely missed the point of my comment by
| interpreting my hyperbole so literally.
| cameldrv wrote:
| They're trying to be a site that makes a lot of money from
| ads. To do that they need to get people to spend a lot of
| time there. To do that they need an infinitely scrolling feed
| that has addictive characteristics. What else are people
| going to spend a lot of time on there? Scrolling through
| their contacts?
|
| Oh, you meant what are they trying to be that's helpful to
| the user? Doesn't matter.
| cbb330 wrote:
| Yes. We hired everyone from meta starting in 2020 and so far
| this strategy is holding true
| AndrewKemendo wrote:
| Interesting ok. Good luck
| yellow_lead wrote:
| Whats part two of the story? Did LinkedIn move to React? Did
| the initiative fail, or is it ongoing?
| cbb330 wrote:
| Preface: I'm not in front end or flagship teams so I'm saying
| hearsay.
|
| The initiative Chris refers to is still alive but likely
| hasn't made any meaningful progress. These things tend to
| have a lot of fan fair and then suddenly leave the front
| conscious of the company as we get a new GDPR/DMA fire to put
| out. at that point it will be dead. Or, it will stay alive
| for 2+ years as slowly but surely 50% of clients migrate to
| it and see even poorer performance.
|
| There was a blog here a few months ago about migration from
| restli to grpc. It's still going on, and nearly every app i
| see is still restli.
| throwaway2day03 wrote:
| dude, you are sharing way too much about us publicly. the
| status of restli -> grpc and talking about DMA? seriously?
|
| you realize stylometry exists, right? even the phrase 'fan
| fair' is a strong signal.
| smsx wrote:
| Sounded like he he was talking about a public blog post
| right?
| cbb330 wrote:
| https://www.infoq.com/news/2023/07/linkedin-protocol-
| buffers...
|
| https://commission.europa.eu/strategy-and-
| policy/priorities-...
|
| https://gdpr-info.eu/
| MisterPea wrote:
| Just for another perspective, I've worked at a few companies
| including LinkedIn as a backend developer and I think LinkedIn
| is probably around 70-80th percentile in code quality.
|
| There was significant emphasis on code quality at least on the
| team I was on, and an ever improving culture.
|
| I did work on one voyager task though and I remember it being a
| nightmare
| gedy wrote:
| This sounds like one of those "soft leadership" roles, that
| almost always suck because you're "responsible" for something,
| but have little or no authority over the rest of the org. The
| real tech leadership (if any) are MIA or no longer in touch with
| the actual problems, even if they've been their forever or are
| the "system experts"
|
| Been there, done that, no thanks.
| swader999 wrote:
| Really feels like mission impossible to do this kind of role
| remotely.
| snotrockets wrote:
| It's very possible, and I've seen a lot of people do it
| successfully.
|
| It does require some organizational changes, and a slightly
| different set of skills, and some fail to realize they lack
| both.
| AdrianB1 wrote:
| Boeing in action. Luckily nobody is flying LinkedIn MAX, so there
| are no deaths as a result.
| callamdelaney wrote:
| It's no surprise that LinkedIn's codebase is a mess if you try to
| use the website, it's a complete nightmare. Say I see an
| interesting post, I click on the person who wrote it to learn
| more about them, then I press back to go back to the post and
| continue reading it - now the feed refreshed and I've lost the
| post.
|
| Say I want to scroll back through messages I've received, I do
| that, the entire webpage starts lagging and takes 10-15 seconds
| to register inputs.
|
| Why do I get 30 fake notifications? They are literally not real,
| made up rubbish to force interactions from people - it's
| disingenuous. The recommendation algorithm is also completely
| terrible.
| pm90 wrote:
| The notifications has long been totally useless.
| dade_ wrote:
| considering they are owned my Microsoft, their windows app app
| is a technical marvel. It's a crippled buggy version of their
| already too buggy web app. I understand why it was made, but I
| don't understand why it was released. It doesn't look good for
| either brand.
| purist33 wrote:
| The thing about the notifications though, is you can turn off
| every single one of them in the settings.
|
| When I tried to turn it off though, I was hit with 100s of
| different types of notifications. I generally like it when
| apps/sites do this. This way I can turn off the garbage i dont
| need, and keep the ones I want. But 90% of thosr categories
| were garbage. It is really shocking that one can take a good
| idea this far and make it frustating.
| nottorp wrote:
| > Note: This podcast is designed to be heard. If you are able, we
| strongly encourage you to listen to the audio, which includes
| emphasis that's not on the page
|
| Seriously? You getting more "engagement" because someone is a
| good actor?
| raziel2p wrote:
| what?
| pell wrote:
| The introduction sounds very GPTy:
|
| > His journey was marked by challenges: from the nuances of
| remote work to the struggle of influencing company culture, and a
| critical incident that put his principles to the test against the
| company's push for speed.
|
| >Chris's story highlights the tension between the need for
| innovation and the importance of project health. This all led
| Chris to a pivotal decision: to stay and compromise his beliefs
| or to leave in pursuit of work that aligned with his principles.
|
| >He chose the latter. Join us as we dive into Chris's compelling
| story, exploring the challenges of advocating for principled
| engineering in a world that often prioritizes quick wins over
| long-term value.
| CoastalCoder wrote:
| FWIW, it seems pretty consistent with the style of Adam's other
| intros.
| padjo wrote:
| No shade on ember as a framework but LinkedIn's decision to use
| Ember always baffled me. As far as I remember they switched to it
| long after it was clear react had "won".
| anytime5704 wrote:
| Some internal stakeholder surely fought for it so they could
| get promoted.
|
| Happy for that person, wherever they are now.
| chriskrycho wrote:
| LinkedIn actually chose Ember well before React had won, and
| definitely before it had become the new "no one gets fired for
| buying IBM." That all preceded my time there, but the
| evaluation was late 2014 and the LinkedIn.com web app rebuild
| in Ember was kicked off in 2015 if I recall correctly. React
| only came out in 2013, and while it was definitely on a rocket
| ship it was way less obvious in late 2014 that it was going to
| be the _de facto_ standard that it was by late 2016-early 2017.
|
| Bonus note: One factor that was _really_ not obvious at the
| point they were doing the evaluation was how much TypeScript
| and TypeScript-powered tooling was going to matter. TS shipped
| support for JSX /TSX in TS 1.6, in late 2015, and I contend
| that while JSX had a lot of pros and cons, the one absolutely
| unambiguous win it had over _everything_ else from that point
| forward was that whether you were writing JS or TS, you got
| outstanding editor support in basically every editor in the
| world courtesy of the TS language server backing your JS or TS
| development. Just absolutely massive.
|
| The fact that the TS team baked support for JSX into TS itself
| and that it's not pluggable is (a) totally understandable and
| reasonable from a maintenance effort POV, especially given that
| Microsoft uses React extensively, and (b) extremely frustrating
| for everyone not using JSX. Vue, Angular, Svelte, and Ember all
| had to build _the exact same_ infrastructure to get that
| support, and that was a huge productivity hit for devs using
| those frameworks compared to React in the meantime.
| ghostoftiber wrote:
| But I bet he still has his resume on LinkedIn. :)
| acqbu wrote:
| LinkedIn comes across as a not-so-great company to work for. I
| find their product to be very toxic, off-putting and full of life
| coaches and scammers.
| pier25 wrote:
| LinkedIn is so awful to use.
|
| There's value in the connections you can make and hiring or
| finding a job... But it's probably one of the worst web apps I
| use.
|
| I'm surprised it ever got this big.
| compacct27 wrote:
| The big rewrite. Risky even on manageable codebases, and the
| leftovers never seem to fully go away. Who wants to score points
| on re-writing a tucked-away settings page several years into
| these?
|
| I've seen so many attempts at these you'd think we'd have a
| framework for rewriting codebases. We don't. Automated codemods
| require a consistency that few adhered to. The code patterns
| evolved so much over time, it's like looking at tree rings.
|
| We're basically putting code in boxes, re-arranging the boxes,
| and rightfully saying some arrangements are more efficient. How
| come we haven't found a better way? Automations operate at the
| code level, but not at the box level.
| jll29 wrote:
| MM lines of JavaScript! That is bloat incarnate.
|
| I've been thinking of re-implementing something like LI, or
| rather implement my own contact database (without the "FB"
| features, please).
|
| The only problem would be how to make my contacts migrate over in
| bulk.
|
| Apart from the bloat, the main problem of Microsoft LinkedIn is
| that it does not let you export your contacts' infos, which
| really is a must-have feature of a contact platform.
| miohtama wrote:
| LinkedIn was originally built in Ruby, containing 60k lines of
| code
|
| https://queue.acm.org/detail.cfm?id=2567673
|
| Summary: https://www.pixelstech.net/article/1395463142-Why-
| does-Linke...
|
| LinkedIn migrated to Node early 2010.
| jspash wrote:
| So they claim the change to node reduces the 60k lines of
| Ruby to only 2k of node. So how did they manage to add
| 1,998,000 more lines of node in the next 10 years and not
| think something was amiss?
| Solvency wrote:
| Developers love to armchair and then you put a bunch of
| them together and you inevitably end up with soaking wet
| bloated festering codebases like LinkedIn
| foofie wrote:
| > So how did they manage to add 1,998,000 more lines of
| node in the next 10 years (...)
|
| Does LinkedIn have the same features it had a decade ago?
|
| Search is far more powerful. It also shifted to add an
| entirely new social media dimension to it.
|
| God knows what features were made available to recruiters
| and marketers.
|
| The hard part of a web app is not the stuff you see; it's
| the stuff you don't even know is there.
| _mayo wrote:
| This is not entirely true. These articles are referring to
| the mobile services. The majority of LIs services are built
| in Java (with some Node, Python, Scala sprinkled in).
|
| Here's a write up from the LI engineering blog briefly
| detailing a history of their architecture (up to 2015):
| https://engineering.linkedin.com/architecture/brief-
| history-...
| soarerz wrote:
| > Apart from the bloat, the main problem of Microsoft LinkedIn
| is that it does not let you export your contacts' infos, which
| really is a must-have feature of a contact platform.
|
| Platform lock in is certainly intended (even though it sucks
| for users)
| game_the0ry wrote:
| Call me crazy but I did not think that was a surprise. I worked
| at big bank that had consumer-facing JS web app with 6MM lines
| of code, though now I am starting to think that that number is
| false, given the feedback in this thread.
| rm_-rf_slash wrote:
| Have you tried grabbing JSON directly from the web server
| responses to the browser to get your contacts?
|
| I haven't tried this on LI but it's my dirty trick for
| exporting conference attendee lists when they're made public on
| websites. YMMV.
| bgirard wrote:
| > Okay. Well, I can keep butting my head against this wall. I had
| conversations with that manager where I was told in literally so
| many words, 'You're too idealistic. You don't care enough about
| the bottom line. You should change your values.' And I was like,
| no, nope, that's not how this is going to work, man. Uh, outside
| I did the politic.
|
| To me that's the most interesting quote of the podcast and that's
| the impression I had gotten before reading this quote. Sounds
| like they received valuable feedback along the way and ignored it
| on purpose.
|
| My career lesson is that being right isn't the hard part of being
| a senior staff engineer. Being right isn't enough. It's building
| alignment across the entire org towards the right solution.
| That's the hard valuable problem that I'm paid to solve.
|
| I worked on the facebook.com rewrite to React in 2019. I find
| this story particularly interesting.
| atomicnumber3 wrote:
| I mean, yes and no. I find staff engineers, and especially the
| senior-staff-engineer-type positions, to be the ones most
| sensitive to the exact specifics of the org they've joined.
|
| I've seen a senior staff engineer (at a popular unicorn you've
| all heard of) who was a very smart, but also friendly and
| reasonable person (how often are those traits in opposition of
| each other!). They were very good at the staff engineer game.
| They were vouching for us to upgrade a framework that we pumped
| 50M of spend through per year. From v2 to v3. It wasn't a huge
| change, not like python 2 to 3 at all. Positively trivial
| compared to py2 to 3. But management just were not up for it at
| all. Even with research done showing we could expect literally
| a baseline of 10% improvement in performance (so you do the
| math on that vs the 50M of spend), they just did not want to
| "waste time on version upgrades." The staff engineer was told
| they were being unreasonable in pushing so hard. Their manager
| tried to ditch them, but again, they're good at the staff eng
| game, so they switched teams due to being in the good graces
| (due to an excellent track record) with a middle manager above
| them. They decided, hey fuck you, I'm going to one man army
| this effort. They did, they had preview versions ready in less
| than a month and had migrated some of the bigger-win jobs
| within 2, saving their annual comp (which was prodigious)
| several times over just in that. Suddenly everyone was keen on
| moving their stuff over now that the startup cost (both
| political and eng-time) was paid.
|
| Anyway fast forward a year, the rollout is complete and nearly
| the entire fucking management chain above him has turned over,
| a 50/50ish mix of firings vs quittings. But he's still there
| and the upgrade migration is complete.
|
| So sure, sometimes staff engineers have a stick up their bum
| and can't dislodge it enough to get work done on the bottom
| line. But sometimes they're the only sane ones in a crazy
| world, and you can only do so much to try to turn a balky mule
| around.
| slimsag wrote:
| One way to read your story is that upper management applied
| heavy downward pressure, to the extent that relatively
| trivial changes in the organization were discussed as
| 'costly' - in an effort to get engineers like the senior
| staff engineer in your example to pull heroic feats of
| engineering in a few weeks all by himself, instead of using a
| small group of engineers / resources.
|
| I think your argument is that 'it worked for him' because he
| is still there while the managers are not, but reality may be
| the managers were there to have that effect in the first
| place and the 50/50 turnover was because they were in the
| thick of it.
|
| That'd be my cynical take, anyway.
| ytx wrote:
| Especially if he wasn't rewarded commensurately for that
| feat.
| atomicnumber3 wrote:
| "upper management applied heavy downward pressure"
|
| Correct
|
| "I'm an effort to get engineers like the senior staff
| engineer to pull heroic feats"
|
| I think this gives them too much credit. The reality, imo,
| is that the org had a pathological emphasis on "impact" and
| ran performance reviews so tight that everyone, managers
| included, were so scared of being PIPd and losing their
| grants that they were extraordinarily risk averse in
| planning and picking projects to bet on. And leadership in
| general undervalued "infra-y" changes as mostly make-work.
|
| The turnover was partly because of this general effect -
| people scared, not betting big or smart enough, and
| optimizing for holding on / not getting fired. And to their
| credit, most of these people vested 2-3 years of their
| 4-year grant before it all fell over, so they made out
| decently lucratively, so maybe they're the real winners in
| all this after all.
| slimsag wrote:
| That's an interesting take, thanks for writing that. That
| gives me something to think about.
| fmbb wrote:
| I'm not American so I don't know what a staff engineer does,
| and googling the term is not helping much. I have also not
| worked in any software company that has "engineering teams"
| which would maybe have helped me understand the Google
| results.
|
| What you are saying is that sometimes a senior engineer
| manages to make good things happen?
|
| Did this person get rewarded for this huge effort they put
| in? You say they already have prodigious yearly compensation,
| was it at least doubled given that they saved it several
| times over?
| mjr00 wrote:
| > Did this person get rewarded for this huge effort they
| put in? You say they already have prodigious yearly
| compensation, was it at least doubled given that they saved
| it several times over?
|
| When you're getting paid $500k-$1m/year or more as a staff
| engineer, putting in huge effort and getting organization-
| wide impactful results is part of the job description. I'm
| sure it had a good effect on their yearly comp review, but
| suggesting that their comp should be doubled because they
| did their job is silly.
| atomicnumber3 wrote:
| Staff engineers (in the FAANG parlance) are, generally
| speaking, ICs who were previously senior engineers who have
| proven themselves enough that they're trusted to be roughly
| autonomous, they're kind of "peers" to engineering managers
| in a way. While a senior engineer will typically not have
| the political capital move big politically-heavy rocks, it
| is typically down in the staff engineer's job description
| that they are expected to. It's basically shifting from a
| focus on code and building systems to people/the org and
| architecting systems that fit the org.
|
| "Did the person get rewarded for this huge effort" Money-
| wise? Haha. No. Only in the sense that they further
| solidified their soft power as someone you shouldn't bet
| against. And of course the sense of pride that comes with
| shipping something (that further enriches some capitalist
| and maybe yourself to 0.0013% of the effect, since that's
| how much of the company your grant commands shares of).
| rvba wrote:
| > staff engineers are ICs
|
| Wtf is an IC? Internal consultant? Individual
| contributors (what does that even mean? One person team?)
|
| Using abbreviations to explain something is a bad way to
| explain...
| bluGill wrote:
| A staff engineer is a level above senior. What they do is
| make them themselves more valuable than the regular
| engineer who has been around for 10+ years. As you get
| experience there is diminishing returns, so most engineers
| will not make a staff level (though there is plenty of
| title inflation and so you will find many). A staff
| engineer needs to know the code (or whatever they engineer)
| well, but they don't normally write a lot of code. They
| tend to spend more time thinking about the larger problems
| (including forcing through upgrades nobody wants to pay
| for), mentoring juniors, and figuring out the hard
| architecture problems (software architects have a role, but
| in my experience they rarely actually create architecture
| despite the name - possibly because what developers need of
| architecture is different from management?).
|
| Staff level developers are trusted to figure out what needs
| to be done without direction. When they are given direction
| it is figure out how the other engineers break this problem
| up and do it - typically not do it themselves.
|
| If staff engineers are doing something it is not important
| to any project. So nobody feels bad about interrupting them
| if they need help or something urgent comes up. (this also
| means you are developing your senior developers into staff
| engineers by giving them responsibility)
| juliusdavies wrote:
| This reminds me of the blog post, "People can read their
| manager's mind," by Yossi Kreinin.
|
| https://yosefk.com/blog/people-can-read-their-managers-
| mind....
|
| In particular:
|
| "Corollary 1. Who can, and sometimes does, un-rot the fish
| from the bottom? An insane employee. Someone who finds the
| forks, crashes, etc. a personal offense, and will repeatedly
| risk annoying management by fighting to stop these things."
| skeeter2020 wrote:
| you gotta a do what you gotta do, but if you have the luxury,
| and conceptual integrity, there's going to be sacrifices in the
| name of "alignment" that you're just not prepared to make. I
| agree with your positioning on lots of "right" solutions, IME
| it's recognizing and understanding and still not accepting when
| it violates your foundational values. I've been there and had
| the ability to not participate, but this isn't always the case.
| bgirard wrote:
| You're exactly right and to me that's part of finding the
| 'right' solution for the organization. Sometimes you
| compromise for 'alignment', and heck sometimes you discover
| that those compromises were a better solution all along.
|
| In other cases you chose to 'pick a battle' and find the best
| way to navigate and get buy in. Because again being right
| -and getting buy in- is the important part. Not later giving
| everyone an 'I told you so'.
| choppaface wrote:
| Separate Church and Business. Get paid while making a
| meaningful contribution. Use the time outside of work to live
| the values you do not otherwise monetize. Critically: Have
| Time Outside of Work.
| samstave wrote:
| Have you ever worked with a BOFH?
|
| BOFHs suck.
|
| Your career lesson is correct.
| wizzwizz4 wrote:
| I have never worked with a BOFH. You can tell this because I
| am still alive, have not been convicted for somebody else's
| fraud, and have never been defenestrated.
|
| Doing the _right_ thing isn 't a known characteristic of a
| BOFH.
| chriskrycho wrote:
| There's an element of truth to this! I alluded to this (and
| Adam totally reasonably cut even more of it for time) but I
| definitely was not as effective organizationally as I could
| have been; that is one of the major things I learned from my
| time at LinkedIn. The biggest challenge for DX teams remains
| figuring out how to (a) communicate but more importantly (b)
| _align_ their work with the key priorities of the business. I
| got to be okay at (a) but never particularly succeeded at (b)
| in my time at LinkedIn. Some of that is on me! Some of it... is
| on LinkedIn.
|
| That said, in this case, the person telling me "You're too
| idealistic" literally meant it in the sense of "You should not
| care about _anything_ unless it directly moves the bottom
| line," and I reject that to my bones. The bottom line matters.
| So, though, do things like user experience, developer
| experience, and for that matter just basic ethics about what we
| build (though happily the latter wasn't in view here).
| bgirard wrote:
| > "You should not care about anything unless it directly
| moves the bottom line," and I reject that to my bones
|
| I have a lot of empathy for that feeling. I've come across
| this often in my career and I have not enjoyed it. I enjoy
| building software that I'm proud off. Thank you for sharing
| your story!
| ejb999 wrote:
| >>My career lesson is that being right isn't the hard part of
| being a senior staff engineer.
|
| Not only that, but your own definition of 'right' may or may
| not align with what is 'right' for the people paying your
| salary - pretending otherwise is just silly.
|
| In any organization, you advocate for what you believe is
| 'right' the best you can, someone else (or a consensus of
| others) will likely decide if they agree - you get to decide if
| you are willing to live with it/compromise, or move on - I have
| done both in my career.
| choppaface wrote:
| A great choice quote, and add to that LinkedIn as an
| engineering culture already heavily skews towards
| "craftsmanship" (as they call it) and refinement versus the
| larger industry. The speaker appears ridiculously unaware of
| how outlier his position is.
|
| Do you want to wax the wheels of a title-winning race car or of
| a 1990 Toyota Camry that happens to be in mint condition? A lot
| of LinkedIn engineers I've met are the latter, and Microsoft is
| trying to push them into being the former. If you are The One
| True Master Wheel Polisher then sure maybe you get to choose
| your car, but don't soapbox it. The audience is smarter than
| that (I hope).
| queuebert wrote:
| Since when is LinkedIn is fast paced? It hasn't substantially
| changed in ten years.
| herbturbo wrote:
| The fast pace comes from product deciding there should be 5
| different types of "like" button and goofy AI "takeaways"
| functionality.
|
| LinkedIn is a solved problem but you can't have that any more.
| margorczynski wrote:
| Probably a lot of work that is being done there is inventing
| useless stuff that makes people, projects and departments
| look needed.
|
| What happened to Twitter shows that in most older orgs with a
| well established product you can throw out ~50% of the people
| and it still will go on.
| mardifoufs wrote:
| LinkedIn learning is pretty new, massively pushed for too. And
| most of the new features are on the recruitment side I think
| uptownfunk wrote:
| In my career I have learned to listen more carefully to
| engineers. Sometimes it is not about saying yes and no to them
| but listening to what they are trying to say. Sometimes if you
| ask a few questions they learned something about the business
| that makes their case come apart. Sometimes if you listen to them
| you find a good idea and it opens up a whole new area of the
| product. So I think it's a false dichotomy and you have to be
| good at listening and asking the right questions to get your
| point across.
| sonicanatidae wrote:
| I have no idea why anyone is still a user on LinkedIn. They've
| had numerous breaches, and give 2 entire shits about it. If your
| account gets stolen, even with MFA enabled, they will take
| roughly 2+ weeks to respond to your ticket and they require that
| you send in even more information that they will callously
| handle.
|
| My advice, leave LinkedIn.
|
| Edit: Or, continue to enjoy taking it in the shorts, for an
| entity that could not care less.. lol
| maximinus_thrax wrote:
| Although I do listen to Adam Gordon Bell's podcasts from time to
| time, I'm not sure if I should invest 40something minutes into
| hearing about why some rando left BigCorp. This may be an
| unpopular opinion, but isn't unprofessional to spill the beans
| about your past employment like this? As an external observer and
| hiring manager, I wouldn't touch this person with a 10-foot pole
| if they're in the hiring pool.
|
| I personally find this type of content to be mundane, basic and
| useless. This is the type of stuff most people run into if their
| engineering career spans more than a couple of years, touches
| large organizations, they can't handle conflicts and have the
| luxury of being more affluent than other employees. I don't
| understand why someone would go out of their way to share
| something like this.
| chriskrycho wrote:
| This is, indeed the type of stuff most people run into if their
| career spans more than a couple years, touches large
| organizations, etc. That is why I thought it would be useful to
| share: it is the kind of thing that would've been really
| helpful for me in the first few years of my career to
| understand some of the dynamics, including what can go well,
| and what can go less well, and the kind of mistakes one might
| try to avoid, as well as what it means to stick with one's
| convictions, even when that might be costly.
|
| Whether it is "unprofessional" or not is something I thought
| about quite a bit. I think I walked the line of being fairly
| politic and avoiding dragging real people's names through the
| mud, while still getting into some of what that kind of real
| world engineering and associated politics is like. At the same
| time, I can see how someone would take the view that none of
| this kind of thing should be shared outside the company. Your
| mileage may vary. I hope at a minimum you can understand why
| someone might choose to air a little bit of it while trying not
| to be nasty about it.
| jrh3 wrote:
| Can't stay in business swapping dollars. Usually, the ones
| declaring their idealism are going to find a problem with you
| regardless. Doing the right thing is what you do, but that's not
| the sort of attitude that works.
| anoncow wrote:
| That's a great blogcast/plot.
| zellyn wrote:
| I found the "finger guns" team and their plan fascinating, and
| I've definitely seen enthusiasm sweep the field many times.
|
| I work on an infrastructure / foundational engineering team, and
| it reminds of something I'm increasingly believing to be true:
|
| It's much harder to get headcount/funding for a two-year plan to
| build/fix an internal tool than to get headcount/funding for a
| six-month plan to use an external product that ends up taking 3-4
| years.
|
| In this situation, I'd say it's much harder to get
| headcount/funding for a five-year plan to perform a well-
| engineered migration than it is to get approval for a two-year
| whizz-bang plan to create an amazing replacement that will
| actually take 3-6 years and/or actually leave 80% of the legacy
| code migration unfinished basically forever, leading to pure xkcd
| 927.
| zellyn wrote:
| As an addendum, I found the episode fascinating. As someone who
| has spent about five of the last eight years on a foundational
| engineering team, involved in _many_ long, grinding migrations
| where the end goal is clearly a win for the company but not
| particularly useful for product teams in the short term, I
| thought the plans Chris outlined showed remarkable maturity and
| good planning in approach. I also enjoyed hearing a public
| discussion of the nature of that kind of migration.
|
| I would not, myself, choose to talk quite so negatively about a
| former employer and colleagues, but I respect and enjoyed the
| decision of idealistic folks like Chris to be open about the
| struggles entailed!
| pousseelettre wrote:
| In other news, Linkedin is down. Probably not a good thing for
| the bottom line.
| pousseelettre wrote:
| In other news, LinkedIn is down.
| elbear wrote:
| The migration from Ember to React sounds like a good case for a
| technology a previous client of mine created.
|
| He created a language called Unhack which allowed you to do a
| kind of search-and-replace but at the level of the AST. You would
| use this language to specify a pattern in the original code, then
| another pattern that would be the replacement. Unfortunately, I
| don't know if it still exists, as I stopped working with that
| client a couple of years ago.
___________________________________________________________________
(page generated 2024-03-06 23:02 UTC)