[HN Gopher] The Value of Institutional Memory
___________________________________________________________________
The Value of Institutional Memory
Author : leoc
Score : 88 points
Date : 2025-08-11 16:53 UTC (6 hours ago)
(HTM) web link (timharford.com)
(TXT) w3m dump (timharford.com)
| freedomben wrote:
| Apologies for bring in "AI" to a non-AI thread, but I really do
| think that things will be a game changer for institutional
| memory, both at recording it and recovering it. I don't
| personally use them but I have many coworkers that use AI tools
| to join meetings and get summaries/transcriptions aftward that
| they can read or query (also using AI). As people get more used
| to it, I would imagine that sort of thing becomes standard
| practice (regardless of whether or not it _should_ , but that's a
| different topic)
| saulpw wrote:
| Except those summaries are deeply flawed and incorrect, so it's
| like having a secretary with memory loss and possibly dementia.
| a_shovel wrote:
| I've heard this is part of why major infrastructure projects in
| America can be so expensive. A city builds one subway line, and
| everyone working on the project has no experience, so it takes a
| long time and is expensive. The expense convinces people to
| oppose any more projects, so the city doesn't build any public
| transit for a decade(s). By the time they're ready to build
| another line, all the experience has evaporated, so the new line
| takes a long time and is expensive. Repeat forever.
| antisthenes wrote:
| That makes sense. It seems like during the continuous "building
| up America" period of the late 40s through mid 70s there was no
| problem of getting shit done at reasonable cost, because of
| continuously available institutional knowledge.
|
| Once large infrastructure projects become sporadic in nature,
| you begin to run into issues.
|
| The solution has to be continuous stimulus, but that also runs
| into problems of corruption and capture by special interests
| (the longer the stimulus, the more incentive there is for 3rd
| parties to appropriate funds).
| stouset wrote:
| Somehow, other nations have managed to figure this out. Of
| the developed world, seemingly only Americans are resigned to
| the belief that such things are sadly impossible.
| convolvatron wrote:
| the important part of the American system you're not
| addressing is that it makes sure no one accidentally gets
| something they don't really deserve.
| lurk2 wrote:
| It has far more to do with respect for private property
| due to the existence of a class of sophisticated,
| politically literate professionals capable of opposing
| development. Europe and Canada are similar; the extent to
| which this retards the economy is more obvious in Europe.
| It isn't hard to build a road when you can just
| expropriate all the land and completely disregard
| environmental impacts.
| antisthenes wrote:
| "no one" = poor people.
|
| All the old money already got a ton of wealth they didn't
| really deserve (conquest through Native american
| genocide)
| renewiltord wrote:
| That's because we're richer and can object. The Europeans
| get bulldozed by their governments. It's why they're always
| protesting some online ID law or some "show your photo ID
| to browse Wikipedia" shit but no one listens to them.
| stouset wrote:
| Yes, Europeans are completely distraught over their
| (checks notes) functioning public transit systems.
| renewiltord wrote:
| On time some 65% of the time? The only thing that gets
| them to stop complaining is knowing that Americans are
| listening.
| bluGill wrote:
| Robert Moses did a lot of bad that we don't want to repeat.
| We have gone too far the other way but those big projects
| often did come at high cost - but the cost wasn't dollars
| pm215 wrote:
| There's an example of this in railway electrification: if you
| scroll down to the graph in
| https://publications.parliament.uk/pa/cm5801/cmselect/cmtran...
| it shows that the UK tends to do electrification as occasional
| big projects, whereas Germany has consistently done about the
| same mileage every year for decades, presumably with the same
| institutions maintaining their expertise and just moving on to
| the next bit of track. Their costs are a quarter of the UK's...
| renewiltord wrote:
| Yeah, but whatever Germany is doing is obviously wrong
| because 62.5% of their stops have trains arriving within 6
| minutes of target time https://www.dw.com/en/over-a-third-of-
| deutsche-bahn-long-dis... while the UK has 85.9% of their
| stops having trains arriving within 3 minutes of target time
| https://dataportal.orr.gov.uk/statistics/performance/passeng.
| ..
|
| So whatever the Germans are doing with their rail, thank god
| the UK isn't.
| clickety_clack wrote:
| There's strategic bidding as well. Specifications cannot cover
| every conceivable occurrence over the course of a 4 year
| construction project, so contractors can structure their bid to
| be low upfront with big pick ups later for change orders when
| issues arise.
| joe_the_user wrote:
| Such tricks, however, are known. The further trick is that
| those looking at bids can flag gaps or not depending on their
| connections to the bidders.
| clickety_clack wrote:
| You'd be surprised how that game plays out... or maybe you
| wouldn't if you've seen how far over budget public
| construction projects tend to go.
| paulddraper wrote:
| tl;dr Economies of scale
| obeavs wrote:
| Thank you for bringing this up. This is profoundly true for big
| projects (toll roads/transport) and small infra projects (e.g.
| community solar). The length of time that it takes to develop
| things like this, combined with the turnover and the sheer
| amount of context that single developer has to have to be
| successful with it, is one of the driving forces in why
| development is such a difficult/risky business.
|
| It's one of the most consequential problems imaginable to
| solve, particularly as the US begins to realize that we need to
| compete with decades of China's subsidized energy and
| industrialization/manufacturing capacity.
|
| Taking it a level deeper, what most don't realize is that
| infrastructure is an asset class: before someone funds the
| construction of $100M of solar technology, a developer will
| spend 2-5 years developing 15 or so major commercial agreements
| that enable a lender/financier to take comfort that when they
| deploy such a large amount of cash, they'll achieve a target
| yield over 20+ years. Orchestrating these negotiations (with
| multiple "adversaries") into a single, successfully bankable
| project is remarkably difficult and compared to the talent
| needed, very few have the slightest clue how to do this
| successfully.
|
| Our bet at Phosphor is that this is actually solvable by
| combining natural language interfaces with really sophisticated
| version control and programming languages that read like
| english for financial models and legal agreements, which
| enables program verification. This is a really hard technical
| challenge because version control like Git really doesn't work:
| you need to be able to synchronize multiple lenses of change
| sets where each lens/branch is a dynamic document that gets
| negotiated. Dynamically composable change sets all the way
| down.
|
| We are definitely solving this at Phosphor (phosphor.co) and
| we're actively hiring for whoever is interested in working at
| the intersection of HCI, program verification, natural language
| interfaces and distributed systems.
| GarnetFloride wrote:
| Not Just Bikes did a YouTube on Seoul South Korea that brought
| this point up. They've got costs down because they're working
| on it continuously.
|
| As a tech writer people have a lot of experience but they never
| turn it into institutional knowledge because it's never written
| down. Ay best it's tribal knowledge passed by word of mouth.
|
| I know some people refuse to document things because they are
| hoping for job security but that never happens. Or sometimes
| for revenge for getting rid of them. But many companies survive
| despite those efforts.
| toast0 wrote:
| I'm not good at writing documentation, and you can't pay me
| enough to care about it, sorry. I've tried enough times, and
| nobody reads it, or it becomes out of date by the time anyone
| reads it, and I don't see the personal ROI. I'll write notes
| for future me, and put them somewhere others can read it, if
| you don't make that onerous. Otherwise, if you want
| documentation from me, you need to have someone else drag the
| information out of me and write it down. But, I've only
| rarely been in organizations that care enough about
| documentation to do that, so there you go.
|
| There's always a lot of talk about how documentation is
| important, but there's never budget for a tech writer (well,
| you must have found some, as you've taken tech writer as a
| title, but it's not often available) or a documentation
| maintainer.
| solardev wrote:
| It's not a binary thing... even just a few scattered "why
| we did it this way" comments in the code base is a lot
| better than no documentation at all.
| hermitcrab wrote:
| My day job is product developer and I have written hundreds
| of pages of documentation. The key is to write it as you go
| along. Not to wait until the release is ready to go!
| Grosvenor wrote:
| That's going to be my new business - Subways, et cetera.
|
| We just do subways and get good at it.
| leoc wrote:
| Via "Coates" on Bluesky
| https://bsky.app/profile/oddthisday.bsky.social/post/3lvzzmj...
| at at Medum https://mulberryhall.medium.com/odd-this-
| day-5b1cfd1fdb32 who provides some other information:
|
| > What happened next, you may not be surprised to hear, comes in
| different versions. The person who spotted that there might be a
| problem may have been a member of Her Majesty's Constabulary...
|
| >> While they were away, a passing policeman noticed an
| extraordinary whirlpool in the normally placid canal. He also
| noticed that the water level was falling. He rushed off to find
| the dredging gang. By the time they all returned, the canal had
| disappeared. It was then that realisation dawned. Jack and his
| men had pulled out the plug of the canal. One-and-a-half miles of
| waterway had gone down the drain.
|
| > It may have been three anglers who raised the alarm, and given
| that they have names -- Howard Poucher, Graham Boon and Pete
| Moxon -- maybe that version's true. Another telling says it
| wasn't until the evening that
|
| >> local police contacted Stuart Robinson, the British Waterways
| section inspector.
| notahacker wrote:
| Other relevant context: sections of UK canals being
| unintentional drained isn't particularly unusual, although the
| culprit is usually a paddle left open on a lock gate or a leaky
| culvert rather than a plug being pulled. Whether that
| inconveniences anyone for any length of time depends mostly on
| how full the reservoirs at the top end of the canal are...
|
| Wouldn't have been that unusual in 1972 when nearly all the
| canals including that one had ceased commercial operations and
| many of them had been intentionally drained either. I suspect
| the transition from the canal being infrastructure maintained
| by locally-stationed full time professionals to a pleasure
| cruiseway which the new waterways board was willing to devote a
| bit of time to maintaining only after the previous one had
| spent several years trying to get it shut down probably had as
| much impact as the Blitz on the work crew having no idea about
| plugholes...
| RcouF1uZ4gsC wrote:
| This misses something very important.
|
| Institutional memory is not information or documents - it's
| people.
|
| Every single real-world process has implicit knowledge. And you
| can't always capture that knowledge of paper.
|
| But, many corporations seem to want to get rid of their most
| experienced people to save money and have better quarterly
| results for the stock market.
| phkahler wrote:
| Yes, I think people create more internal documentation then
| they read.
| antithesizer wrote:
| It can be documents and it can be people, but it's not
| essentially either one. It can take many forms, including being
| lost when none of those forms has it on offer, as every
| business is different. An institution with excellent
| documentation, mature processes, and adept hiring could retain
| its "memory" without a single human member remaining from the
| past. Oral history and other humanistic forms of memory make
| everyone feel warm and fuzzy, but they're not to be idealized
| as the only real memory simply because they were
| underappreciated for a some time.
| stackbutterflow wrote:
| For instance TSMC is discussed a lot on HN and every time I'm
| thinking that even TSMC itself probably couldn't produce their
| latest chips if they had to start from scratch tomorrow.
| tolerance wrote:
| Perhaps tangentially related Re: "Chesterfield's plug",
| _Chesterton's_ fence came to mind today while mulling over this
| sort of "forgetfulness" (which tends toward outright negligence)
| in my own circles.
|
| https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
|
| Solid writing.
| Pingk wrote:
| This is often made worse as a result of hiring outside
| consultants. Firstly they don't have the institutional knowledge
| you have when starting a project, but they also aren't
| incentivised to properly document and hand over their knowledge
| at the end since that means less future work.
|
| This is why a lot of government projects take so long, they don't
| see the value in keeping an in-house team of trained experts (see
| the difference in train line contruction costs in the UK compared
| to Spain), until you realised how good they were but you can't
| hire them back.
| BJones12 wrote:
| I suspect this is why it's good for the USA to be constantly at
| war. If you're only at war occasionally, you forget how to make
| war and can lose. If you're at war constantly, you'll remember
| how to do it.
| jppope wrote:
| Tragically, there is some truth to this.
| lordnacho wrote:
| Too much emphasis on documentation. It's people that matter.
|
| If you build the sort of culture where people hang around, they
| will occasionally have time to tell each other the internal
| folklore. "When I started, an old guy told me about the plug
| under the canal".
|
| People who work with software know this. Yeah, there are
| documents describing the system. No, reading them does not mean
| you understand the system.
|
| Alas, it's an intangible, and doesn't get counted with the rest
| of the beans.
| antithesizer wrote:
| This is one reason why what ServiceNow does is so important.
| Nevermark wrote:
| I had long term business relationship with a company, originating
| and developing a product for them.
|
| From 50 - 1000 employees things worked very well. There was a
| great deal of continuity in the relationship. Lots of trust and
| flexibility in both directions. Our product quickly became the
| best available, by a long margin, and for a couple decades.
|
| But after they passed about 1500 - 2000 employees they got more
| organized. A formalized organization and process system. Things
| quickly went downhill. As someone working from outside the
| company, their processes were incredibly disruptive and
| inefficient for me. Likewise, their turnover replaced a situation
| of working with long time friendly colleagues, who knew me very
| well, to working with people who had no idea what my positive
| reputation was, my track record of delivering quality without the
| hammer of conformance, etc.
|
| The project's ambitious upward trajectory stalled. Even then it
| took about ten years to fall behind other players. But it never
| recovered. Today it operates deep in the shadows of others.
|
| Virtually every employee I worked with was wonderful, inclined to
| be as supportive as restrictions allowed, etc. But the
| institutionalization smothered the organizations ability to
| operate with any flexibility, no matter how dysfunctional or
| value destroying the results.
|
| The company became like someone who has permanently lost the
| ability to form new memories.
|
| You can't build anything special with someone who keeps
| forgetting any context. I spent many years cycling between
| depression and resurrected determination trying. But finally gave
| up.
| robertlagrant wrote:
| Sorry to hear this. It's such a tricky thing for an org to
| balance, if not impossible.
|
| One thing I notice is it's very easy to add additional layers
| of relatively small actual value that look like lots of value.
| So you might say you've earned a degree of respect by working
| consistently for years, and people don't mind that you don't
| always update your status reports. But then if you don't defend
| vigorously in the org, someone might come in who does very
| little work in terms of company output, but always gets your
| status reports in and reports up the chain so you "don't have
| to". And that looks like value to the person above, but it
| wasn't really. And now you have a new boss.
| throwaway13337 wrote:
| >You can't build anything special with someone who keeps
| forgetting any context. I spent many years cycling between
| depression and resurrected determination trying. But finally
| gave up.
|
| Was that an LLM reference or is it the myopia in me?
|
| There's a parallel here, either way. All the documentation in
| the world will not make a person, or llm session
| interchangable.
|
| In some sense the new way of coding feels like building a big
| org with people without memory. If you can document the process
| perfectly, there is a holy grail out there somewhere.
|
| Or maybe there isn't.
| fuzzfactor wrote:
| That company not only failed to "institutionalize" the
| specialized knowledge they had, once they became big enough for
| bureaucracy to self-assemble they ended up institutionalizing
| the concept of not valuing things that led to initial success.
| paulorlando wrote:
| That story is very Chesterton Fence. If Chesterton was working in
| a canal instead of walking on a country path. There's a balance
| between preserving memory and maintaining and benefitting from
| that knowledge and choosing what not to remember.
|
| When Kurt Cobain shot and killed himself in 1994, his widow went
| on TV to say that what he did was wrong. Cobain's death then did
| not result in others' killing themselves (known as the "Werther
| effect"). Robin William's suicide 20 years later, however, did
| result in more deaths as the story spread widely.
|
| But otherwise, I do agree that we should preserve institutional
| memory and that putting processes over people can lead to
| forgetting.
| stego-tech wrote:
| This is one of the biggest consequences of layoffs in
| corporations. There's this misconception that everything can and
| is "objectively" quantified, and thus layoffs targeting otherwise
| well-performing individuals are being done because this will
| quantifiably save the institution money and resources. Then
| something inevitably happens where someone they previously let go
| could've saved the cost of their employment and then some in
| damages, but the company is often too blind to realize this.
|
| Thing is, I've seen this time and time again. A lot of us have, I
| suspect, seen this story repeatedly in our own current or prior
| organizations. Someone who worked for the company for a decade,
| or who had intricate knowledge of prior M&As, technology stacks,
| codebases, customers, and/or processes who was thrown out as a
| line on a spreadsheet.
|
| I do my best to buck the trend in my work by documenting
| everything (the "bus problem", as I call it) I can and sharing it
| with my colleagues, but the continuous churn of M&As and software
| deprecation means that documentation is often discarded with old
| systems rather than reviewed and preserved, thus further erasing
| any lingering institutional memory.
|
| To be fair, this issue isn't likely to kill a company outright on
| its own. Sure, it could lead to a serious problem and cost gobs
| of money, but it rarely kills a company or project outright in
| the process. Still, it's _preventable harm_ just by keeping some
| additional persons around for knowledge or managing an
| organizational library of content. It 's ultimately such a minor
| cost in the grand scheme of things that shareholders won't really
| care. $1m a year for a corporate library and a handful of staff
| to support it is _peanuts_ on a multi-billion dollar enterprise
| balance sheet, and will almost certainly improve outcomes across
| the organization as a whole.
|
| Or to put it far more simply: institutional memory is the fat on
| an animal. Cutting fat down to the bone leaves the animal weaker
| and vulnerable as a result, as it has no emergency stores of
| energy (or in this case, knowledge) to pull from and thus must
| cannibalize itself in times of crisis.
| Zigurd wrote:
| I call this "process arbitrage." Until the most recent CEO,
| Boeing was ruined by this. They have extremely well documented
| processes. So you can fire the people that understand the
| reasons behind those processes, and the ghost of their
| expertise lingers. For a while.
|
| This starts out looking very attractive because you've cut
| costs and your profitability is up. Then it all suddenly goes
| to shit and your planes crash.
| hermitcrab wrote:
| There is a (possibly apocryphal) story of cars being specified to
| understand a 100kmh air speed on the rear windscreen.
| 'Ridiculous, it can't reverse at more than 30kmh said the car
| designers' and ignored the spec. The first time new cars were
| transported on a train, all the rear windscreens blew in.
|
| A long time ago I worked on a software product to try to record
| design decisions in the creation of long-lived artefacts, such as
| nuclear reactors. The idea being that engineers looking to make a
| change 20+ years later (when the original engineers had retired)
| would understand _why_ something had been designed the way it
| had.
|
| The project was not a success, despite some initial enthusiasm
| from some commercial sponsors. I think this was due to 2 main
| issues:
|
| a) The software infrastructure of the days wasn't really up to
| it. This was just before wikis, intranets etc, which would have
| made everything a lot easier.
|
| b) The engineers working on the design had no incentive to record
| the rationale of their decisions. It was extra work with no
| benefit for them (any benefit was by someone else, years down the
| line). In fact it could make it more likely for them to be held
| liable for a bad decision. And, in an age of cheap outsourcing,
| it could reduce their job security.
|
| The second problem was by far the more important and I don't know
| how you get around it.
| purplezooey wrote:
| It seems that a lot of businesses barely function. They're often
| stuffed with overpaid executives while the actual business
| wheezes along, barely managing to get its product out the door.
| More attention is usually paid to reducing competition,
| increasing one's moat, and restricting supply, so customers have
| little choice, as in the aerospace industry from this article.
| dan-robertson wrote:
| A few thoughts:
|
| 1. Institutional memory does seem important. It feels like lots
| of government things are bad at this - big infrastructure
| projects tend to come in occasional bursts which means each time
| they are learning from scratch; Japan moves lots of civil
| servants around every few years which means that no one really
| remembers how to do things.
|
| 2. I think there is a negative side of this too, a kind of
| 'institutional trauma' where some bad memory can cripple an
| institution. Eg one reason Microsoft lost so much to Google in
| the early Internet was the memory of the late '90s antitrust
| action making them less aggressive. Other companies can have one
| particular close shave which then causes them to focus too much
| on avoiding a repeat, a situation you also see writ small in tech
| teams.
|
| 3. I think a bit about production incidents in tech too here.
| When things are small and the systems are relatively new and they
| break a lot, this may be ok for the business and recovery can
| hopefully be fast because it is possible to quickly hypothesise /
| fix stupid problems. When most silly bugs have been squashed and
| systems are big and reliable, problems can snowball faster, the
| business may be more sad about them happening, people can't
| understand the whole picture well enough to have good ideas, and
| the lower base rate of incidents means people will be more
| stressed or otherwise unable to focus on the actual problem
___________________________________________________________________
(page generated 2025-08-11 23:00 UTC)