[HN Gopher] Theory-building and why employee churn is lethal to ...
___________________________________________________________________
Theory-building and why employee churn is lethal to software
companies
Author : bwestergard
Score : 291 points
Date : 2023-01-10 17:13 UTC (5 hours ago)
(HTM) web link (www.baldurbjarnason.com)
(TXT) w3m dump (www.baldurbjarnason.com)
| darth_avocado wrote:
| Tell that to Elon or Bezos or the other geniuses of our lifetime
| tdumitrescu wrote:
| "Many teams in the industry constantly rewrite parts of their
| code. Not because they keep figuring out better ways of
| approaching the problem but because nobody on the team has an
| accurate mental model for how it works and how the code fits in
| with the rest."
|
| This part rings very true, as someone who's been through plenty
| of rewrites / "make module X not suck" projects. A side
| observation here is that usually the original author (sometimes
| long-departed or departing) understands very well the
| limitations/flaws of the original system/implementation, but can
| be excluded from the design of the replacement for any number of
| misguided reasons. Trying not to bruise the ego of an early
| employee, thinking that they wouldn't agree with any changes,
| whereas those are often the people in the best position to guide
| a rewrite, and often would love the chance to fix some of the
| decisions they had to make originally.
| layer8 wrote:
| This is very true. On multiple occasions, after taking over
| ownership of code for which the original author wasn't
| available anymore, I ended up rewriting it bit by bit,
| according to my progress in understanding it. In some ways,
| rewriting a piece of software is the best way to understand it
| deeply, or to uncover the gaps in one's understanding.
| robertlagrant wrote:
| I agree. I would add that sometimes the existing team only want
| to make incremental changes to their existing software, rather
| than make something new, and that can make team dynamics...
| complicated!
| jimmyhmiller wrote:
| This blog does pretty good job summarizing Naur's points. (My
| only quibble is equating mental models with theory in Naur's
| sense, but its an okay approximation). I definitely recommend
| people read the original paper. It's a flawed but beautiful
| paper.
|
| Shameless self plug:
|
| We just did an episode on Naur's Programming as Theory Building
| in the most recent episode of the Future of Coding Podcast.
|
| https://futureofcoding.org/episodes/061
| kasbah wrote:
| I was going to post this! I really enjoyed this episode (and
| also, all the other episodes of your podcast).
| passwordoops wrote:
| We're having an issue of employee churn, and I think a lot of
| that is 20+ years of tech debt is catching up and new
| developments are becoming extremely difficult.
|
| I was just wondering if it would be good general practice to
| refractor piecemeal every couple of years. This would (in theory)
| reduce tech debt, and if the refactoring is done by relative
| newbies, you ensure the base is understood and stays fresh.
|
| I haven't hashed out or thought this through yet, so happy to
| have holes poked in this idea
| raincom wrote:
| The biggest problem is that software companies don't see
| "software development" as theory-building. Instead, they see
| "software development" as solutions to plumbing problems; that
| way, they can call a plumbing company at 1-800-CLEAN-ME whenever
| new problems emerge.
|
| Even not many people subscribe to Peter Naur's "Programming as
| theory building"[0]
|
| [0] https://pages.cs.wisc.edu/~remzi/Naur.pdf
| commandlinefan wrote:
| > they can call a plumbing company at 1-800-CLEAN-ME
|
| And, in fairness, that is a reasonable thing to want. The
| problem is it's not clear whether or not that's possible, and
| it _is_ clear (but completely ignored by the people who ought
| to be aware of it) that our current software development
| methodologies don 't even come close to allowing it.
| deathanatos wrote:
| > _The problem is it 's not clear whether or not that's
| possible_
|
| It's not.
|
| Attempting to say it is is very akin to wanting a single,
| pre-created bridge to be able to span any valley. But valleys
| come in all shapes and sizes, and so too do the bridges
| required to span them.
|
| Certainly, there are some parts that are the same. I think
| most of us complain about cookie cutter CRUDs or byte
| shuffling, but even in all that, there's generally some
| higher purpose business reason as to _why_ the software is
| doing the byte shuffling that I am, or more deeply, why the
| software is doing the byte shuffling the way it is[2].
|
| And honestly, I think that oversimplifies it, when it comes
| to the problem and the software that attempts to solve it.
| The problem can and often does change in software ... so it's
| akin to a valley that shapeshifts, adds new local laws of
| physics, and requires the bridge to change and adapt. How
| changes to the software mesh with the existing software, and
| why that might be -- is _unique_ to the project. (And, as
| Naur argues, contained only in the heads of the developers,
| and what is in their heads is not even necessarily the same.)
|
| [2] And to extend this with the shifting valley analogy, when
| new byte shufflings come along, would they benefit from this
| program's current design? If I need to extend the design,
| would a given extension work well? Are the requirements
| different enough that I need to consider something separate
| for this new requirement? etc.
| bvirb wrote:
| I have plumbers in my apartment right now fixing a leaky drain
| pipe between two floors. The first thing they did was show me
| why they were going to need to rip out a 10 foot length of
| ceiling and re-do a whole section of pipes to fix the leak
| because the previous plumbers did such a wacky job.
|
| So I guess it doesn't actually work for plumbing problems
| either.
| compiler-guy wrote:
| I have seen half a dozen otherwise useful projects retired or
| criticized because "the original developers are all gone and no
| one understands it any more."
|
| The code worked, and continued to work, and didn't need really
| active development, so the org decided the project's programmers
| should move on to something new, or that the knowledge they have
| isn't especially valuable, and the programmers aren't worth
| keeping happy.
|
| And a couple of years later, no one knows how to update the code
| for a changing outside environment, updated input styles or
| whatever. And all that investment is gone.
| doix wrote:
| But someone can figure it out. It doesn't matter that nobody
| knows anymore, if there's a real need with no alternative,
| someone can figure it out.
|
| I used to work for a company that had an internal tool. It
| wasn't used anymore, everyone that worked on it was long gone,
| the source code was "lost".
|
| There was some emergency and someone needed to use the tool to
| look at some old data. Lots of panic and people claiming it was
| impossible.
|
| I opened up the executable in ollydbg, patched the executable
| so it worked properly, and life went on.
|
| I'm sure the source code wasn't lost, it would have been on a
| tape backup that people were too lazy to try and recover.
|
| If the source code is available then it's even significantly
| easier. I don't buy this whole "nobody knows how to update the
| code" reasoning.
|
| If the code can't be updated, it's a matter of incentives and
| prioritization.
| civilian wrote:
| 100% agree. One of my friends, in her first software
| engineering role, was handed a binder from 1987 and told to
| update some things.
|
| Other coworkers had interacted with the system a little, but
| not extensively. It was a bunch of C for (rock & ore) mining
| software. It was intimidating and it took time, but she
| cracked it, and then happily (& justifiably) bragged that she
| added features to code that had been written before she was
| born!
| AnimalMuppet wrote:
| I'd be proud if I were the original author, too. My code
| lived long enough that someone who wasn't even born when I
| wrote it was able to add features to it.
| charlie0 wrote:
| I think most devs _don 't_ want to figure it out. Let's face
| it, working with legacy code is just not fun unless the code
| is well written (and even then a lot of devs still don't want
| to touch it because it's "old tech" and doesn't help their
| resume).
| laserlight wrote:
| > It doesn't matter that nobody knows anymore, if there's a
| real need with no alternative, someone can figure it out.
|
| Well, by that logic, someone can build it from scratch.
| pasquinelli wrote:
| what was the update? the kind of update that "programming as
| theory building" is talking about is like, the next version
| of a piece of software.
| doix wrote:
| Yes, but the comment I replied to implied that software
| will just stop working. If software has been working for
| years and then stops working, it's usually something fairly
| small than can be hacked quickly.
|
| In my specific case, the tool had a whole bunch of
| hardcoded values. It checked authorization against an LDAP
| server, that all changed. Connected to a sql sever that
| moved + credentials changed. Assumed certain files existed
| on certain network drives which were moved.
|
| Technically, it could have been solved by faking all the
| infrastructure. It also could have been solved by digging
| out the source code from tape drives. But everyone was
| running around saying it was "impossible". I was shocked
| how quickly people tossed out "impossible".
|
| It could have been done with a simple hex editor, but the
| new values were longer than the old values, so the strings
| didn't fit. I put the new strings at the end of the .data
| section and then just changed a bunch of offsets.
|
| I agree that if you're writing the next version of the
| software, you can't do that quickly when nobody knows how
| the current version works. But chances are, you're not
| writing the next version of the software quickly even
| without churn.
| pasquinelli wrote:
| it's not really about _how_ the software works. the
| source code says what it 's doing. the thing is the
| source code is the product of a theory; a theory
| determines how you respond to the world. if that theory
| is lost, then updating--in the long-term sense--becomes
| hard, because the theory that you form might (probably)
| be incompatible with the theory that made the software.
| this incompatibility might make producing functioning
| software difficult, but more likely, the mismatch will
| make choices that don't make sense to, say, users, appear
| to be the obvious thing to do.
| cratermoon wrote:
| > I don't buy this whole "nobody knows how to update the
| code" reasoning
|
| As an anecdote, your example demonstrates the truth of the
| reasoning. Absent extraordinary or unorthodox effort, such as
| patching the executable manually, programs do suffer from
| loss of understanding. Yes, you made it work, but can you
| really claim to understand the program and how it was
| written?
|
| Of course, given enough time and effort, most code _can_ be
| reverse engineered to a level of understanding good enough to
| work on it. In practice, most organizations are going to
| decide it 's better to spend time and money on a replacement.
|
| > I'm sure the source code wasn't lost, it would have been on
| a tape backup that people were too lazy to try and recover.
|
| That's certainly one, rather uncharitable, way to look at it.
| But what constitutes sufficient effort, given time and money
| constraints in the real world? Unless that backup literally
| contained the only copy of information essential to the
| continued existence of the company, there's a limit. And what
| if they did find the backup, but it was on a magnetic medium
| that had degraded to unreadability?
|
| I've made my career working on legacy code. There are
| definitely ways to understand systems even when nobody who
| originally worked on them is around, but business constraints
| have always determined whether or not the effort proceeds or
| is abandoned.
| Existenceblinks wrote:
| > I don't buy this whole "nobody knows how to update the
| code" reasoning
|
| Me too. At best, they would need the previous guys only in
| one condition: very short time constraint (have to figure it
| out "today"). Otherwise, it's not hard to figure out at all,
| even codes from a complete idiot. If he can code the program
| that worked, the code is very likely be figured out by
| experienced people very soon.
| commandlinefan wrote:
| > be figured out by experienced people
|
| Sure, you can figure out how to change it to do some very
| specific thing that somebody identifies as needed. What you
| probably can't do (at least not in any reasonable amount of
| time) is figure out all the things you're going to break in
| making this change. If they followed good software
| development practices, left a good comprehensive set of
| unit tests and documentation then you could do that - but
| they didn't.
|
| > even codes from a complete idiot
|
| If I was wrong, there wouldn't be anything wrong with
| global variables or goto statements - any code that "works"
| would be isomorphic to any other code that "works".
| Existenceblinks wrote:
| > identifies as needed
|
| That's how it works even with the same guys on projects.
| If you think it would break (no tests found), write the
| tests first. I don't believe someone would write a
| complete mess million line of codes that works without
| some tests and docs, that would make him kinda genius.
|
| And global vars and goto is not hard to figure out, it's
| hairy for sure.
| [deleted]
| JonChesterfield wrote:
| My very first commercial program was a Fortran 90 thing that
| could write its own source code out to the filesystem if
| asked. Updated itself as part of the build. I still quite
| like that as a feature.
| fathyb wrote:
| > I opened up the executable in ollydbg, patched the
| executable so it worked properly, and life went on.
|
| That makes you a great engineer that saved the day. Not all
| companies have great engineers that can save the day, or keep
| them, or even identify them.
|
| Sometimes you also need to add features, which can be quite a
| task if you don't have the source code.
| [deleted]
| commandlinefan wrote:
| > But someone can figure it out
|
| Actually, in most cases - no, they can't. Or, rather, they
| can't figure it out as fast as they're "needed" to.
|
| > if there's a real need with no alternative
|
| What happens instead is that somebody with no real knowledge
| of software development says, "we need this change made in
| two weeks. We _need_ this. " That this might be impossible
| for any human being, living or dead, is completely ignored.
| If the programmer this is dumped on even suggests making any
| clarifying improvements to the code, even _that_ is rejected
| because "we need" this.
|
| And two weeks come and go, something was changed because
| there was a deadline, but the something that was changed
| broke a dozen other things nobody even remembers were there,
| and now there are bugs waiting to be discovered by
| unsuspecting users.
|
| > it's a matter of incentives and prioritization
|
| I know what you mean - it is, technically possible but it's
| never feasible because it's not predictable.
| pydry wrote:
| >Actually, in most cases - no, they can't. Or, rather, they
| can't figure it out as fast as they're "needed" to.
|
| I've forensically self-onboarded a few times and yeah, it's
| at least 12x more expensive than well written docs +
| pairing.
|
| It's economic for some projects but it's still crazy to do
| it if it's at all avoidable.
| flatline wrote:
| When you run up against this wall, it is often easier to
| write new software than to maintain the legacy code. Long
| term you are duplicating a ton of work at huge expense, but
| that is frequently the fate of successful projects: to be
| replaced with something sub-optimal that is nonetheless
| more tractable because there is less of it at the beginning
| and rapid feature development makes it easy to demonstrate
| value.
| seti0Cha wrote:
| More often I've seen that all that fast development time
| is spent trying to replicate features of the old system
| that were missed or deferred when designing the new
| system. Strangely, when they have achieved feature
| parity, the new system often doesn't seem so easy to work
| with.
| simmerup wrote:
| It's definitely easier to write new code, but faster? I
| doubt it.
|
| I see a lot of the time programmers doing the easy/fun
| thing rather than the hard beneficial to the business
| thing.
| JauntyHatAngle wrote:
| Part of this is resume driven development.
|
| Loyalty gets you very little in today's climate. Why
| should I spend time understanding the current system that
| was developed on yesterdays hotness?
|
| I need to spend my time here developing on the latest
| trends on a new solution (learning more from running into
| the same walls the current system did) - not learning the
| domain knowledge + old framework, so I can move on after
| a year and earn more at the next place.
|
| Is this good for the world? No. But unless you want to
| swing the hammock and work at institions that value
| incumbents, the incentive is to do whatever work improves
| your resume, not what benefits the company.
|
| I don't personally like it, and I often see old systems
| that are just fine get thrown out and replaced with the a
| less featured replacement that grow into the same kludge
| as the first, but I can't ignore that when you do that
| you learn way more and have better examples to point at
| in interviews.
| ramesh31 wrote:
| It's not "gone", it served its purpose. Software is born,
| lives, and dies. By the time a project is 3-4 years old, unless
| it is a core central service, the business (if it is competing
| successfully) has most likely moved far beyond the initial
| assumptions laid out at its' creation.
| giaour wrote:
| 3-4 years is very short as far as software lifespans go.
| Software libraries that are 10-15 years old are commonplace
| in both open source and at FAANGs, and some industries
| practice "extreme longevity": any domain where you wouldn't
| be surprised to run into a mainframe is probably also running
| code with a 30 year+ lineage.
| compiler-guy wrote:
| Don't know what kind of projects you are working on, but in
| the world where I work, source-code that is literally decades
| old is common, and companies pay extremely good money for
| people who understand it.
|
| This includes things like airline reservation systems,
| scientific computing models, compilers for obscure dsls, and
| the like.
| tyre wrote:
| If a business is competing successfully I don't think you're
| going to see software obviated every three years. Software
| isn't like living beings that up and die. They evolve over
| time.
|
| Companies doing rewrites every three years have a hard
| ceiling.
| marcosdumay wrote:
| > it served its purpose
|
| That's a blind jump. We don't know the purpose, we don't know
| how much was invested, how much was already recovered, and
| how much more could be earned if the program was allowed to
| live.
|
| All we know is that it has recently hit fitness to purpose,
| and that the fitness caused it to be abandoned.
| guntherhermann wrote:
| It sounds like you haven't been in the situation the parent
| is describing.
|
| I have and still am. We have stuck one ancient piece of
| software in a docker container and have our fingers crossed
| that it continues to work. This software is OLD, early Perl 5
| old. Nobody in the department understands Perl or wants to.
| The business (read: product owner) has deemed it unimportant
| to update or understand, but I assure you that if it stopped
| working, it would affect millions of tax payers, upper
| management would find out (read: VP, CEO), and heads would
| roll.
|
| Everything is fine, until it isn't.
|
| Your thinking from the perspective of a startup might be
| true, but from the perspective of a company that is many
| decades old and has a dozen layers of management... perhaps
| not so much
| btilly wrote:
| _Nobody in the department understands Perl or wants to._
|
| And that is the problem.
|
| I don't know how bad that code is - a lot of Perl code from
| that period is not good - but learning enough Perl to have
| a shot at it literally should take a competent developer a
| few minutes if you had someone explain it to you.
|
| You might find the language weird. $ means "the". @ means
| "these". % indicates you're building a lookup. Which you
| then access with braces. So %foo is a lookup. And
| $foo{$bar} should be read "the foo of the bar...".
|
| Early Perl 5 means it probably doesn't use references or
| objects. You do use regular expressions. While regular
| expressions are built into Perl in a way that makes them
| easy to use, Perl-like regular expressions are in pretty
| much every language. So the effort you need to use them is
| directly portable to Python, JavaScript, Java, etc, etc,
| etc.
|
| Fluency takes a lot longer. But getting to the point of "I
| can puzzle my way through a language I don't know well"
| should be doable very quickly. And it is a skill that every
| developer should build. Because I guarantee that whatever
| language and toolset you are using now, you'll need to be
| able to switch at some point in your career. And practicing
| in advance will help you. Plus being the person who can
| puzzle out old ganky code in another language lets you be a
| hero in fun ways!
| jasonlotito wrote:
| > And that is the problem.
|
| No, that's not the problem.
|
| > the business (read: product owner) has deemed it
| unimportant to update or understand,
|
| That's the problem. It's not on a random person in the
| department to spend their own time trying to solve a
| problem the business doesn't want invest time or money
| into solving. I don't see you offering to do this for
| free, so don't claim that's a problem for others.
| btilly wrote:
| I've been around the block enough to read between the
| lines on this one.
|
| All of the talk about how old and bad the language is,
| and how nobody wants to learn it is tech speak for, "We
| can't support it as is, we need a rewrite!" Anyone who
| has been near tech for a long time knows "rewrite" means
| "tech is going to break it for unclear reasons and
| uncertain gain, and in a few years will want to rewrite
| it again. Tech people always underestimate the cost and
| risk of a rewrite, and also overestimate the benefits.
|
| Business is not being stupid in saying that tech can't
| rewrite the software. The software is there, works, and
| is being relied on. Tech should figure out how to support
| it without a risky rewrite. Don't stand on, "We don't
| want to learn Perl." Learn how to be a polyglot. At least
| when it comes to reading. And to the extent that tech
| understands the value of being prepared, tech should
| prepare in advance. And figuring out how to read the code
| is most of that battle.
|
| Now you accuse me of not offering to do this for free.
| But in fact I offered the best tip I could for a non-Perl
| programmer who is trying to puzzle their way through an
| early Perl 5 script. Given that I'm an internationally
| recognized expert in Perl, that tip probably does more
| good than you realize. Learning how to navigate legacy
| code in a legacy language is a learnable skill. And you
| learn it by trying, then finding you can. And no matter
| how much you don't wish to learn any particular legacy
| language, having the confidence to know you can is
| valuable.
| guntherhermann wrote:
| > All of the talk about how old and bad the language is,
| and how nobody wants to learn it is tech speak for, "We
| can't support it as is, we need a rewrite!" Anyone who
| has been near tech for a long time knows "rewrite" means
| "tech is going to break it for unclear reasons and
| uncertain gain, and in a few years will want to rewrite
| it again. Tech people always underestimate the cost and
| risk of a rewrite, and also overestimate the benefits.
|
| No, it really isn't. I'm happy to learn a new language.
| The reason is correctly stated by a sibling post here
| (WesolyKubeczek).
|
| That we've been able to containerise it has limited a lot
| of the risk, and any upstream issues we have going
| forward we should be able to write some middleware to sit
| between it and transform the data into the format we
| need. We can extend the container to proxy any requests
| from the ancient application to the new middleware. We
| have a strategy, the product owner is happy with that,
| but we've outlined that maybe the solution won't work as
| we think, there are always unknown unknowns...
|
| _The issue is that if a problem does arise, it will be
| immediate and the effects will be there for as long as it
| takes to get a fix up and running, which could be weeks
| to who knows how long._ That 's a real risk in my eyes,
| but it wasn't to our product owner.
|
| It will affect a lot of people, it's a pretty widely used
| service! It's entertainment, so people won't die if it
| fails.
|
| The risk was deemed low enough by the product owner that
| we can just leave it in a container until it's finally
| sunsetted in 5-10 years (or so...). We've had 2 product
| owners since the OG product owner made that decision, so
| it could just be the product owner who made the decision
| thought they'd be long gone by the time it became an
| issue.
|
| You're making assumptions about people you've never met,
| you don't know me and you don't know what motivates me.
|
| You aren't as good at "reading between the lines" as you
| think you are.
|
| ---
|
| From your profile: _My main programming language is Perl_
|
| Oh, it all makes sense, you're hurt that I called Perl 5
| old! That's funny. I've never written production Perl, I
| am not making a judgement on the language.
| WesolyKubeczek wrote:
| The problem that your product owner cannot see any value in
| replacing/understanding/modernizing that piece of software,
| but should it fail, the heads would roll, speaks of a
| bigger problem in your organization than an ancient piece
| of software.
|
| You have been able to put it in a container and it's
| running, so unless the kernel itself changes drastically so
| old libc will just stop working, it's going to be able to
| work unchanged for a decade more.
|
| But the communications problem between management layers is
| way bigger than that, and could probably bite in some other
| situation.
| intrasight wrote:
| What would be an example of such a software company?
|
| I am always faced with this issue with non-software
| companies in industries where I've worked such as nuclear,
| defense, energy, and healthcare. At these places, I've
| confronted "hands-off" modules that are effectively black
| boxes of complex logic that has been vetted over decades,
| is written in C or Fortran, and nobody is allowed to
| modify.
|
| I can't imagine that situation existing in a software
| company.
| guntherhermann wrote:
| In my haste to give an example I missed the part about it
| being for only "software companies", we're not a software
| company, but we do have a lot of developers
| Micoloth wrote:
| That's amazing! Can you share more about what sector you
| are working in and what the software in question does?
| civilian wrote:
| This is absolutely a failure of management. Both the
| general management and _especially_ the engineering
| management.
|
| Eng management needs to convey that this is an Achilles'
| heel that is important to understand & be able to update.
|
| No engs want to work on perl5? Bribe existing employees,
| hire someone who wants to work on perl5, or a darker third
| option.
|
| The product owner can even spin this as them doing a _good_
| thing to future-proof the business, even if it means their
| team loses velocity on other projects.
| alex-nt wrote:
| Fun story about the 3-4Y window, one of the execs at my
| company was having a Q&A, and he gave an example in another
| question a new set of components, saying that he expects them
| to be in use for the next 20Y. I asked why does he think he
| can keep the know-how and outsmart the competition
| (components were part of algo trading and other trading
| related things) for the next 20y with just this investment,
| and he just downplayed my question as an IT guy wanting more
| money/work. -\\_(tsu)_/-
| scioto wrote:
| I'd like to disagree with you there, because of this one data
| point ;-).
|
| I worked on an application where I knew just enough to make
| spot changes, while the other developer who had made the most
| core changes continued on with the core. Then he left.
|
| So as more core changes needed to be made, none OMG due to
| Production issues, that made me dip into more and more of the
| legacy code, and I grokked what was going on. I wasn't happy
| with it, but I understood it enough to know that to make it
| truly maintainable would require a total re-do. But the code
| that was there worked, and as time went on I even took other
| developers' bolt-on additions and made their functionality take
| advantage of the sub-optimal idioms the core used, so that at
| least if someone else had to make changes, they'd only have to
| deal with one pattern. I've made wiki pages to describe it, but
| still any other developer who works on it ends up asking
| question after question and complaining why the documentation
| isn't in the wiki (it is; they just don't want to look).
|
| We eventually moved to another servlet container, and I filled
| in the missing functionality the new server lacked, still
| keeping the original core code.
|
| I've done that with two other applications at our company as
| well. Why haven't I moved on? Hey, I'm older now, and it's an 8
| to 5 job you can't easily find any more.
| dec0dedab0de wrote:
| Sometimes it's worse when they start tacking on changes in an
| inconsistent way. It's always painfully obvious when a company
| is shipping a product that has been acquired multiple times,
| and/or hacked on by contractors that don't care about the past
| or future of the project. I'm looking at you Aruba, Cisco, and
| pretty much every network vendor.
| majormajor wrote:
| I've seen the first part of this story a lot.
|
| Software product is delivering value, maintenance isn't needed
| much, people move on to other projects[0], people forget about
| it, then a couple of years later a change is needed.
|
| But how would the first response in that "couple of years
| later" scenario be "throw away all that investment and start
| from scratch" instead of "dig into how it works"? That's not a
| long time, really.
|
| [0] this is a good thing, IMO. If every project needs constant
| maintenance from its "owner" you as a company are going to
| rapidly run out of resources to do new things.
|
| (There's a closely related thing, here, though, that as an
| individual or a low-to-medium level manager, keeping track of
| these risks and low-level maintenance areas is part of your job
| to find ways to keep it from ever turning into a disaster. At
| some point - 10 years, 15, 20... maybe the knowledge gap gets
| way too big for the current team - then get proactive.)
| commandlinefan wrote:
| > But how would the first response be "throw away all that
| investment and start from scratch" instead of "dig into how
| it works"?
|
| Because start from scratch is more predictable ("estimable")
| than "dig into how it works". "Dig into how it works" is
| better in the long run, but will take longer. All software
| management is laser focused on speed and predictability
| rather than long-term quality.
| seti0Cha wrote:
| Two terms you may want to refresh your memory on: second
| system syndrome and Chesterton's fence. I've never seen a
| rewrite come in remotely on time, and I've seen quite a few
| fail altogether. Often what happens is, in trying to solve
| the problem again, the developers come to understand the
| original system, which already works. It becomes impossible
| to articulate tangible benefits for the rewrite and it gets
| shelved.
| yamtaddle wrote:
| Rescuing everyone with a shiny new system also takes more
| resources and looks (as you kinda note, too) a lot more
| like a _real project_ , so product managers and project
| managers and various other managers can get comfortably
| involved and do their thing, with plenty of people to
| manage (= more prestige) and new UI to show off in
| powerpoints (= more prestige, doesn't even matter if it
| functions worse than the old one) and ordinary, regular
| development milestones.
|
| Learning the old system enough to keep working with it,
| meanwhile, looks a lot like a couple developers staring at
| screens for months without results that look impressive to
| anyone who matters (as far as granting rewards goes) and
| with very uneven progress. Plus they've gotta be pretty
| good, experienced developers (so, likely expensive), or the
| risk of breaking shit as they begin trying to change the
| system, goes up a ton.
| majormajor wrote:
| This is exactly the case where I think starting from
| scratch is the most difficult to estimate. This thing we
| need to update is a mystery-box, right? You don't know the
| code. You don't know all its edge cases. Tons of potential
| bugs could be introduced (or re-introduced) if you change
| it in the wrong way.
|
| Replace-from-scratch has all the same risk of getting
| things wrong because of that lack of knowledge _and_ a
| bunch more new-code-writing on top of it. Estimating new
| development often fails; estimating it in the face of
| business requirement uncertainty is even worse.
| WorldMaker wrote:
| Can you quantify that to the management in that case?
|
| To certain styles of management focused on "always
| building new products" to guarantee their worth/value,
| they absolutely will get blinders on and believe "new
| products are easy and more predictable than maintenance
| of old products". Those blinders are hard to take off
| once they get into that sort of pattern. All of their
| processes are built around "new products". That starts to
| be the hammer that wants to dismantle old products
| because they don't do maintenance (that's someone else's
| problem) but they surely know how to build new products.
| They are the build new products group, that's what
| believe they do. Building new products is exciting and
| they can be passionate about it.
|
| I also think that's generally the wrong way to work and
| does plenty of harm and creates a lot of unnecessary
| stress on development teams caught in those cycles. But
| I've certainly seen plenty of management teams that do
| work that way in the real world and maybe don't question
| it enough.
| mfuzzey wrote:
| "Dig into how it works" may take longer than rewriting or
| it may not. And the rewrite may (and often does) run into
| unexpected difficulties too.
|
| I think it's a mistake to make the "fix the old" or
| "rewrite from scratch" decision up front from a position of
| high uncertainty about the old system.
|
| I'd go with a strategy of starting a timeboxed exploration
| of the old system with the objective of understanding it
| better, hopefully well enough to have a reasonable idea of
| what it will take to fix it. Then make the decision from a
| better informed position. Yes sometimes the decision may
| still be to rewrite because you now _know_ that the old
| system will be excessively hard to change but in many cases
| you may find out that the old system wasn 't as bad as
| expected.
| AnimalMuppet wrote:
| The real disaster is management. This thing was working
| perfectly, and therefore was ignored. Then it became a
| frantic crisis because nobody saw that some other change
| would break it.
|
| And then they responded with an absence of reality. A two-
| week deadline that can't be moved? Um, no. Ever hear of
| reality? You might visit there sometime. It's a bit harsh on
| people with ridiculous demands, though, so adjust your
| attitude before you go.
|
| And _then_ , nobody can fix the existing one? You not only
| have the ridiculous demand, you rule out the _only possible
| way_ of meeting it? What do you think is going to happen, the
| elves are going to work all night to magically create a
| solution?
|
| This sounds like an epic management failure from start to
| finish.
| [deleted]
| avereveard wrote:
| People joke about militaries writing docs about the exact number
| of turns each screw can take, but a-10 are still airworthy and
| the last one was built in the 80s
|
| Quips aside, software life expectancy matters, so total
| documentation approach is not something that works for
| everything. But if your software has generations of developers,
| maybe it's worth thinking about it.
|
| That or a total rewrite every skip generation. I can see both
| working, and maybe total rewrites are a better approach in a
| world of ever changing requirents.
|
| Thing is, documentation is great to capture static knowledge, and
| a evolving software is not static by any means
|
| If I had to pick and choose, I'd say written requirements with
| change management for every domain object APIs within the
| software object model, and anarchy underneath.
|
| The code conforms to the requirements, and not the documentation
| conforms to the code, so to say. Contract driven, in a way, but
| only where it's relevant.
| mr_toad wrote:
| > Thing is, documentation is great to capture static knowledge,
| and a evolving software is not static by any means
|
| There are many open source projects with great documentation
| and they manage to keep abreast of changes.
|
| This attitude among programmers that documentation is hard, so
| don't bother is just that - attitude.
| ikrenji wrote:
| the documentation for something like angular or rxjs is about
| how to use the framework, the documentation doesn't explain
| the underlying code...
| giantg2 wrote:
| My company actively promotes rotating to different areas of the
| company every 18 months. I think it's terrible for continuity as
| well as skills development.
| ranting-moth wrote:
| It's good to know how different areas of the company, but
| that's taking it to the extreme!
| rpod wrote:
| Seems related to the concept of conceptual integrity as stated by
| Fred Brooks: https://wiki.c2.com/?ConceptualIntegrity
| dang wrote:
| Related to Naur's classic:
|
| _Naur on Programming as Theory Building (1985) [pdf]_ -
| https://news.ycombinator.com/item?id=31500174 - May 2022 (4
| comments)
|
| _Naur on Programming as Theory Building (1985) [pdf]_ -
| https://news.ycombinator.com/item?id=30861573 - March 2022 (3
| comments)
|
| _Programming as Theory Building (1985)_ -
| https://news.ycombinator.com/item?id=23375193 - June 2020 (35
| comments)
|
| _Programming as Theory Building (1985) [pdf]_ -
| https://news.ycombinator.com/item?id=20736145 - Aug 2019 (11
| comments)
|
| _Peter Naur - Programming as Theory Building (1985) [pdf]_ -
| https://news.ycombinator.com/item?id=10833278 - Jan 2016 (15
| comments)
|
| _Naur's "Programming as Theory Building" (2011)_ -
| https://news.ycombinator.com/item?id=7491661 - March 2014 (14
| comments)
|
| _Programming as Theory Building (by Naur of BNF)_ -
| https://news.ycombinator.com/item?id=121291 - Feb 2008 (2
| comments)
| Terretta wrote:
| > _Accepting program modifications demanded by changing
| external circumstances to be an essential part of programming,
| it is argued that the primary aim of programming is to have the
| programmers build a theory of the way the matters at hand may
| be supported by the execution of a program. Such a view leads
| to a notion of program life that depends on the continued
| support of the program by programmers having its theory._
|
| Agree with the theory of programming as theory*, but here's the
| problem with that essay in an era of software-as-a-services and
| platforms:
|
| A SaaS offering or an Enterprise runs on a stack or assembly of
| software, _each of which has its own Theory._
|
| The problem with no churn is that most of the Theory is not in-
| house, it's elsewhere, and evolving. If you can't tap into
| that, you're falling behind. If you don't know it, you can't
| remix it into your own.
|
| It's just math to convert an org's "refresh rate" of outside-in
| knowledge (aka churn) into the average lag of that organization
| behind current best theory and patterns. A 10% churn means an
| average of 5 years out of date. A 20% churn means _at least_ a
| fifth of your org is 5+ years out of date.
|
| A couple key antidotes, though (and there are of course more):
|
| Realize that knowledge transfer from (a) outside in, then (b)
| among, is vital to theory maintenance and evolution.
|
| That means (a) have a great "always be hiring" pipeline of
| fresh Theory and build processes to make knowledge sharing of
| outside and inside theory frictionless and routine, and (b)
| have a great "always be learning" approach to exposing
| developers not just to the internal theory but to the best and
| most up to date outside theory.
|
| The better an org is at (b), the less it needs (a). The worse
| it is at (b), the more it needs (a), but -- crucially -- it
| must refresh with candidates who know more recent theory than
| the ones doing the hiring!
|
| Churn is not bad, that word is spin. Internships are a thing,
| so are externships (firms should do more of this). People, and
| theories, grow by exposure to novel challenges. Circulation is
| good for the whole ecosystem of software development because we
| are building a collective Theory bigger than any one org.
|
| This remixing of ideas is how our collective Theory evolves.
|
| * Most recent decade of my career involved a lot of
| evangelizing to enterprise management that developers are not
| project teams to form and disband as fungible resources. Think
| in terms of capabilities to garden, not projects to remediate.
| FeistySkink wrote:
| Here's a more readable version I came across:
| https://gist.github.com/nwh/60eb3115754e68f2d1c46aa31961b660
| roomey wrote:
| This happens even if I step away from my own code for a few
| months.
|
| The last program I built was coded around characters from
| Sherlock Holmes, by making the code like characters that had to
| interact in a certain way, it helped me maintain a mental model
| over a longer time. It genuinely made the code more enjoyable to
| work on, as it is a story now
| kens wrote:
| > That mental model can only be learned by having worked on the
| project while it grew or by working alongside somebody who did,
| who can help you absorb the theory.
|
| Maybe it's because I've done so much reverse-engineering, but I
| reject that claim. It's quite possible to start with a binary ROM
| dump and build up the theory of the code. If you have source code
| that's even better. And the whole point of design documents is to
| explain the theory.
|
| Curiously, the book quoted at the top of the article gets this
| correct: "Revival of a program is the rebuilding of its theory by
| a new programmer team." But the article says that the only way to
| rebuild the theory is by writing new code.
| deathanatos wrote:
| Certainly you can build _a_ theory via reverse engineering. The
| question is whether the theory in _your_ head matches the
| theory in the head of the person who wrote the code.
|
| Naur's argument is first that there _is_ a theory at all, and
| that the question of whether it can be transplanted to another
| person is really unanswerable. We can attempt to answer it, or
| build confidence towards an answer, by doing things like
| writing /reading design docs, having experience in the industry
| that shows us how other eng tend to think and design, talking
| to the person who wrote the code, testing our theory against
| the code, etc. But we can never be certain of what is in the
| mind of another person.
|
| Anyone whose ever taught anything to a student, I think, has
| come to see this. It's hard enough to teach people how to do a
| thing in response to a known circumstance. The real trick is to
| teach them well enough to have them extend their knowledge on
| known circumstances to new and unforeseen ones.
|
| And Naur, IIRC, also argues that churn is inherently bad, and
| that programmers are not replaceable cogs, because one of the
| valuable contributions they are making is holding, in their
| minds, the theory of how the code solves problems:
|
| > _More generally, much current discussion of programming seems
| to assume that programming is similar to industrial production,
| the programmer being regarded as a component of that
| production, a component that has to be controlled by rules of
| procedure and which can be replaced easily_
| unsui wrote:
| Reverse-engineering is inherently an inductive process, whereby
| you have to make educated guesses as to what the model is,
| iteratively.
|
| Samba is a great example of how this is necessarily an
| imperfect model, and definitely not an optimal process:
|
| - https://news.ycombinator.com/item?id=488089
|
| Development as theory-building is intended as a metaphor to
| understand the impact of the collaborative mental model that is
| created as part of any complex system, and the losses incurred
| when this mechanism is not appreciated.
| lll-o-lll wrote:
| Having worked on reverse engineering complicated proprietary
| network protocols (but not for a long time), this is true to
| an extent. The "mental model" developed may not match the
| original authors in the exact way intended, but a working
| model can be constructed none the less.
|
| We were often able to make better models than the assumed
| original authors (at least the software that used said
| protocols was much more performant due to better packing
| algorithms), so I'm unconvinced that reverse engineering puts
| you in a worse state.
|
| Fully agree that the cost of reverse engineering is large,
| requires exceptionally skilled people, and no company should
| ever willingly put itself in the position where it becomes
| necessary for their own software.
| jcul wrote:
| It's been a while since I've read the original paper, but that
| is my recollection of it.
|
| That if a new team takes on a codebase, they are rebuilding the
| theory and reviving the program.
|
| The new theory may not be exactly the same as the old but it is
| probably still adequately compatible.
|
| In a sense, the new team is reverse engineering the theory,
| from the source code. Even though this is a much easier task
| than doing so from a binary.
| deathanatos wrote:
| The paper's conclusion is stronger: a new person on a still
| _existing_ team is rebuilding the theory. Albeit much quicker
| and with much higher confidence, since they can talk to the
| existing devs and learn directly from them, but they 're
| still having to build their own theory, separate from the
| original. (As we don't have SFTP between brains, yet.)
|
| > _A very important consequence of the Theory Building View
| is that program revival, that is reestablishing the theory of
| a program merely from the documentation, is strictly
| impossible_
|
| > [...]
|
| > _Similar problems are likely to arise even when a program
| is kept continuously alive by an evolving team of
| programmers, as a result of the differences of competence and
| background experience of the individual programmers,
| particularly as the team is being kept operational by
| inevitable replacements of the individual members._
|
| Reading documentation & talking to the original dev are
| inherently the same thing: we're using some medium (written
| docs, spoken word) to attempt to convey information.
| lll-o-lll wrote:
| Yes exactly. It's reverse engineering. That doesn't mean it
| will be easy, but once the reverse engineering is done you are
| back to where you were.
|
| Anecdote: Worked for a company that had a large software
| component (graphical design tool) that had been written in C
| with a very complicated object model/macro layer, file system
| format etc. Zero documentation, hadn't been updated in a decade
| and not a single person remained who understood it or had done
| more than simple bug fixes. Couldn't rewrite as customers had
| huge investments in assets built from the tool.
|
| One principal dev and maybe 1-2 months? And we had the theory
| of mind back, explained to everyone else and a new team of 5-6
| devs working productively. Component was brought into the 21st
| century, and this time some documentation was written.
| majormajor wrote:
| > What keeps the software alive are the programmers who have an
| accurate mental model (theory) of how it is built and works. That
| mental model can only be learned by having worked on the project
| while it grew or by working alongside somebody who did, who can
| help you absorb the theory. Replace enough of the programmers,
| and their mental models become disconnected from the reality of
| the code, and the code dies. That dead code can only be replaced
| by new code that has been 'grown' by the current programmers.
|
| and
|
| > Team stability is vital for software development. Each team
| needs to be composed of a majority of first-generation
| developers.
|
| How does this interact with individual or career growth? I don't
| want to work for the manager who insists I stay on the first few
| projects I worked on at the company forever...
|
| To take this seriously, internal churn (even if just someone
| being promoted into management, or specializing on expertise in a
| small part of the original project, or switching projects when
| they get bored) is almost as costly as external churn. Five, ten
| years later how much is their domain knowledge still that
| relevant to the people working on the code day to day today? It
| could be harmful if they've been out of the loop and make
| outdated suggestions, even!
|
| Assuming and desiring your employees will stay in the exact same
| role indefinitely seems like a huge _cause_ of churn.
| GalenErso wrote:
| > How does this interact with individual or career growth? I
| don't want to work for the manager who insists I stay on the
| first few projects I worked on at the company forever...
|
| By documenting your work and/or mentoring a junior employee to
| pass on your tribal knowledge.
| wpietri wrote:
| > How does this interact with individual or career growth? I
| don't want to work for the manager who insists I stay on the
| first few projects I worked on at the company forever...
|
| To me this is the a huge incentive for building strong teams.
| Nobody should be indispensable; it's a disservice both to the
| company and the person. Pair programming, collective code
| ownership, intentional cross-training, good automated tests,
| continuous deployment, and good automated monitoring have all
| been very helpful to me there. And vacations! Everybody should
| take long vacations where they are out of contact. By
| managerial force if necessary.
| nickfromseattle wrote:
| +1 on long vacations. My team is in Europe with 38 days of
| PTO and while it offends my American capitalistic
| sensibilities, it's created so much resilience because
| everyone is always on vacation, we've built our company ops
| to reflect this reality.
| satvikpendem wrote:
| Like Netflix's chaos monkey but for people; unplug anyone
| from the system and the system should still work.
| grahamplace wrote:
| At least one CTO has actually tried exactly that:
| https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-
| chao...
| yoyohello13 wrote:
| Kind of funny that being allowed to live life outside of
| work is offensive to American sensibilities.
| shuntress wrote:
| It doesn't "offend" "capitalistic sensibilities". It's just
| a result of optimizing the wrong metric.
|
| Decision makers for American companies tend to use _" Time
| spent at work"_ as a proxy for _" Productivity"_. That used
| to work just fine when the job was pulling levers on an
| assembly line but turns out to not make sense when you are
| talking about software development.
| antihipocrat wrote:
| Is having longer PTO in your business a detriment or
| neutral/net gain to overall productivity in your
| experience?
| naravara wrote:
| An employer shouldn't want that for you either. I think what
| this ends up being is an argument for good documentation and
| onboarding so you can transmit the "theory" to other people.
|
| Reasonable amounts of churn are manageable if you do a good job
| of acculturating new people into the team. This has actually
| been one of my concerns about the switchover to WFH. I know a
| lot of that acculturation used to happen by osmosis and people
| working next to each other. I think my team doesn't do as good
| of a job maintaining it when we're all working separately.
| We're a lot more efficient now, but I think we haven't figured
| out a good way to share and transmit context like we used to
| and I worry what happens if we hit a period of flux in our
| company. I worry our team culture is a little more fragile now.
| jpadkins wrote:
| > I know a lot of that acculturation used to happen by
| osmosis and people working next to each other.
|
| I used to think this too, but I think we need to be cautious
| of magical mechanisms for knowledge transference. I think the
| real mechanisms are code reviews, discussions of
| issues+code+project, consults / shared problem solving,
| design reviews. Just sitting next to someone doesn't work,
| there needs to be a shared task or problem for knowledge
| transfer to happen.
|
| I think all of these shared tasks happen with remote workers
| just as effectively, as long as you have good team comm
| hygiene.
| majormajor wrote:
| The biggest difference between remote and in-person is that
| many of those things can happen more easily without being
| scheduled. The organizational management skill bar you need
| to hit as a team is lower when the communication friction
| is much lower.
|
| Talking to your teammate who sits nearby in person is both
| lower friction and higher bandwidth than sending them a
| chat method.
|
| "Adhoc conversations" don't magically lead to knowledge
| transfer, the probability of any adhoc convo being long-
| term-meaningful is low, but they greatly increase the
| cumulative probability.
| feoren wrote:
| > How does this interact with individual or career growth? I
| don't want to work for the manager who insists I stay on the
| first few projects I worked on at the company forever...
|
| Grow a garden, not a tree. Let's disambiguate the word
| "project" from the word "system" (not "product" -- that's
| sales). Let's call a project a time- and scope- limited effort
| to create or modify some system(s) in your suite (your garden).
| I think it's a common mistake to mix these two up. If projects
| just drag on forever, then yes, employees can feel stuck. By
| this definition, I certainly would not want to work on one
| project forever.
|
| What about working on one system forever? The problem there is
| that to stay busy, you're forcing attention on a system that
| _might not need attention_. At the end of a project, a system
| should be fulfilling its purpose, and it should do so
| indefinitely, forever. That purpose may expire, but the system
| shouldn 't. There should be no urgency to keep messing with a
| working system.
|
| So let's work in one _garden_ forever. Systems will grow; we
| 'll prune and re-plant them; we'll give the healthy ones the
| barest maintenance it needs; we'll give the dying ones the
| sweet release of death; we'll shape and braid them together.
| All your institutional knowledge about how System A works will
| be great when you're writing System F that depends on System A,
| or when we're splitting System A into Systems A1 and A2. I
| would absolutely _love_ to work at a company where I can work
| in one garden for the rest of my life. That 's my dream job.
|
| Sidebar: do not make the mistake of assuming "system" means
| "microservice"! That's only one of many ways to define system
| boundaries.
| majormajor wrote:
| I agree with this - while at a single less-than-a-few-
| thousand-people-company I'd generally call that a single
| "garden" - and think it's a good extension on the original
| article. The author, though, focuses very narrowly on the
| "code" level instead of the "components of the system" level.
|
| If you can grow people into being the ones who know how the
| components of the system work together at a layer _above_ how
| the code for any given component works, you can have a
| retention plan and keep institutional knowledge. (Or even
| then move people into a layer above that, as the system
| grows, etc).
|
| You still won't retain everyone, some people want to jump
| between gardens, but you will be able to retain a LOT of
| people, and it's much better than telling people they have to
| own all their individual lines of code forever.
| escapedmoose wrote:
| > At the end of a project, a system should be fulfilling its
| purpose, and it should do so indefinitely, forever. That
| purpose may expire, but the system shouldn't. There should be
| no urgency to keep messing with a working system.
|
| Wonderfully stated! Losing sight of this is I think where
| projects start to run into trouble with scope-creep and the
| "fix" vs "enhancement" issue. It also shifts the perspective
| of what is a satisfactory state for a finished project vis-a-
| vis maintenance plans, dependencies on aging systems, etc.
| Your garden metaphor is also beautiful and IME accurate.
| banannaise wrote:
| Lead moves on to something else, everyone moves up a role, add
| a new junior. Repeat as often as necessary.
|
| Alternatively: just promote and pay your people as they
| develop, even if they're in the "same role".
| hinkley wrote:
| I worked on a project that was paying the bills while another
| team was getting to work with new cool stuff.
|
| What I learned from that project is that being indispensable
| sucks. That lesson has recurred in my career either for me, or
| when I got wiser, for other people.
|
| It's not enough to write reasonable code that 'works' or
| document it as it is now, there's a whole unmeasured and thus
| under-encouraged aspect around maintaining a legible edit
| history. I often am trying to get into the head of the bug
| author to figure out why they thought this was a feature in
| order to preserve functionality while fixing bugs. They aren't
| always around, and they aren't always fun to talk to. If you
| have the history, the person is free to work on other things.
| To gift them to people they mentor, or otherwise re-home them.
| If you don't have that, if it's all in their head, they have to
| leave to get away from the project.
|
| If I can figure out what you're doing I can figure out how to
| change one or two conditional blocks to fix a problem. Your
| code stays intact. If there's no history anyway, and the code
| is sketchy, I'm _far more likely to take the code away from
| you_ by rewriting it. We can 't have black boxes on the team.
| That's a ball-hog, and ball-hogs get benched.
|
| If you don't show your work, then you're going to find you have
| fewer and fewer interesting responsibilities. Because "how we
| got here" is also a "why" and we ask Why all the time.
| codazoda wrote:
| I don't wholesale agree with the writing but it does mention
| working with someone who worked on the project and can share
| the theory. In that case, a new dev is hired, and you instill
| all the knowledge of the system in the new dev. Once you've
| trained them enough, you can move on. New devs will adapt and
| create new theory and solutions. Rinse and repeat.
| MarkMarine wrote:
| That kind of misses what is being said in this article. The
| first generation programmers support a constrained population
| of second generation developers, and over time the second
| generation developers have written enough code that they
| become the first generation.
|
| A big component of what this article is saying is basically
| that knowledge transfer in the way you're describing doesn't
| work, and if you dig down to the theory this article is based
| on, new devs adapt and create their own theory and solutions
| was specifically called out as a problem. In case 1 of this
| paper, it's described as second generation programmers
| submitting patches to the first generation programmers that
| don't make use of the structure of the program, and add
| things that are trivially done by the existing systems. Then,
| a third, disconnected generation of programmers takes over
| the maintenance from the second generation, and they've added
| so much cruft to the system that the original powerful
| architecture can still be seen, but isn't in use because of
| all the additions.
|
| [0] https://pages.cs.wisc.edu/~remzi/Naur.pdf
| vosper wrote:
| > A big component of what this article is saying is
| basically that knowledge transfer in the way you're
| describing doesn't work
|
| Knowledge transfer doesn't work at all :) It's not a thing
| that's possible. The best we can do is for someone with
| knowledge to attempt to explain it, hope they do a good and
| complete job of it, and then hope that the explainee is
| able to internalise it and come to the same/similar
| conceptual understanding that the explainer had in their
| mind.
|
| It doesn't tend to be done very well at corporations. It's
| a skill that good teachers have developed after much
| practice, which isn't something we tend to hire for.
|
| And it requires favourable conditions, and one of those
| conditions is not "it's my last day tomorrow and the boss
| asked me to bring you up to speed on what I've been doing
| these last six months. You only had 30 minutes free in your
| calendar, so... let's go"
| MarkMarine wrote:
| In the underlying theory it's described as actually worse
| than that. A fully documented system explained by the
| first generation developers wasn't enough to transfer
| knowledge, the documentation only made a difference once
| the programmers had built a working mental model of how
| the software worked, and then they still submitted
| patches that would duplicate functionality because they
| didn't understand the system.
| mr_toad wrote:
| > they still submitted patches that would duplicate
| functionality because they didn't understand the system.
|
| A lot of the time they don't _want_ to understand the
| system. Maintaining some old legacy system is no fun.
| Programmers want to do new things their own way. And if
| there's no one around to guide them then they will.
|
| This is a management and cultural problem. Enterprise
| software resembles failed open source projects - no one
| wants to work on them so they die, get forked, or re-
| written and replaced.
|
| Contrast this with successful open source projects. A
| vibrant project will integrate new developers into the
| culture (or show them the door), and conversely
| developers who want to change things will have to prove
| themselves.
| majormajor wrote:
| But this doesn't boil down to "you have to replace the
| impossible problem of knowledge transfer with infinite
| employee retention"?
|
| Is that an improvement?
|
| (I'd also argue that I've worked on systems where I've
| developed a better mental model of the problem than the
| first-gen people I inherited it from, because I as a
| maintainer was under less pressure to get _anything that
| worked_ out the door initially.)
|
| EDIT: and looking at the paper you linked there,
| https://pages.cs.wisc.edu/~remzi/Naur.pdf , I'm in
| complete agreement with "the designer's job is not to
| pass along "the design" but to pass along "the theories"
| driving the design" - it was always easier as that
| maintainer to take over a project where people could tell
| my _why_ not just _what_.
| majormajor wrote:
| That's that "second gen" vs "first gen" thing but it actually
| leans in hard to the "no movement" right after introducing
| that concept - the "Each team needs to be composed of a
| majority of first-generation developers" quote is from that
| paragraph after the distinction.
|
| If you try to fix the theory knowledge/documentation/practice
| problems purely through insisting the "first gen" stays where
| they are, they're going to leave you for more interesting
| work, and then you're in exactly the situation you wanted to
| avoid. The only way out is "more effective practice of theory
| and knowledge transfer" but all the author seems to have to
| say about that is "This is a team management issue that can
| only be addressed by a manager who understands the
| fundamental nature of software development. There is no magic
| development process that will prevent these problems. It
| takes skill."
|
| E.g. that's the real hard part yet they completely punt on
| trying to dig into it.
| dathinab wrote:
| > How does this interact with individual or career growth? I
| don't want to work for the manager who insists I stay on the
| first few projects I worked on at the company forever...
|
| The first project is maybe a bad example but if we say the
| "nth" project instead then:
|
| - not always changing members doesn't mean never changing,
| companies should just not want you to jump between projects all
| the time (except maybe as junior programmer) and you shouldn't
| want this either as it's hard to build up proper deep expertise
| this way (instead of a lot of shallow expertise)
|
| - if you are still in the same company and the company is
| healthy then it should be fine if you get asked questions about
| your previous project for some time after you switched (through
| not arbitrary, just as an "emergency" thing)
|
| The problem is that the concept of carrier and seniority is
| seriously messed up in the software industry. Often shallow
| knowledge about a lot of things is more valued then actual
| expertise in the things needed. Similar you will find a
| surprisingly lot of people believing that a senior engineer
| which doesn't try to get into team lead/people managing
| positions is doing so because they "surly are incompetent and
| shouldn't be hired" (IMHO completely rediculus but I have still
| seen it often enough, just most times in a less directly
| phrased way). But in the end senior software development skills
| and team lead skills are distinct skill sets. Implicitly
| pressuring someone who is good at solving technical problems
| into a team lead position (or the other way around) is just a
| pretty stupid decision from a human resource utilization POV.
| (This depends a bit on the definition of team lead and yes team
| leads also need to have technical skills but most important
| they need to have the skill to listen to their team and not get
| stuck on their opinions, etc.).
| eternalban wrote:
| _The Tacit Dimension_ , Michael Polanyi, 1966
|
| https://press.uchicago.edu/ucp/books/book/chicago/T/bo603536...
|
| _Programming and Tacit Knowledge_
|
| https://mbuffett.com/posts/all-tacit-knowledge/
|
| _Tacit Knowledge blogpost series_ , Commoncog
|
| https://commoncog.com/the-tacit-knowledge-series/
|
| _Acquiring and Sharing Tacit Knowledge in Software Development_
| , 2013
|
| https://www.researchgate.net/publication/264125152_Acquiring...
| bwestergard wrote:
| I'm curious what you see as the connection between this article
| (which I think captures something very important) and the
| "tacit knowledge" concept (of which I'm very skeptical, having
| read Michael Polanyi's book and a bunch of the secondary
| literature).
| thenobsta wrote:
| I'm curious about the skepticism. Could you elaborate? The
| explanation of tacit knowledge hit me quite viscerally. I see
| a part of my engineering work as turning tacit knowledge into
| explicit knowledge.
| eternalban wrote:
| " What keeps the software alive are the programmers who have
| an accurate mental model (theory) of how it is built and
| works. That mental model can only be learned by having worked
| on the project while it grew or by working alongside somebody
| who did, who can help you absorb the theory."
|
| wiki says:
|
| " Tacit knowledge can be defined as skills, ideas and
| experiences that are possessed by people but are not codified
| and may not necessarily be easily expressed.[5] With tacit
| knowledge, people are not often aware of the knowledge they
| possess or how it can be valuable to others. Effective
| transfer of tacit knowledge generally requires extensive
| personal contact, regular interaction,[6] and trust. This
| kind of knowledge can only be revealed through practice in a
| particular context and transmitted through social
| networks.[7] To some extent it is "captured" when the
| knowledge holder joins a network or a community of
| practice.[6]"
|
| The distinction appears to be the implication of OP that the
| 'theory' is not communicable, thus "first generation" workers
| need to be around. But you can clarify (am interested in your
| views) (a) what's your crit of TD and TK, and (b) the matter
| that I missed, the "something very important" (which is not
| TK transfer).
|
| p.s. "we know more than we can tell" is pretty much the tldr
| of T.D.
| bwestergard wrote:
| We can paraphrase "we know more than we can tell" in at
| least two ways:
|
| 1. "We know [the manner in which we carry out a task] more
| than we can tell [...but are unable to articulate the
| procedure by which we do it]." 2. "We [can do] more than we
| [can explain in writing or verbally]."
|
| The second statement is uncontroversial. No one would say
| that the ability to ride a bicycle logically entails the
| ability to explain how we do those things in writing such
| that, by reading alone, someone could learn to ride a
| bicycle.
|
| But this is only mysterious if you consider the second
| statement to be a paraphrase of the first. I don't think
| that it is. The first statement suggests a mystery; it's as
| if we are always following rules when we act skillfully,
| but are sometimes unable to articulate those rules.
|
| But this is a mistaken view. Not all skillful performance
| is rule-governed. In fact, most isn't (statement two). And
| that is easy to forget, but does not require a theory of
| "tacit knowledge".
|
| See also: https://plato.stanford.edu/entries/knowledge-how/
| eternalban wrote:
| Possibly the issue is your conflating _knowing_ with
| _doing_.
|
| The straightforward parsing of that sentence for me is:
| We are not capable of _completely_ expressing our
| (personal) knowledge with language.
| bwestergard wrote:
| We are in agreement. Doing should not be conflated with
| knowing, as the "tacit knowledge" approach tends to.
| eternalban wrote:
| Good, we (the 3 of us) are in agreement. So what's your
| beef with brother Polayni?
| Guthur wrote:
| After 200+ years of capitalism It's now deep rooted in our
| society that all work is a commodity, with human labour wholly
| fungible.
|
| This in and of itself pretty dreadful but I have been wondering
| recentl whether or not abstract thought is quite so. I'm not
| quite sure.
| robocat wrote:
| Capitalism doesn't treat people as fungible, but instead
| according to a power law according to their capitalist value.
| Rare valuable skills tend to get paid highly, with
| professionals in the middle somewhere. The most common skills
| get paid near a minimum set by regulations, and also minimum
| wage is set by competition - see
| https://en.m.wikipedia.org/wiki/Baumol%27s_cost_disease
|
| High level capitalists (such as VCs) treat rare people
| unfungibly e.g. Paul Graham "the success of a startup is almost
| always a function of its founders" or
| http://paulgraham.com/founders.html
|
| Edit: also fungibility is by no means a capitalist idea, and it
| is a central concept to some non-capitalist doctrines. What is
| equality?
| jononomo wrote:
| When I first decided to get into web development after taking a
| couple of years off from software (after doing Java development
| in the finance world) I started with Ruby on Rails because it was
| hot at the time.
|
| I spent several weeks going through about three books on Ruby on
| Rails and working endlessly on tutorials, etc. But I got nowhere
| and I couldn't stand it.
|
| Then one day I read the Django documentation and it took a
| completely different approach. The approach that the Django
| documentation took just made sense. It began by explaining what
| the problem was from a high level and then explaining different
| solutions that had been tried over the years and what their
| drawbacks were and how this led to further approaches, and so on.
| By walking me through the system that way it made Django clear
| and sensible to me.
|
| All the Rails stuff I had been reading seemed to assume that I
| already knew everything but just needed to understand some magic
| incantation. Django treated me like a child, which meant that it
| actually treated me like an intelligent adult.
|
| I'll never forget the experience of reading that Django
| documentation. It was crystal clear and it changed my life.
|
| This all happened around 2012.
| unity1001 wrote:
| Losing institutional knowledge kicks the org. in the proverbial
| balls. Even if stellar seniors come in to replace the ones who
| left, they still take 3-4 months to get up to speed on
| institutional knowledge, and they rarely can get all the
| knowledge that the ones who left had.
| sublinear wrote:
| This is of course just my opinion, but I'm pretty sure what kills
| off a legacy project is whether the organization is committed to
| maintaining it rather than it being some kind of property of
| software itself.
|
| Minimizing churn is just one aspect of that commitment. If the
| project was always a mess it probably wasn't as important to the
| business as the developers believed.
| dathinab wrote:
| I have only limited experience but I think the worst thing you
| can do when taking over an startup (with the intention to
| continue it) is to start with firing people. I know it's
| expensive and not always viable but you should try to "let it
| operate like normally" while auditing it before starting firing
| people.
|
| Like some especially stupid examples:
|
| - firing the main tech lead which had worked month of unpaid
| overtime (not for month, but the accumulated equivalent of month
| of work) because he supposedly didn't work enough during the
| insolvency proceeding/first week after it.
|
| - cutting the cost of renting the storage system with essential
| data without making backups
|
| - firing most people which where important for the main product
|
| - firing people in such a bad way that most of the rest quit, too
|
| - deciding it's more important to keep marketing people then
| people from the development team in a situation where the
| insolvent company had multiple bigish contracts they just where
| missing a few month of work to be able to deliver to and
| interests from other customers and the buyer being the main
| potential customer buying the company to get the product
| cloogshicer wrote:
| This is an excellent paper, it changed my view on programming.
| I've written a similar essay [1], summarizing the paper and
| providing some case studies as examples from real-life projects.
|
| [1] https://hiringengineersbook.com/post/autonomy/
| SilverBirch wrote:
| I've got a little pet theory related to this. Which is that no
| one really solves difficult problems. They turn difficult
| problems into easy problems and then solve that. Often that
| literally means "We have this class of problems, I'm only going
| to solve the easy subset". And I've observed repeatedly people
| say "No we have to solve the real, hard, problem" - and a decade
| later, with a team of a hundred engineers, they've not really got
| anywhere.
|
| Applied to this, that theory is very simple. It's enormously
| difficult and expensive to keep old code running, especially when
| the engineers who originally wrote it have left. Not only is it
| difficult, but the engineers who have to do maintain it will hate
| it, churn themselves and you'll be back to square one. So, solve
| the easy problem. Get your new guys to write something new, and
| give up on the old software once all the people who knew how it
| worked quit. And it's easy! No one left at the company is going
| to argue to keep the code. The new people will enjoy writing
| something new. Oh and people will be amazed at how happy and
| productive your team is - because you literally just choose not
| to do things that are difficult and no one appreciates.
|
| But you may ask, what if the product is essential, you can't just
| abandon it. Well, call me flippant but if it were essential you
| shouldn't have let the engineers quit. And if you're really going
| to saddle me with it, I'll quit too. Then you still don't have
| your software, and it's not my problem - it's yours.
| JanisErdmanis wrote:
| > They turn difficult problems into easy problems and then
| solve that.
|
| I think this needs to be said more carefully. For instance,
| problems could become manageable when a well-thought-out mental
| model is built. On the other hand, they could become easy when
| the problem itself is simplified. For me, the e-voting problem
| fits quite well in the picture where thousands of systems are
| being built with cute mobile UI but, in the end, are really
| holding on to the assumption: trust us, we will be honest.
| eddsh1994 wrote:
| If it's essential it could last decades, greater than the
| career length of a developer. What happens then?
| clhodapp wrote:
| I guess you budget to rewrite it even though it's "working
| fine"
| [deleted]
| SilverBirch wrote:
| Successful products are like the ship of theseus, they can go
| through the hands of thousands of engineers, the important
| part is having people actively working on it overlapping with
| more experience people. You don't literally expect the
| original author to be there forever, butyou do expect people
| to come onto the project and learn about it whilst the more
| experienced people are still there. If literally everyone
| working on a project is gone, the project is probably
| unmanageable at that point anyway, there's no point starting
| CPR, the project is dead.
| coffeemug wrote:
| What is the bear case for this essay? It's notable that almost
| all large organizations operate in a diametrically opposed way,
| yet we have thousands of successful incredibly complex software
| systems. To me the essay feels intuitively true, but I'm not sure
| how to reconcile it with this empirical fact.
| gigatexal wrote:
| Isn't it obvious? Senior or long-term employees that leave take
| with them tombs of process/folk-knowledge with them. They take
| the answers of "who do you talk to in what team to get x done?"
| and "how best do you navigate this team and their hot-headed
| manager to get what you need?" etc. etc.
|
| Documentation is helpful but the intangibles that come and accrue
| to just someone or someones who have been in the trenches for
| years and years can't be documented and when they leave it can be
| fatal at least for a team/unit.
|
| Add to that the onboarding cost and the cost of hiring: take in-
| house or (worse) external recruiters and their
| salaries/commissions, add to that senior engineers taking time to
| interview candidates and both prep for the interview (hopefully)
| and then write up detailed notes afterwards -- count not only
| their time doing this but their time NOT mentoring a junior
| engineer or working on something else during that day (yes this
| might sound like double counting but senior engineers really do
| make 1+1=3 so pulling them off something is extra costly imo),
| and then add the expense of the employee being hired, all their
| gear that they'll need, and then the 3-6 months until their
| productive and all the help/training they'll need to get to
| productivity AND THEN imagine all that and then person leaves
| after 6 months to a year: how much does that cost? now multiply
| that by your attrition rate... it gets hella costly.
| cratermoon wrote:
| Meir Lehman wrote [1] extensively about software lifecycles and
| evolution, and came up with what he called "laws". Most of them
| should be familiar to HN readers. To sum up as succinctly as I
| can: programs in use undergo continual change; as a program is
| changed, it becomes more complex; the quality of the system will
| degrade over time unless rigorously maintained.
|
| He also wrote that over time a software system, through dynamics
| of the programming process, the rate of change tends towards
| being statistically invariant. This is because, he said, the team
| involved in the system, including not just the developers, but
| also the users and others, have a finite ability to maintain
| understanding of the system, and excessive and too-rapid change
| outpace the ability of the people to maintain understanding.
|
| A Sixty North blog post[2] provides some interesting graphs
| showing the measured effects of Lehman's laws, including one
| showing that after 5 years, almost 2/3rds of the code is written
| by people who joined the team after the original team members
| left.
|
| 1 Metrics and Laws of Software Evolution - The Nineties View
| https://ieeexplore.ieee.org/document/637156
|
| 2 https://sixty-north.com/blog/predictive-models-of-developmen...
| scrubs wrote:
| Totally agree - perhaps put more succinctly many rewrites are an
| attempt to independently rediscover the business requirements by
| reversing it out of code:
|
| "Team stability is vital for software development. Each team
| needs to be composed of a majority of first-generation
| developers. Too many second-generation developers and the first
| generation gets overwhelmed, and work stalls. The work slows down
| either because too many core tasks are waiting on the first
| generation or because the second generation keeps having to
| rewrite components from scratch because they didn't understand
| the theory behind the original component. Too few second-
| generation developers and there is no renewal--each developer
| that leaves the team is a potential catastrophe.
|
| Many teams in the industry constantly rewrite parts of their
| code. Not because they keep figuring out better ways of
| approaching the problem but because nobody on the team has an
| accurate mental model for how it works and how the code fits in
| with the rest. If you can't hold onto the original team, if you
| grow the team too quickly, you end up running to stay in place.
| Code keeps getting written without any improvements of substance
| to the software itself."
| bentobean wrote:
| > Documentation only works up to a point because it can both get
| out of sync with what the code is doing and because documenting
| the internals of a complex piece of software is a rare skill. A
| skill that most developers don't possess. Most internal
| documentation only begins to make sense to a developer after
| they've developed an internal mental model of how it all hangs
| together. Most code documentation becomes useful after you have
| built the theory in your mind, not before. It operates as a
| mnemonic for what you already know, not as a tool for learning.
|
| This is ringing particularly true to me right now.
|
| The organization I work for has allowed bit rot to occur for a
| number of years, to the point that it's becoming increasingly
| common for things to break whenever even seemingly small changes
| are implemented.
|
| The "solution" that we've implemented is to require a detailed
| RFC document whenever you want to create or modify anything.
| You're required to outline every anticipated step along the way,
| all the tiny minutiae that you foresee yourself encountering and
| how you'll deal with it. And in a Google doc, no less.
|
| It's utterly exhausting.
| yarg wrote:
| I worked in an environment where a senior architect got to veto
| every major effort that I publically made to improve the
| structure and maintainability of the software.
|
| At some point I got so pissed off about this that I started
| assigning code reviews to him - he contacted my manager and
| asked that I stop, because he had no understanding of the
| project in question.
|
| What did this mean from the documentation side?
|
| I would have loved to accurately document what I was doing and
| how the project was structured - but I actively avoided doing
| so.
|
| Every time my manager's boss knew what I was doing, I was about
| to be shat on from great heights.
|
| Months of work and planning erased at the drop of a hat - time
| and time again.
|
| Dependency injection? Fuck off. Spring's too heavy.
|
| Modularity? Fuck off. The project's too small.
|
| Hierarchical profiles? Fuck you, keep using a properties file
| (with an increasing number of dynamically generated keys).
|
| I was constantly asking for a new developer to be brought in
| for the transfer of knowledge - at best I was told "That's not
| a bad idea."; yet nothing was ever done about it.
|
| I took an unmaintainable pile of bugs and made it stable and
| maintainable. But nothing I ever did meant that I was someone
| worth listening to.
|
| Eventually I got so burnt-out (and consumed by feelings of
| uselessness) that I walked away without finding another job.
|
| I was damned near suicidal from fighting for acknowledgement.
|
| I cannot imagine that my departure went too well for them.
| andrewmcwatters wrote:
| Here's the kicker, I hate to say it, but they probably didn't
| notice you left.
|
| That's the thing about places like that.
| fatneckbeard wrote:
| yeup. and realizing that is the only way to stay sane.
|
| nobody will care if you leave, nobody cared about the last
| person that left, they have other priorities and software
| engineering is not one of them.
|
| if its that much of a problem, their business will fail and
| some other business that does care will take over their
| market. but that might take decades because software
| engineering isn't actually core to a lot of business
| activity. youd be amazed at how bad some software is for
| major companies, and yet they are very profitable.
|
| you only have control over yourself not other people.
| yarg wrote:
| I was the main developer on the single most critical
| deployment tool in the company.
|
| My departure would've been less than ideal.
|
| Either way, the company's haemorrhaged talent (now
| including long term CEO and long term CTO) and the stock
| price has plummeted since I left.
|
| (In no way am I crediting myself for this; at best I was a
| canary in the coal mine - but I take my solace in
| schadenfreude.)
| devwastaken wrote:
| It's a job, not the fifth world wonder. That Corp is making
| money based on their market position, the software
| functioning is an afterthought. If it all breaks, the
| customer isn't going anywhere.
|
| Don't fight management, leave.
| mr337 wrote:
| That does sound horrid. Makes green field projects look ever
| more so nice.
| ahepp wrote:
| Sure, but your greenfield project today will just be someone
| else's brownfield project in a couple years
| jerf wrote:
| "Most internal documentation only begins to make sense to a
| developer after they've developed an internal mental model of
| how it all hangs together. Most code documentation becomes
| useful after you have built the theory in your mind, not
| before. It operates as a mnemonic for what you already know,
| not as a tool for learning."
|
| This reveals the secret goal of documentation: To build that
| model, not to document every last detail. The code can do every
| last detail, it's the model that is hard.
|
| Set out with that goal and... yeah, I can't guarantee success
| for you either. It's a fundamentally challenging task. But at
| least you'll be off to a better start than if you think the
| primary purpose of documentation is to document what happens if
| you set this combination of flags. That has its place too,
| certainly, but if I can only pick one, I'd take the high-
| quality high-level overview over the stuff I can read the code
| for in 30 seconds.
| galaxyLogic wrote:
| The mental model really is not in the documentation, it is in
| the source-code modules how they interact with each other.
| Therefore I think OOP is a great way for building software.
| you need to create a mental imaginary world where the objects
| exist and interact. Then it makes sense.
|
| But like you say it is challenging, even finding good names
| is hard. But basically, can you write a story about the
| software working, that would allow readers to understand it?
| Create user-stories but not (only) about the users but
| understanding that any module is a "user" of its components.
| sz4kerto wrote:
| Ruthless end-to-end testing helps. Really hard on the short
| term, pays back on the long term. The suite is the
| specification.
| andrewfong wrote:
| That's a good way to prevent regressions but it's not the
| same thing as documentation or a mental model.
|
| The tests tell you what it's supposed to do, not how it does
| it.
| spankalee wrote:
| Or _why_
| naasking wrote:
| 100% this.
| bentobean wrote:
| I agree. What about a situation wherein you have a QA team
| that stands entirely apart from the devs and _they_ are the
| holders of the specification?
| lmm wrote:
| Don't do that?
| [deleted]
| bluGill wrote:
| Automate the tests. Sure the QA group holds the final say,
| but if you have a test that breaks you don't have to ask
| them. Or at least you have something to base a.conversation
| on.
|
| Note that requirements change, and bad tests will break on
| no requirements. Telling the difference is an exercise for
| the reader.
| bentobean wrote:
| The tests _are_ automated... by the QA team. Using a web-
| based tool for which none of the devs have access.
| ghotli wrote:
| I see two ways forward, both probably not great advice.
|
| 1. Ask for forgiveness and do whatever you think is
| prudent regardless of org structure.
|
| 2. Find a better org structure elsewhere :)
| bentobean wrote:
| I tend to agree. When I am capable of summoning the
| energy to care at this point, I make a run at #1. I have
| recently decided to actively pursue #2, as well.
| bluGill wrote:
| Write your own automated tests that you control.
| dathinab wrote:
| One of the most common tasks for QA teams is to write
| automated "outer layer" integration tests.
| jimmaswell wrote:
| You guys have QA teams that know how to code?
| mjevans wrote:
| Observationally, C and related languages frequently use headers
| as a documentation and API specification point. The internals
| aren't revealed, but the expectations are distilled into one
| easier to reference location. I know I appreciate that more
| than machine (only) readable blobs that expect auto-doc tools
| to split out the salient information with the technical
| signatures. The human curation also allows for ordering that
| matters to organics who must integrate related data, rather
| than just alphanumeric enumeration.
| layer8 wrote:
| > documenting the internals of a complex piece of software is a
| rare skill. A skill that most developers don't possess.
|
| I believe this is partially due to lack of training (technical
| writing rarely gets trained, and companies rarely invest in it
| as a skill), and sometimes probably also due to lack of
| exposure to high-quality documentation (which are role models
| you'd then want to live up to in your own documentation).
|
| In addition, companies often do not reward expending time and
| effort on writing documentation, because it isn't crucial in
| the short term. It's a bit like tech debt, just documentation
| debt.
|
| While writing good documentation isn't easy, the fact that it
| usually gets neglected is a cultural issue.
| cm2012 wrote:
| Friction is the enemy of productivity. I guarantee that most
| employees simply decide not to mention ideas for projects that
| could help the company because of the headache. They will do
| the minimum needed to keep their jobs.
| bentobean wrote:
| I know for a fact that this is true. Office Space was not a
| comedy - it was a documentary.
| windowshopping wrote:
| Do you work at Squarespace? This was exactly my experience
| there.
| bentobean wrote:
| No.
| dathinab wrote:
| Worse in my experience the quality of documentation (and tests)
| decreases the more a company is heading for insolvency for
| multiple overlapping reasons (at least for "startup| like
| companies).
|
| This means if you want to buy a struggling or insolvent company
| it will most likely have bad documentation and tests.
|
| But one of the most common steps when buying a struggling
| company is firing people (making the company fire people before
| buying it) and the selection of who goes and stays is rarely a
| good one. And it sets signals which make it mote likely for
| further important people to quit.
| smeagull wrote:
| > The "solution" that we've implemented is to require a
| detailed RFC document whenever you want to create or modify
| anything.
|
| I'd suggest just using the documentation. Manuals get used for
| doing testing and implementation. Using the documentation is
| like testing for it, and finding those problems right away is
| the best way I've found to ensuring that the documentation is
| correct and up to date.
| tomcam wrote:
| > And in a Google doc, no less.
|
| That whole process sucks and I'm sorry you have to go through
| it. It's probably the best solution of the time. My one
| question is... Isn't that why we have github issues?
| birdyrooster wrote:
| Bit rot is such a crappy metaphor.
| cjbgkagh wrote:
| Pretty sure it's a metaphor. Edit: parent comment was edited.
| birdyrooster wrote:
| A bad one
| cratermoon wrote:
| Why is it bad? What's a better one? What would a better
| one look like?
| tyingq wrote:
| "context rot" is more aligned with what they are
| describing, though that's not a metaphor. I can see
| bitrot being a bad fit, as it's decay of the actual thing
| rather than the surrounding context.
| bentobean wrote:
| In this context, I'm using "bit rot" as a metaphor in the
| same way that the author does:
|
| > The first is that it's a partial explanation for bitrot,
| the phenomenon where working software seems to deteriorate
| even when the code is untouched.
| birdyrooster wrote:
| Yeah I think it's a crap metaphor.
| DiggyJohnson wrote:
| Why? I'm not even saying you're wrong, necessarily...
| jacobsenscott wrote:
| I would call it developer rot. The code hasn't changed -
| and runs just as well (or badly) as the day it was
| written. It is the developers who somehow lose the
| ability (I don't know how - just read the code) to work
| with that code.
| icedchai wrote:
| It's not 100% accurate, but what metaphor is?
|
| I worked at an organization that had gone years without
| any updates: 10 year old dependencies that had never been
| updated, actively developed Python 2 code base in 2022,
| servers with 1000+ day uptimes and hadn't been updated
| since before the pandemic, on and on.
|
| It made it difficult to get anything done because you had
| to wade through a dumpster fire to make the smallest
| change.
|
| It certainly felt like bit rot to me.
| ahepp wrote:
| Well, the author describes bitrot as:
|
| > the phenomenon where working software seems to
| deteriorate even when the code is untouched
|
| If the code is touched, and fails to keep up with
| dependencies, I might argue that's more a case of bit
| neglect. But maybe that's semantics.
|
| I do agree that bitrot can be real though. Links rot off
| the web. If you're going to leave a project sitting on
| git without maintenance, you should probably capture the
| build context. Like shrink-wrapping a boat and storing it
| on land rather than just leaving it rusting away in the
| water for years.
| kortilla wrote:
| It felt like "bit rot" to you because that's how you
| defined "rot", but it's not how real rot works, which has
| nothing to do with changing.
|
| It's something that just breaks down without any changes
| in use case. If a wooden bridge rots away, nobody means
| that they were unable to add extra lanes. They mean it
| ceased to do what it previously did.
|
| "Code calcification" is likely a better approximation of
| what you're trying to convey.
|
| "Bit rot" would be better fitting to actual storage loss,
| but this industry chose poorly.
| feoren wrote:
| What do you suggest instead?
| nycticorax wrote:
| "Software subsidence"? "Digital subsidence"? The problem is
| that over time, the foundations upon which software is
| built decay, or need to be replaced...
| feoren wrote:
| I've worked with geologists and geological data and I had
| to look up "subsidence". I kinda had a feeling for what
| it meant, but it certainly didn't bring up any salient
| imagery nor help me understand the concept, which is
| kinda the point of a metaphor.
| codemac wrote:
| This is the similar to the classic "design concept" from "the
| design of design." It is the idea that must be shared between the
| software developer's minds to increase velocity, not software
| itself.
|
| The pressure you face in communicating this theory is time to
| market. I am obsessed with cleanroom, zero defect software, and
| literate programming. However, I have only worked on one project
| that these tools would have been relevant for. All others, the
| code/system moved on faster than the time it would take to get it
| communicated.
|
| The real measure of programming languages to me then becomes raw
| productivity. If we could make programming literally a 10% or
| less task, and documenting and communicating the 90%+ task, I
| think software companies would do very well. This bodes well for
| things like ChatGPT/copilot.
| yowlingcat wrote:
| This is a poorly written article by someone clearly projecting
| their own individual experience (or more clearly, the lack
| thereof) onto a giant world of professional phenomena.
|
| It is trivially easy to think of a structural counterexample to
| their primary point, which is how experienced senior engineers
| can inherit production codebases without documentation or the
| original team that wrote it and are able to successfully operate,
| understand and improve the entire codebase. Is there a gradient
| of pain? Yes. Is it possible to inherit a codebase that is
| falling apart, without the resources to get the ship back on
| track due to organizational issues? Absolutely.
|
| However, it doesn't change the fact that every year, significant
| quantities of senior engineers do something that the author does
| not believe to be possible. This isn't to say employee churn is
| particularly health for software companies; it's often not. But
| that really varies on a case by case basis.
| smeagull wrote:
| Honestly, I think all these layoffs hurt companies. Hiring is
| difficult and a slow process. Being unable to think of something
| profitable for devs to do inside your company is a sign that your
| management is being ineffective, or that ideas are being squashed
| inside your company.
|
| I'm skeptical that these companies were so bloated that they
| needed to lose large percentages of their headcount.
| twawaaay wrote:
| I specialise in joining teams with problems (inability to ship,
| reliability issues, performance issues, etc.)
|
| At least thrice I joined teams where all original developers and
| the manager was long gone. The situation for some reason looked
| very similar in all cases. The team is complaining about
| supposedly bad design that is preventing them from getting work
| done. But the system is also large and complex and they are
| unable to fix/rewrite it.
|
| Lo and behold, in two out of three cases I found that the
| original developers actually did pretty good job designing the
| system but in all cases any one of new developers understands
| only small part of the design and is unable to contribute changes
| across the entire system, there is a core of the system that
| nobody understands even though everybody uses and the team has
| made changes that are against original ideas of the system
| quickly creating more problems and exploding complexity.
|
| In all cases developers are only able to comfortably do some
| kinds of changes to the system that are routinely done (like
| adding new fields to forms, messages, etc.) that do not change
| how the system functions. Changing the basic structure of the
| system is a taboo.
|
| In essence, developers treat "legacy" code as if it was some kind
| of framework or language runtime -- to be used but never to be
| studied to have its internals understood. (I will never
| understand this approach to development)
|
| Making "non-standard" changes (yes, one team used this exact
| term!) pretty much always ends up with something resembling long
| archeological expedition with ancient traps, diseases and wild
| animals.
| soared wrote:
| Similarly I've seen sales and business teams start
| using/selling the product in a way it wasn't intended, and then
| difficulties arise when the product can't deliver what it's
| been sold to do.
|
| My currently company only did a specific niche in digital
| marketing but for sales goals needed to sell outside of that
| niche. Unsurprisingly, the product didn't work for those use
| cases.
| jacobsenscott wrote:
| > as much as possible, do not allow custom patterns/structures
|
| This is key. Most engineers want to build frameworks instead of
| applications. But leave the frameworks to the framework
| builders unless you were hired to build a framework.
|
| I can jump into a 10 year old rails code base and be productive
| right away if they stuck to the framework defaults. But
| invariably someone read some blog posts about how "rails
| doesn't scale" and built some rats nest of service
| objects/decorators/portsAndAdapters wtfs and it takes some
| serious code spelunking to figure out what's going on (hint -
| the end result is always creating or updating a database
| record, or making a single api call).
|
| I would argue microservices also encourage bitrot. You can
| leave some service running on some ancient libraries for years
| it doesn't need changes. But if you have a monolith where you
| constantly keep the dependencies up to date - everything stays
| up to date.
| twawaaay wrote:
| I am sorry I have quickly deleted that part with advices
| while you must have been responding to it.
|
| But this in my experience is truly important -- from the very
| start plan how a new member of the team is going to get
| _really_ proficient at the system. Ideally, they should be
| able to immediately recognise /reuse patterns/idioms they
| already know.
|
| People bitch and moan about boilerplate of Java applications.
| But when I join the project and I see controllers, models,
| repositories, entities, services, etc. I am pretty much at
| home as I know what all those things are doing and how they
| are connected to each other.
|
| Whenever developers decide to run with their intellectual
| desires it ends badly.
|
| I try to explain to people that typical business backends
| applications do not have custom problems. Most of those REST
| services are just supposed to receive query, run a piece of
| business logic, translate it into database call and then map
| response into specified format.
| bvirb wrote:
| We're working on a project that was inherited from previous
| engineers with no crossover. The first thing we did was write an
| end-to-end integration test suite for all the features we could
| find, which has grown more comprehensive over time. We still have
| old code we know is working alongside newer code written in a
| completely different style. We've been able to make major changes
| in tools and conventions over time while keeping progress steady.
| While we'd love to clean up all the code and have a single set of
| the latest conventions the decision is always refactor it now or
| maybe punt and we'll get to just delete it later, we usually go
| for the latter.
|
| I think the biggest benefit has been that the tests are isolated
| from the implementation details so we can always be sure
| everything is still working without having to refactor the tests
| themselves.
|
| There is still a learning curve for new people because they don't
| know which of the multiple conventions in the code are the
| presently-agreed-upon ones, and that is where we could probably
| use some better documentation, but I think the end-to-end tests
| really saved us from the idea that some knowledge is lost and we
| might need to start over.
| deathanatos wrote:
| If you haven't yet read it, go straight the source on this one
| and read Peter Naur's "Programming as Theory Building". The paper
| is _excellent_ , and an easy read. It's linked to by TFA.
|
| The consequences of the logic contained within Naur's essay are
| far stronger than I think the article understands them to be.
| DalekBaldwin wrote:
| It's not exactly an easy read -- whenever I recommend it to
| someone I also warn them that it contains the most passive
| sentence in the history of the English language: "Information
| obtained by a member of group A about the compiler resulting
| from the further modification of it after about 10 years made
| it clear that at that later stage the original powerful
| structure was still visible, but made entirely ineffective by
| amorphous additions of many different kinds."
|
| But the content more than makes up for the style!
| PathOfEclipse wrote:
| I've long felt that this is an issue unaddressed by the industry,
| though I've framed it as "knowledge acquisition and
| dissemination." As part of building a software product, the
| developers acquire deep knowledge of both the problem being
| solved and the system that solves it. The question then becomes:
| how do we efficiently capture and disseminate that knowledge?
|
| Capturing the knowledge is expensive both initially and over time
| as the software itself evolves. Disseminating it is expensive
| because it costs O(NM) time, where N is the number of people
| receiving the information and M is the average amount of time
| taken to absorb it. Dissemination is so costly, so we tend to do
| it in a "lazy" manner, meaning non-strict or as-needed.
|
| I think the best software systems I have worked on do a number of
| things:
|
| 1. They define enough ad-hoc proofs or invariants via unit tests.
| It's possible to work around these by simply disabling the tests,
| but, assuming your team hasn't devolved to that point, the tests
| are an excellent source of machine-readable and machine-
| executable knowledge.
|
| 2. The software is designed modularly so that one can operate on
| a part of the system without having to understand the entire
| system or worry that an unrelated part of the system is going to
| break.
|
| 3. The system is kept as simple as possible. The less knowledge
| one needs to accomplish the same task, the better! This is why
| KISS is so important. It's not for the person writing the code so
| much as the person debugging it a year later! As a corollary, a
| simple system tends to have less dependencies. Dependencies are
| actually costlier than people realize. Every dependency suffers
| its own bitrot problems and requires knowledge to handle them as
| part of maintenance.
| mike_hearn wrote:
| The lowest hanging fruit on any project is always
| documentation. Dissemination isn't O(NM) if your docs are well
| organized. At least, that's been my consistent experience over
| the years. I feel like our industry approaches documentation
| today the way it approached testing in the 1980s, before unit
| testing became widespread and culturally accepted/enforced.
|
| The problem is, just like how at the start you find lots of
| devs who don't want to write unit tests, you usually have a lot
| of devs who resist writing (and reading) docs. They'd rather
| interrupt someone to ask, or thrash around, or move onto the
| next task the moment some code is written.
|
| What helps:
|
| 1. Keep your docs alongside the source, in version control.
| This lets you gate code reviews/merges on docs being updated.
| Someone changed the way the code works but didn't update the
| docs? Then the PR is rejected until it's fixed.
|
| 2. Leading by example. If the TL for the project updates docs
| with their code changes, and visibly spends time gardening the
| docs, then they can't be accused of hypocrisy when they demand
| the same standards of others. Praise quality docs work when
| done by others.
|
| 3. Make the docs beautiful. Let people have pride in their
| work. It's so easy these days with great static site
| generators.
|
| 4. Answer questions by linking to the docs when possible, not
| ad hoc Slack/email/ticket discussions. If the docs don't tell
| someone the answer ask them to hold on for a moment, update the
| docs quickly and _then_ link to them. Obviously this requires a
| fast / lightweight turnaround process on review and merge of
| docs changes, but it can be done.
|
| 5. Avoid wikis. Quality requires ownership, someone who is
| directly responsible and can enforce their will. Wikis don't
| let you do that so they become a commons, and you end up with a
| tragedy. Use Markdown or equivalent in a VCS repo so you can
| refactor docs as they expand, make sure there's someone (e.g.
| yourself or a trusted lieutenant) who is expected to spend time
| paying down docs debt every so often.
|
| 6. Docs bugs should be a thing. File them. Expect people to fix
| them.
|
| 7. Make it clear that people are expected to read the docs, not
| just write them. If someone is regularly asking colleagues
| questions that they could have answered by reading, ensure
| that's treated like other performance issues i.e. their
| managers know and the need for improvement is clear to them.
|
| There's more that can be done but these things do work. The
| biggest problem is always passive resistance. Just like with
| unit testing or code review, if a culture of avoiding it grows
| then you'll find it hard to introduce, people will passively
| resist because it's not as much fun as just banging out more
| code.
|
| With these approaches the robustness of a program to churn can
| be improved a lot, you can scale teams up quicker and
| productivity improves.
| PathOfEclipse wrote:
| I believe what you're saying is true, I just haven't seen it
| in practice. I've long joked that the best documentation is
| the source code, and, really, it's not so much a joke as a
| truism. I've found that having readable code has resulted in
| the best documentation I've seen, and you don't have to worry
| about it being out-of-sync!
|
| I do believe documentation is important, but, having joined
| recently the largest corporation I've ever been a part of
| where I work on products of a larger size than ever, I am
| amazed to find documentation here to be as bad as, or even
| worse, than anywhere else I've worked, and most of it is
| simply outdated and irrelevant. I guess I need to work
| somewhere that does documentation well to believe in it more.
|
| I also consider myself a good technical writer.
| Unfortunately, it seems that many software developers don't
| share this skill, and this could be a major reason why so
| many resist writing it. I think a litmus test for this is
| whether someone prefers to share information via a document,
| or via an in-person meeting, a conference call, or a video
| recording of a meeting. Unfortunately, videos and meetings
| are not searchable, indexable, updateable, or quick to scan.
| But they seem to be preferred by the same people who don't
| want to write anything down.
| mike_hearn wrote:
| I've used that strategy a few times now, albeit mostly with
| "user" facing docs where the users were other developers.
| My current company is tiny but I wrote both internal and
| user facing docs from day one, so practicing what I preach.
| It's pretty useful. Writing stuff down is permission to
| forget and I often end up consulting my own docs,
| especially for procedures that only happen occasionally.
|
| In prior projects I did it too and success was mixed. On
| one hand, the docs we got out of it were objectively great,
| as in, users complimented us on the docs regularly, they
| were a source of competitive advantage and we could tell
| they worked because users would regularly show up
| successfully using advanced features of the product without
| having asked us any questions.
|
| On the other hand, creating a self-sustaining docs culture
| was hard. People would do it when faced with the
| expectation of doing it, and often the resulting docs were
| fine - tech writing isn't as hard as it's sometimes made
| out to be - but getting other devs to really buy into it
| and become docs evangelists themselves was much harder.
| Also of course you'd have inevitable team friction when
| someone really hated writing stuff and would have to be
| forced to do it, and then they'd submit min-viable or low
| quality work. No different to having a team member who
| refuses to write tests, of course.
|
| I think this can be solved with time, with cultural change
| in the industry. One problem, and please don't take this
| personally because I'm talking really generally here, is
| the view that source code is as good as documentation. I
| feel like lots of devs convince themselves of this because
| they like writing code and don't like writing docs, but I
| never found it to be true in practice. Any time I have to
| use an API or tool that has tons of missing or obsolete
| docs my heart sinks, partly because rummaging through
| source code is slow and tiring compared to reading a well
| written document, and partly because it tends to be a proxy
| for low effort work in other ways. The claim that docs are
| always obsolete is self-fulfilling. If you don't put in
| place procedures to ensure they're kept fresh and good then
| they'll rot, and then that's used as a justification for
| not putting in the effort, so it's a vicious circle. Again
| by analogy to testing, competent teams don't just let devs
| comment out tests instead of updating them, it's just not
| culturally accepted in high-functioning environments. Same
| can be true of docs.
| jonnycat wrote:
| Of course, this often works the other way as well: a project is
| failing or stalled by the "wrong" theory being saturated through
| the team, and it can only be fixed by replacing individuals and
| developing new mental models of the problem.
| holtkam2 wrote:
| Can someone help me understand why Twitter had not crumbled from
| the overwhelming weight of software decay after 50% of the staff
| was laid off? I'm not being sarcastic, it's just honestly hard
| for me to understand...
| AnimalMuppet wrote:
| People leaving doesn't _cause_ decay. Decay happens over time -
| usually not just in a month.
|
| People leaving means that there's nobody there who can fix the
| decay when it happens, though.
| DethNinja wrote:
| Did they ever disclose exact amount of engineers that were laid
| off? Twitter laid off lots of people but I doubt majority of
| them were engineers.
| JonChesterfield wrote:
| Patience. It hasn't been long, and they probably left a test
| suite running under CI. Ideally it doesn't push to production
| without the tests passing and noone is left who knows how to
| disable it.
| ranting-moth wrote:
| I mean, this test is really annoying and it's testing a case
| that I'll guarantee 100% won't happen often in production.
| bakuninsbart wrote:
| There's a number of plausible scenarios:
|
| 1. It takes more time for the rot to show.
|
| 2. Twitter devs were amazing at building really solid software
|
| 3. The culture of twitter was really that 1 person did the work
| and 9 others watched, enough of the productive devs stayed.
|
| 4. The effects of rot are overestimated.
|
| I have no insight into how twitter worked as a company before
| Musk's takeover though, and I still think it is quite likely
| major problems will occur in the next couple of months.
| NordSteve wrote:
| > What keeps the software alive are the programmers who have an
| accurate mental model (theory) of how it is built and works. That
| mental model can only be learned by having worked on the project
| while it grew or by working alongside somebody who did, who can
| help you absorb the theory. Replace enough of the programmers,
| and their mental models become disconnected from the reality of
| the code, and the code dies. That dead code can only be replaced
| by new code that has been 'grown' by the current programmers.
|
| While it's possible that there are systems out there that can't
| be understood, it's plainly false that - to paraphrase - code
| dies and must be replaced when the original developers are gone.
|
| A couple examples from my direct personal experience:
|
| 1. I work every day on code that's 20-30 years old, in use in
| production by hundreds of millions of people every day. There's
| no one around who worked on it originally. And yet it works, 100s
| of pull requests a day, and customers continue to pay us to use
| it.
|
| 2. I took on a volunteer project this fall, working on a ~million
| line distributed system that runs on clients at about 30 sites
| and integrates with 3 cloud services. No original developer is
| working on it. For the client code, it took me about a day to
| understand how it works and make meaningful changes.
| marmetio wrote:
| The author starts with the implicit assumption that the
| companies in the target audience will not be able to restart
| the metaphorical engine, which is why they recommend letting it
| idle. They tactfully don't call attention to the fact that not
| everyone has that problem.
| compiler-guy wrote:
| For both of these cases, the article author would say that the
| original developers have been replaced by newer developers who
| have acquired the theory.
|
| The solution to the problem, at least according to the article,
| isn't "no churn ever", but "low enough churn that the newer
| developers acquire an accurate mental model of the system."
___________________________________________________________________
(page generated 2023-01-10 23:00 UTC)