[HN Gopher] My approach to building large technical projects (2023)
___________________________________________________________________
My approach to building large technical projects (2023)
Author : mad2021
Score : 316 points
Date : 2025-10-10 03:45 UTC (19 hours ago)
(HTM) web link (mitchellh.com)
(TXT) w3m dump (mitchellh.com)
| tomhow wrote:
| Previously:
|
| _My approach to building large technical projects_ -
| https://news.ycombinator.com/item?id=36161397 - June 2023 (27
| comments)
| davidkunz wrote:
| I have huge respect for Mitchell, it's impressive what he
| achieved.
|
| I agree with all the points of this article and would like to add
| one: Have a quick feedback loop. For me, it's really motivating
| to be able to make a change and quickly see the results. Many
| problems just vanish or become tangible to solve when you
| playfully modify your source code and observe the effect.
| amenghra wrote:
| If you have the time, what Bret Victor's talk Inventing on
| Principal. The talk covers feedback loops.
| https://www.youtube.com/watch?v=PUv66718DII
| chrisweekly wrote:
| YES that is one of the all-time most inspiring talks I've
| ever seen. DX is so important. I got a taste for this kind of
| thing when I first encountered LiveReload (circa 2012?) and
| radically upgraded my and my team's webdev workflows.
| ramon156 wrote:
| Would you say that testcases help here? I've been thinking
| about applying e2e tests on any bugs I find so I know they're
| fixed
| zingar wrote:
| E2E tests in a high ratio to other tests will cause problems.
| They're slow and brittle and become a job all on their own.
| It's possible that they might help at the start of debugging,
| but try to isolate the bugs to smaller units of code (or
| interactions between small pieces of code).
| skydhash wrote:
| And I would add that e2e tests should be more about the
| businesses rules. Making sure everything is there for a
| specific flow and not caring that much about the intricacy
| of things. And such, it should really be part of Ops, not
| Dev.
|
| Quick feedback with unit tests can help. It can be a pain
| to decouple stuff so you can test them better, but it's
| worth it IMO.
| MoreQARespect wrote:
| Hermetic e2e tests (i.e. ones that can run offline and fake
| apis/databases) dont have that problem so much.
|
| They also have the advantage that you can A) refactor
| pretty much everything underneath them without breaking the
| test, B) test _realistically_ (an underrated quality) and
| C) write tests which more closely match requirements rather
| than implementation.
| sjdjsin wrote:
| > i.e. ones that can run offline and fake apis/databases
|
| I can see a place for this, but these are no longer e2e
| tests. I guess that's what "hermetic" means? If so it's
| almost sinister to still call these e2e tests. They're
| just frontend tests.
|
| > A) refactor pretty much everything underneath them
| without breaking the test
|
| This should always be true of any type of tests unless
| it's behavior you want to keep from breaking.
|
| > B) test realistically (an underrated quality)
|
| Removing major integration points from a test is anything
| but realistic. You can do this, but don't pretend you're
| getting the same quality as a colloquial e2e tests.
|
| > C) write tests which more closely match requirements
| rather than implementation
|
| If you're ever testing implementation you're doing it
| wrong. Tests should let you know when a requirement of
| your app breaks. This is why unit tests are often kinda
| harmful. They test contracts that might not exist.
| westurner wrote:
| > _try to isolate the bugs to smaller units of code (or
| interactions between small pieces of code)._
|
| This is why unit tests before e2e tests.
|
| It's higher risk to build on components without unit
| tests test coverage, even if the paltry smoke/e2e tests
| say it's fine per the customer's input examples.
|
| Is it better to fuzz low-level components or high-level
| user-facing interfaces first?
|
| IIUC in relation to Formal Methods, tests and test
| coverage are not sufficient but are advisable.
|
| Competency Story: The customer and product owner can
| write BDD tests in order to validate the app against the
| requirements
|
| Prompt: Write playwright tests for #token_reference, that
| run a named factored-out login sequence, and then test as
| human user would that: when you click on Home that it
| navigates to / (given browser MCP and recently the Gemini
| 2.5 Computer Operator model)
| mhaberl wrote:
| This perfectly aligns with my experience. Every large project I
| have worked on showed a clear correlation between the ease of
| setup and running and the number of problems on the project,
| like bugs and missed deadlines.
| jebarker wrote:
| Totally agree. I work in LLM training software and I believe
| progress in the field is actually much slower than it should
| be because of the excruciatingly long feedback loops involved
| in development. The software stacks are deep and abstract and
| much of the testing involves full integration tests that take
| a long time to spin up.
| reddit_clone wrote:
| Couldn't agree more. Quick feedback is so important, it
| requires its own post.
|
| When I want to try/fix something, if the setup itself takes
| hours, I lose heart and move on.
|
| Thats why I love lisp (or anything with a decent Repl). Instant
| gratification.
| ericmcer wrote:
| seriously, especially for personal projects.
|
| The second you lose motivation the whole thing poofs into non-
| existence, so making it enjoyable is almost the most important
| facet.
| geokon wrote:
| I feel there is a balance to strike with rushing to a demo. ..
| (maybe it's an implicit tension between the satisfaction of
| finishing a component and getting a demo)
|
| I think this is where the choice of language makes a big
| difference. In Clojure, the difference between a "component" and
| a separate library/application is literally just adding a
| `deps.edn` file and then pointing to the directory from the
| parent project.
|
| I think breaking the project in to small achievable goals is very
| sensible. But if you take the extra time to make the component
| stand on its own as a mini-lib .. it's very satisfying. For
| instance I had to write a "component" that would read some
| GeoJSON and segment it (it's took me a couple of days and was
| mostly a wrapped around GDAL or something). I could have hacked
| together a solution to rush to a demo - but instead I made a
| small little library out of it. When I was done with it, I had a
| sense of "I made a thing". To be clear.. it's still kind of ugly
| and I would be horrified if someone else tried to use it and
| submitted PRs.. but it's also not a coupled tangle of code in my
| larger codebase.
|
| as he says "Build for yourself" - the library/application should
| only do what you need
|
| By contrast, if I was working in C++ making an API and decoupled
| library would be such a chore that'd never bother
|
| The most important aspect is that this all ends up not just much
| more satisfying at every step - but it makes your code incredibly
| decoupled and refactorable. The more you rush to a demo the more
| your code is coupled and hard to refactor.
| ArcHound wrote:
| This is how agile should look like. Focused, iterative and always
| functional.
| Copenjin wrote:
| One of the few people that has done consistently good open source
| projects, with very clean and clear vision of what the project
| should accomplish.
| iamflimflam1 wrote:
| For me just starting is the best way. So many people see a big
| project and get trapped in analysis paralysis.
| hk__2 wrote:
| Oh, starting is easy. It's finishing that's hard.
| iamflimflam1 wrote:
| Well yes - best to make that part somebody else's problem :)
| iamflimflam1 wrote:
| Joking obviously!
| HeyLaughingBoy wrote:
| This is where a Definition of Done statement becomes
| important
| mrheosuper wrote:
| >This is an area where I think experience actually hurts. I've
| seen senior engineers get bogged down building the perfect thing
| and by the time they get a demo, they realize it sucks. The
| implementation doesn't suck, but the product or feature itself
| actually sucks.
|
| This resonates with me. Sometime i want to "turn off" my brain
| and write shitty code.
|
| Back in the day, i made a lot of toy project. Sometime all the
| source code is in single file. No respect to modularity. But it
| was fun, and it worked. Now just try to finish a toy project seem
| much harder than ever.
| Bukhmanizer wrote:
| I think this is why often new languages get so hyped early on.
| A bunch of inexperienced people get to leave all the boring
| "best practices" back in their old languages
| lordnacho wrote:
| Isn't this called the second system problem? When you've done
| it before, you feel like you need all the bells and whistles?
| mrheosuper wrote:
| You are right, this is the first time i've heard about
| "second system problem". Never know it has its own term.
| theryan wrote:
| The Mythical Man-Month by Fred Brooks is still a good read
| today and goes into depth about this topic. He called it
| the second system effect and considered it to be the most
| dangerous system an experienced developer will create.
|
| His 'No Silver Bullet' theory may or may not stand up to
| what AI is doing today as well.
| skeeter2020 wrote:
| From what I've seen it's standing up. His original
| statement/hypothesis is often misrepresented.
|
| "there is no single development, in either technology or
| management technique, which by itself promises even one
| order of magnitude [tenfold] improvement within a decade
| in productivity, in reliability, in simplicity."
|
| If he had also included "volume" then AI would have
| disproved him! More anecdotally, the hands-on experiences
| of senior+ developers seem to firmly fall into the
| accidental camp, with the essence of software problem
| solving getting marginally easier as you'd expect with a
| new, powerful tool, but far from "solved".
| dasil003 wrote:
| I agree with the quote as stated, but I would refactor it
| for a more powerful insight.
|
| I do believe order of magnitude improvements in
| productivity and reliability are possible, but they don't
| come from technology or management technique, they come
| from simplicity. The simplest possible thing that gets
| the job done can be infinitely more reliable than
| whatever baroque contraption comes out of typical fog-of-
| war enterprise environments. The trick is having the
| judgement to understand what complexity is essential and
| how to distill things down to the most valuable essence.
| This is something AI will never be able to do, because
| the definition of value is in the eye of the human
| beholder.
| nemetroid wrote:
| It is interesting though that when he mentions AI as one
| of the non-silver bullets, one of the arguments is that
| the AI models of the time were problem-specific and not
| easily transferable.
| Cthulhu_ wrote:
| I think you can practice it; do some coding excercises, e.g.
| Advent of Code is coming up again, usually it's only the later
| ones that require some optimization or clever algorithms (...I
| never got that far). Or a constrained environment, you can't
| write long code in pico-8. Or time limited, like a hackathon or
| game jam.
| CaptainOfCoit wrote:
| Game jams and hackathons are a fun and easy way to get a lot
| better at scoping and saying no, and being able to predict
| what works or not once everything is in place.
|
| It's a playful environment with low stakes too, compared to
| working in a startup, so really advice new programmers to
| participate in order to learn faster.
| saaspirant wrote:
| LLMs (Cursor) have pretty much solved this problem for me. I
| design the DB tables manually and let it run somewhat wild with
| implementation.
| meander_water wrote:
| Great read, but I was expecting something different based on the
| title.
|
| This sounds like his approach working on personal projects. I'm
| really curious about large technical team projects though. What's
| the best approach to getting stuff done and making sure everyone
| is working towards the same goal etc.
|
| After 15 years I have yet to see a technical project that hasn't
| run over budget, over time, under delivered or burnt people out.
|
| I'm sure there are people out there with counter examples who
| know exactly how to deliver projects at a massive scale. Any
| links/suggested reading would be appreciated!
| makeitdouble wrote:
| > over budget, over time, under delivered
|
| TBH these are fine in my book.
|
| "over budget" is only an issue if there's really no more money,
| and I think it's pretty rare for IT projects. Most of the time
| it's just someone complaining the estimate was off.
|
| "Over time" is the same, it's an issue if there was a real
| deadline, but the best practice is not plan unflexible events
| (e.g. do a huge PR campaign with the date on it) before it's
| basically done.
|
| "under delivered" is a matter of expectations, the real pointis
| that it was delivered at all.
|
| > or burnt people out.
|
| This one is not like the others. People shouldn't burn out over
| bullshit deadlines.
| dboon wrote:
| well, the project in question is ghostty, which definitely
| qualifies
| BrkAway21 wrote:
| Maybe I'll get maligned for saying this, but as someone who's
| managed and successfully delivered software projects of all
| sizes, I've become somewhat of a convert to the Scaled Agile
| Framework. I treat it exactly as intended, as a framework. It's
| a strawman to adapt based on context, and what I value most is
| how it led me to explore deeper source material and form my own
| conclusions.
|
| Over the years, I've learned that true success starts with
| clearly understanding why you're building something. Without
| clear goals, it's impossible to prioritize or even know where
| to start. That clarity drives better sequencing, and sometimes
| the wisdom to not build something at all.
|
| The next critical factor is empathy. You have to see through
| your customer's eyes and validate that what you're building
| actually solves their problem. That doesn't mean giving them
| everything they ask for, but rather understanding their pain
| points well enough to deliver real value.
|
| Ultimately, most projects go over budget or underdeliver
| because teams spend too much time building the wrong things. If
| we instead focused on continually steering toward desired,
| valuable functionality (things people genuinely want or will
| pay for), more software projects would appear like successes.
| cloudexpat wrote:
| I remember reading this when it was published, and the milestone
| breakdown Mitchell describes was exactly what got us from a
| theoretical solution to a production platform.
|
| That combined with launching an MVP first rather than the
| "complete" vision. Shipping early meant we avoided the trap of
| spending months perfecting features no one actually used.
| SteveLauC wrote:
| Really enjoy the post, thanks for sharing.
|
| > My goal with the early sub-projects isn't to build a *finished*
| sub-component, it is to build a good enough sub-component so I
| can move on to the next thing on the path to a demo.
|
| This is so enlightening. And I realized that to do this, one has
| to "skip" something. Other folks mention they ignore code
| modularity when doing this, I don't think I will do that, keeping
| code clean and reading/working in such a codebase actually make
| me satisfied and _motivated_. For me, I am going to "skip"
| algorithms, data strucuture and performance.
|
| So the point here is probably, we should skip things, but if a
| thing motivates you, it should not skipped?
| Vipsy wrote:
| An interesting consideration is how the chosen modularization
| approach can impact onboarding time for new contributors. A well
| structured breakdown might not just aid initial development
| speed, but also reduce ramp up friction for future team members
| or external collaborators. This is an impartant factor often
| under-estimated in solo-driven projects.
| piltdownman wrote:
| //Decompose a large problem into smaller problems. Only solve the
| smaller problem enough to progress on a demo... ...then continue
| to iterate on more functionality. Make demos as frequently as you
| can. Prioritize functionality that enables you to adopt your own
| software Go back and iterate on each component as needed for
| future improvements//
|
| So in essence we have: Empirical Process Control, Self-
| Organization, Collaboration, Value-Based Prioritization, Time-
| Boxing, and Iterative Development.
|
| Is this just solo-SCRUM or am I missing something here?
| jiggunjer wrote:
| For me the best demo is a test module lighting up all green.
|
| How do make this into a sexy image for management. Sure, business
| logic is stubbed, but my carefully crafted strongly typed
| interfaces all mesh together! Imagine the future dividends!
| poszlem wrote:
| This isn't an "everyone who struggles to finish projects or is a
| perfectionist has it" kind of post, but if this has been a long-
| term pattern for you, one that's caused real suffering, lost jobs
| or contracts, or kept you from finishing important work - please
| consider getting checked for ADHD. It took me 42 years to get
| diagnosed, and starting on stimulants genuinely changed my life
| and pretty much solved those issues for me.
| dorianniemiec wrote:
| Interesting, especially the "Build for Yourself" section!
|
| Building software for yourself can help you solve a problem you
| have. If you're also using the software yourself, you can fix
| bugs in the software. I found some bugs in a web server I'm
| building by trying to use it myself.
| gbuk2013 wrote:
| What I really want it some advice on providing reliable estimates
| to the business when delivering a large technical project. :)
| kkukshtel wrote:
| Building demos as a key part of development is really key imo.
| Demos act as a half-step between working on the software
| (programming) and writing about the software (writing). Demos act
| more as a way to continually validate your own theses about what
| a project SHOULD do, and act as nice feedback mechanism as you
| continue to work. They are also long-living, so when you break
| something you can see you broke it and continue the feedback
| system again.
|
| I do this as part of the work on my own game engine:
|
| https://github.com/zinc-framework/Zinc.Demos/tree/main/Zinc....
___________________________________________________________________
(page generated 2025-10-10 23:00 UTC)