[HN Gopher] Ask HN: What is the best code base you ever worked on?
       ___________________________________________________________________
        
       Ask HN: What is the best code base you ever worked on?
        
       And what made it so good?  Was there someone enforcing good
       practices top down? Just being in a group of great engineers? Or
       something else?
        
       Author : pcatach
       Score  : 496 points
       Date   : 2024-06-28 08:40 UTC (1 days ago)
        
       | cqqxo4zV46cp wrote:
       | A blank slate, so I can make the first mistake :)
        
         | escapecharacter wrote:
         | I agree, the best code base is always a zero-line code base.
        
           | alex_suzuki wrote:
           | After that first line, it's all downhill from there... :-)
        
         | azangru wrote:
         | So true!
         | 
         | I spend so much time obsessing over how what I am about to
         | write ties in with what has already been written; or fuming
         | over the stupidity of earlier decisions (usually made by
         | myself). A blank slate is incredibly refreshing.
        
       | mulholio wrote:
       | Although I'm only on job three and have not had that much
       | involvement with open source, I think my current employer (Attio)
       | has one of the best codebases I've seen.
       | 
       | Qualitatively, I experience this in a few ways: * Codebase
       | quality improves over time, even as codebase and team size
       | rapidly increase * Everything is easy to find. Sub-packages are
       | well-organised. Files are easy to search for * Scaling is now
       | essentially solved and engineers can put 90% of their time into
       | feature-focused work instead of load concerns
       | 
       | I think there are a few reasons for this:
       | 
       | * We have standard patterns for our common use cases * Our hiring
       | bar is high and everyone is expected to improve code quality over
       | time * Critical engineering decisions have been consistently
       | well-made. For example, we are very happy to have chosen our
       | current DB architecture, avoided GraphQL and used Rust for some
       | performance-critical areas * A TypeScript monorepo means code
       | quality spreads across web/mobile/backend * Doing good migrations
       | has become a core competency. Old systems get migrated out and
       | replaced by better, newer ones * GCP makes infra easy * All the
       | standard best practices: code review, appropriate unit testing,
       | feature flagging, ...
       | 
       | Of course, there are still some holes. We have one or two dark
       | forest features that will eventually need refactoring/rebuilding;
       | testing needs a little more work. But overall, I'm confident
       | these things will get fixed and the trajectory is very good.
        
       | kleton wrote:
       | google3, all the devex tooling was taken care of by other teams.
       | Tons of useful library functions available to import, accumulated
       | over decades.
        
       | chrisvenum wrote:
       | My favourite projects are small, with very focused goals and
       | features.
       | 
       | I have a Laravel project that I have maintained for a customer
       | for seven years. The app is straightforward and allows users to
       | create portals that list files and metadata, such as expiration
       | dates and tags.
       | 
       | Every other year, they ask me to add a new batch of features or
       | update the UI to reflect the business's branding. As the app is
       | so small, I have the opportunity to review every part of the app
       | and refactor or completely rewrite parts I am not happy with.
       | 
       | It is a joy to work on and I always welcome new requests.
        
         | gepardi wrote:
         | This sounds like wonderful! Maybe one day I can have a few of
         | these "babies" that I can lovingly raise, providing joy to both
         | the dev and the customer.
        
       | 2Pacalypse- wrote:
       | Any codebase that I had complete control over.
       | 
       | No, but more seriously, I've found that familiarity with the
       | codebase is more important than having it be perfectly
       | engineered. Once you're really familiar with the codebase, you
       | know where dragons be, and you can make changes more easily. And
       | God (PM) forbid, if you ever find yourself with some extra free
       | time you might even reduce the size of dragons over time.
       | 
       | This brings me to my final point. Any codebase that I really
       | enjoyed working with was the one that was constantly evolving. I
       | don't mean rewriting everything from scratch every few months,
       | but as long as I have permission (and time) to refactor the
       | things that have been bothering me for months as patterns emerge,
       | I'm a happy bee.
        
         | sam_lowry_ wrote:
         | > Any codebase that I had complete control over.
         | 
         | No excuses. Code ownership is important. Sometimes it works for
         | a team, sometimes only for individuals.
         | 
         | But not having to submit to core teams, architects and self-
         | proclaimed experts of all kinds is a blessing.
         | 
         | I now work for an organization that discourages code ownership,
         | and it struggles on many fronts:                       1. core
         | teams are dysfunctional             2. people find niches and
         | stick to them             3. top talent is leaving, although
         | pay is good and business creates real value for citizens
         | 4. there is virtually no horizontal communication
         | 5. mediocre ones rise to the level of their incompetence and
         | infest the lives of others             6. and so on and so
         | forth...
         | 
         | And I think the root cause of all this is lack of individual
         | (code) ownership.
        
           | Aeolun wrote:
           | I've had exactly the same issues but because I couldn't
           | change anything without getting approval from 16.5 code
           | owners on every PR submitted. It's a real pain if you start
           | modifying your coding for 'least code owners hit' instead of
           | 'best architecture'.
        
             | rsanek wrote:
             | I like how my workplace does it -- there are rigorous
             | codeowners and usually you only need approval from 1-2. if
             | you do need approval from 5+, you can request a select
             | 'super' codeowner review which will approve it for all.
        
           | ownagefool wrote:
           | I think core teams being helpful or harmful really comes down
           | to the individuals.
           | 
           | The problem is at this level, most orgs don't have anyone to
           | really judge or contest competency, so they hire the salesmen
           | rather than the doers and when they don't, they tend to cheap
           | out and just get inexperienced people.
           | 
           | Logically it makes a bunch of sense, though.
           | 
           | Why rebuild yet another platform? Why is your central
           | platform bad? Usually it's not self-service, sometimes it's
           | because it's built in cumbersome ways, other times its
           | because it actually enforces good standards on you rather
           | than just giving app your apps admin.
           | 
           | It's difficult for the person who hires the core team to
           | differentiate between those complaints, unless they
           | themselves both have the technical competency and the empathy
           | to really understand the problem. They usually don't.
           | 
           | Point being, done well, it's great, but most folks can't do
           | it well.
        
         | threatofrain wrote:
         | Also nice is when an entire community has agreed to architect a
         | codebase more or less the same. You're basically psychic when
         | that happens.
        
         | blitzar wrote:
         | > Any codebase that I had complete control over.
         | 
         | Anyone other than myself would instantly observe it as the the
         | worst codebase they have ever seen.
        
           | sam_lowry_ wrote:
           | For that, there is a saying: "code as if your kids will
           | maintain it".
           | 
           | But I think it does not convey the right meaning. When I code
           | something I will have to maintain for a long time, I try to
           | make it as simple as possible for my future, older, less
           | motivated and weary self.
           | 
           | The worst codebases are written by people who landed the gig
           | a few months before and do not expect to stay around longer
           | than a year or two.
        
         | null_glyph wrote:
         | This is very much true. Initially when I joined the industry I
         | used to work for a product from its inception. So I was aware
         | of what section of the code affects what part of the product.
         | Although I hadn't worked on all of those, I kept an eye for the
         | all the changes that were coming in. I knew where I should look
         | immediately a bug is reported even if it is not something
         | related to my line of work.
         | 
         | Recently I switched teams and now I find myself taking up bugs
         | that are only related to my line of work. Not being familiar
         | with the codebase decreases productivity and wants you to rely
         | on other people in the team for most of the time.
        
         | sgbeal wrote:
         | > Once you're really familiar with the codebase, you know where
         | dragons be, and you can make changes more easily.
         | 
         | This is an interesting, and often overlooked, point. A month or
         | two ago someone asked us, the Fossil SCM maintainers, if we'd
         | be open to them refactoring the tree to something which better
         | matches modern sensibilities (i'm paraphrasing here). Despite
         | its many proverbial dragons, the long-time maintainers are
         | comfortable with it and know where those dragons are and how to
         | defeat them (or, in some cases, sneak around them), so, despite
         | our collective geek tendencies to prefer "perfect code," we're
         | happier with the dragons we know than those we don't. (That's
         | not to say that fossil's code is awful, but its structure
         | varies significantly from what one might write today if one was
         | to start such a project from the ground up.)
        
         | nevi-me wrote:
         | I also echo the code ownership part.
         | 
         | I have a part-time gig where I maintain accounting software for
         | a former client of mine. It takes up a few months' weekends a
         | year.
         | 
         | I wrote about 60-70% of it when I was working for the owner of
         | the software. It's something where as long as the client's
         | happy, and they get new integrations and updates on time, they
         | could keep using it for a decade longer.
         | 
         | I had almost complete ownership of the architecting of the
         | software. It's broken down into a few microservices (think
         | database, core business logic, reporting, auth, logging etc).
         | The best thing I did at the time was pushing to use gRPC even
         | though management felt it was too new tech.
         | 
         | The UI is in Angular, pain-free periodic upgrades. I've even
         | rewritten some perf-sensitive code in Rust, and everyone's
         | happy with snappier calculations.
         | 
         | The code hygiene is relatively good.
         | 
         | The only downside's that if someone else were to take over the
         | code, they'd struggle (it's one of those things where I'm
         | wearing many hats). I've been fortunate to be a professional
         | accountant who moved into software engineering, so everything
         | makes sense to me.
        
         | KronisLV wrote:
         | > Any codebase that I had complete control over.
         | 
         | This is probably one of the pillars of good codebases, or at
         | least decoupling the bits that you don't control as well as you
         | can (this includes external services). I remember needing to
         | write a wrapper around another JWT library, but because it was
         | quite important, I aimed for >95% test coverage, some of the
         | tests acted as documentation for how to use the code, there was
         | also a nice README, there was CI configuration for pushing the
         | build artifacts to Maven and suddenly even managing dependency
         | updates became easy because of the test suite. Years later,
         | it's been integrated in a bunch of services and "just works".
         | 
         | Come to think of it, things always get less pleasant once you
         | add a bunch of complex dependencies and libraries/frameworks.
         | Need to make a few RESTful Web APIs in Java? Something like
         | Dropwizard will probably give you fewer headaches than Spring
         | (or Spring Boot) and all of its inherent complexity, in the
         | case of the former you might even need to do configuration in
         | XML and that has honestly never been pleasant. If you need to
         | integrate with a bunch of other stuff and want something
         | opinionated, going for Spring Boot will make sense then, but
         | for simple use cases it's overkill. Same for ASP.NET, Ruby on
         | Rails, Laravel and many others, while they might be easier to
         | use, updates (especially across major versions) and breaking
         | changes will give you a headache and just add a bunch of churn
         | to the project.
         | 
         | Similarly, if you need a message queue, externalizing that into
         | RabbitMQ might make a lot of sense, same with storing files in
         | an S3 compatible store, using something like Redis or Valkey
         | for key-value storage, as well as not trying to shove too much
         | logic into your RDBMS. Just pick whatever tool feels best for
         | the given task at hand, instead of shaping things into what
         | they're not (using the database for blob storage, for example),
         | unless you have a whole bunch of constraints to contend with.
         | Otherwise, sometimes you just get the worst of both worlds,
         | like needing to use Oracle for a project, not having easily
         | launchable local environments (because Oracle XE doesn't
         | support some features), having to share DB instances with
         | others during development and also running into weird obscure
         | issues like DATABASE LINK queries taking 100 times longer in
         | some cases, even when executing the same SELECT query on the
         | remote DB works without issues.
         | 
         | To not go into a rant, I'd sum it up like this: be in control,
         | isolate the things that you cannot control, pick the correct
         | technologies, do the simplest thing that you can get away with
         | without overengineering and think about the person who'll have
         | to maintain, debug and grow the system in the following months
         | and years (which might also be yourself, sans knowledge about
         | what the code did, if you don't make it explain itself or don't
         | comment the non-trivial stuff).
        
         | idrios wrote:
         | > And God (PM) forbid, if you ever find yourself with some
         | extra free time you might even reduce the size of dragons over
         | time.
         | 
         | Honest question, what is the company like where you can do
         | that? Everywhere I've worked (only been working in industry for
         | 6 years) has had such rigid agile development that even when I
         | do find myself with free time, there's no flexibility to work
         | on things that haven't been assigned to you and the best I can
         | do is work on profiling/debugging tools.
        
           | arp242 wrote:
           | Usually at smaller companies with just a handful of
           | developers. Can be a "start-up", or just a smaller business
           | running fine with small teams. Or sometimes in a larger
           | company where there's a small team doing its own thing.
           | 
           | There's up-and-downsides to everything. The pay is usually
           | less, and "fewer processes" often also means fewer processes
           | surrounding HR and such. This can mean you're subject to the
           | whims and mood swings of one person. I was once fired over a
           | pretty far-fetched misunderstanding, but 1) they didn't
           | mention it because "I should know myself", so I didn't really
           | have a chance to clear it up, and 2) when I got fired they
           | agreed it was a complete misunderstanding and that they made
           | a huge mistake, but fired me anyway because that was the road
           | already taken now. Great ... thanks ... It's hard to imagine
           | that happening at a larger company with an actual HR
           | department and stuff.
           | 
           | Also: a Wikipedia-esque "be bold" attitude helps. Obviously
           | refactoring things just for the craic isn't a good idea, but
           | if there's something that can really make an impact then I'd
           | just do it and send a PR, regardless of what I've been
           | assigned. Obviously "refactor everything" on your own
           | initiative isn't a good idea, but usually there's tons of
           | smaller stuff that really helps: "introduce new
           | function/package that helps reduce boiler plate", "split up
           | this difficult to understand function", things like that.
           | 
           | Most PMs will realize they don't have a 100% in-depth
           | understanding of the code, and will trust you when you say it
           | helps you do your work better. Usually it's fine as long as
           | you apply proper judgement and don't spend too much time on
           | it. That said, there are some places where this really isn't
           | appreciated regardless. IMHO that's pretty dysfunctional and
           | when I start looking for somewhere else to work.
        
             | james_marks wrote:
             | Counter-point: whitespace PR's, refactors I didn't ask for,
             | aren't attached to a larger initiative, etc actively annoy
             | me as an owner.
             | 
             | Every PR takes time to manage, compounds complexity, and
             | carries risk. We only get so many out a week, and I want
             | them to count.
             | 
             | Maybe it sometimes goes in hand with disfunction, but I'm
             | guessing there's more going on there.
        
               | arp242 wrote:
               | > apply proper judgement
        
         | lanstin wrote:
         | Your successors just hope you can only remember 3 things at
         | once so you end up with elegant designs rather than one of
         | those monsters who can remember "to add a new wrinkle to this
         | feature, just edit these 25 files."
        
       | ysris wrote:
       | Considering the biases, the one I wrote for the company I
       | created.
       | 
       | When we have the opportunity to be in this context, keeping in
       | mind what bothered us in the codebases with which we were able to
       | work in the past, we can force ourselves not to reproduce the
       | same errors. Like the unmaintained unit and integration tests,
       | the lack of refactoring, other developers that use fancy
       | technologies instead of simpler concepts more for the opportunity
       | to play with technologies than real need..
       | 
       | And also, I guess, because we are more aware that the code is a
       | reflection of the company that we want to have, that the simpler
       | the better is a key point when we need to debug.
        
       | anonzzzies wrote:
       | Generally code that hasn't been tested commercially. Unencumbered
       | by pesky client driven features, just code for dreamt up features
       | that are fun to code but perhaps will never be used.
        
       | jmartin2683 wrote:
       | Hopefully the last one you were responsible for.
        
       | jmartin2683 wrote:
       | Hopefully the most recent one you were responsible for.
        
       | bittermandel wrote:
       | Any code base that doesn't use the advanced features of it's
       | language(s) are always better in my experience. Heavy usage of
       | e.g. meta-programming in Python or perhaps uber's fx (dependency
       | injection) in Go makes projects infinitely harder to get into.
        
         | jamestimmins wrote:
         | Such a great point. I audibly groan when I come across Python
         | meta-programming.
         | 
         | While not an advanced feature, I have a similar response when I
         | see lots of decorators in Python. They quickly become a
         | debugging nightmare.
        
         | sebstefan wrote:
         | I can see a few cases where that depends...
         | 
         | Really simple languages: Ruling out meta-programming is really
         | going to limit you in Lua for example. Just being able to do
         | `mySocket:close()` instead of `Socket.close(mySocket)` involves
         | meta-programming.
         | 
         | Older languages: For C++ the "simple" features are going to
         | include raw pointers and macros. Maybe it's not so bad to allow
         | smart pointers and templates to avoid those
        
           | wruza wrote:
           | Both of these examples are examples of an under-programmed
           | core though. Lua is notorious for lacking batteries, so
           | everyone has to reinvent their own. There's literally no
           | serious Lua program without some sort of classes, but they
           | still resist adding them into lauxlib.
        
         | BiteCode_dev wrote:
         | Depends who is providing it.
         | 
         | Django and pydantic meta programming usually make the code
         | easier to deal with.
         | 
         | In shop written meta programming usually sucks.
        
         | tommypalm wrote:
         | I worked at GOV.UK for a few years on what was effectively
         | specialised CMSs all written in Rails. Mostly basic CRUD stuff.
         | A contractor came along and built the most insane CMS I've ever
         | seen. I found out later it was flavor of the Java Repository
         | Pattern using dependency injection. It became so difficult to
         | work with that the last thing I worked on there was to delete
         | it and rebuild it using standard Rails patterns.
         | 
         | The KISS philosophy exists for a reason and that includes over-
         | using advanced language features just to show off.
        
           | nijave wrote:
           | Besides just KISS, a lot of messes I've seen have been
           | implementing patterns outside the framework or implementing
           | complex patterns that didn't add value.
           | 
           | Besides KISS (or maybe as an extensive), try to keep
           | framework-based codebases as close to the official documented
           | setup as possible. You automatically get to s of free, high-
           | quality documentation available on the Internet.
        
           | andrei_says_ wrote:
           | Props for gov.uk! I've looked at its documentation and design
           | system and see both as peak user experience and clarity.
        
           | RangerScience wrote:
           | I've had a consistent experience in Rails where I think for a
           | day or two I've got a legitimate use case for one the whacky
           | things you can do... and then as I work the problem more, it
           | turns out: nope, the simple stuff is still the Right Way.
           | 
           | Someday, Ruby shenanigans, someday...
           | 
           | PS - Being able to pull the shenanigans _is_ super useful
           | during the dev process, usually to skip some yak shaving
           | during exploration, so it 's nice to have anyway.
        
         | Mikhail_Edoshin wrote:
         | Having just removed a metaclass from my Python code I totally
         | agree.
        
         | rvdginste wrote:
         | I think one should not use advanced language features just
         | because, but I also think one should not avoid using advanced
         | language features where it is useful.
         | 
         | Why would the code base be worse when advanced language
         | features are used?
        
           | lanstin wrote:
           | Because unless you hire steadily more intelligent developers
           | you will be headed towards a mass of code that is hard and
           | scary to change.
        
         | peelle wrote:
         | I disagree with this, but I understand where it's coming from.
         | I think you have a form of whiplash from things like: - Novices
         | overusing the new shiny. - Java/C++/etc Jr programmers
         | overusing design patterns. - Perl programmers solving
         | everything with regexes. - Small startups with GraphQL, or any
         | other large enterprise tool. - Metaprogramming, Maco's,
         | Dependency injection, Recursion, etc when a simpler solution is
         | a better fit.
         | 
         | IMHO, a "best codebase" will be just a bit more advanced than I
         | am, with good resources for me to grok it. I want to be able to
         | learn from it. I also don't want to be so far out of my depth
         | that I can't make a reasonable contribution.
        
           | arp242 wrote:
           | A pinch of salt can really liven up a dish. Not every dish
           | needs it, but when used appropriately it's almost magic in
           | how much difference it can make.
           | 
           | A lot of salt always make everything disgusting.
        
             | tracerbulletx wrote:
             | Huh? Salt is one of the most important foundational
             | elements of basically all cooking.
        
         | pphysch wrote:
         | It would be cool if newer (imperative) languages had a clear
         | protocol to delimit the simple core of the language from the
         | advanced, library-building features. Like Rust has `unsafe`
         | blocks, imagine a `advanced` or `metaprogramming` block that is
         | required in order to use advanced metaprogramming features.
         | Then you could tell a junior to implement $MUNDANE_INTEGRATION
         | without use of `advanced`, and that constraint could be
         | statically verified in PRs, etc.
         | 
         | It seems like it would vastly simplify language evolution. The
         | core language can have rigid limits on changes and follow the
         | Zen of Python or whatever, while the `advanced` language
         | extensions can be a bit looser. Features could graduate from
         | advanced to core without breaking anything. You get a single
         | powerful+expressive language with a "porcelain interface"
         | without necessarily giving juniors ammo to shoot themselves in
         | the foot.
        
           | teleforce wrote:
           | I think D language fits your very descriptions.
           | 
           | You have GC by default that's Python like and intuitive
           | programming constructs. Since for the most part D compilation
           | is much faster than C++ and Rust compilation as long as you
           | stick to the core non-CTFE. Heck you can use D as a compiled
           | scripting language with REPL using rdmd [1].
           | 
           | Then if you want to go gung-ho, you can use the other advance
           | features like D's excellent template meta programming, CTFE,
           | bit banging in-line assembly, etc. D has so many advanced
           | features that can be explored later on, even modern C++ is
           | playing catch up with D in every releases. Nowadays D
           | compiler also support C natively [2] and GDC compiler is
           | included inside the venerable GCC eco-system.
           | 
           | There are university that teach software development and
           | engineering class with D due to its managed, on-demand and
           | gradual complexity, not like "in your face" complexity of the
           | big kahuna programming language like C++ and Rust [3]. Like
           | other compiled languages you can build real-time system even
           | firmware device drivers in D unlike Python.
           | 
           | [1] https://dlang.org/areas-of-d-usage.html#academia
           | 
           | [2] Adding ANSI C11 C compiler to D so it can import and
           | compile C files directly:
           | 
           | https://news.ycombinator.com/item?id=27102584
           | 
           | [3] Teaching Software Engineering in DLang [pdf]:
           | 
           | https://news.ycombinator.com/item?id=37620298
        
         | burutthrow1234 wrote:
         | I worked with someone who insisted on using fx for DI in go.
         | It's so antithetical to the entire Go philosophy, I don't even
         | think it's an "advanced feature". It's just bringing Java cruft
         | to a language where it isn't necessary and making everything
         | worse.
        
       | ethegwo wrote:
       | A Python version about core features of the Hindley-Milner type
       | inferencer: https://github.com/ethe/typer
        
       | donatj wrote:
       | The latest Go micro-service I have built.
       | 
       | About once a year roughly, for the last couple years, the
       | opportunity has arisen to greenfield a Go micro-service with
       | pretty loose deadlines.
       | 
       | Each time I have come into it with more knowledge about what went
       | well and what I wasn't particularly happy with the last time.
       | Each one has been better than the last.
       | 
       | I've been building software professionally for twenty years, and
       | these micro-services have been one of the few projects in that
       | time that have had clear unified vision and time to build with
       | constant adjustments in the name of code quality.
        
         | lanstin wrote:
         | Sounds fun. I have a similar policy but only every other year.
         | It means I know what the hell I am talking about when in
         | meetings with people who have slow or inefficient services. It
         | also means my job of mostly telling other people how to solve
         | problems is still a lot of fun for me.
        
         | intelVISA wrote:
         | Go micro-service with pretty loose deadlines
         | 
         | Sounds like A Money Printer, congrats.
        
       | Traubenfuchs wrote:
       | Every code base I have ever worked on was a legacy nightmare.
       | Every "greenfield" project I joined turned into a legacy
       | nightmare within weeks. I have never encountered enjoyable code.
       | I had the displeasure of wading through Spring, Hibernate and
       | Apache HTTP client code before and they were all an
       | incomprehensible mess.
       | 
       | My conclusion: You know the claim "any medication that really has
       | an effect must also have side effects". I would like to adapt
       | that for code: Any code that does a lot of useful and complex
       | things must be an arcane, barely maintainable mess that can only
       | be understood by deep study.
        
         | alex_suzuki wrote:
         | IMHO library code is especially challenging, as cruft has a
         | tendency to accumulate, historical behavior needs to be
         | preserved, and APIs are set in stone once they're built.
        
         | lukan wrote:
         | "Any code that does a lot of useful and complex things must be
         | an arcane, barely maintainable mess that can only be understood
         | by deep study."
         | 
         | "Every code base I have ever worked on was a legacy nightmare.
         | Every "greenfield" project I joined turned into a legacy
         | nightmare within weeks"
         | 
         | I am sorry to say this, but it really sounds like you were
         | either really, really unlucky, or part of the reason why it
         | became a mess.
         | 
         | Complicated things are complicated. Nothing can ever change
         | that and it requires study to understand it.
         | 
         | But it still does matter a lot, how one organizes the whole
         | thing. How it is structured, documented(!), refactored. Are
         | there competent people in charge who understand it all and kick
         | peoples asses if they make even a temporary mess or forget to
         | document, or do random people make changes wherever they see
         | fit, because a deadline is ticking?
         | 
         | Modularisation is usually the key. Small modules do one thing
         | and are as seperated as much as possible with as little side
         | effects as possible.
         | 
         | And if one has to ship things, it is not always possible to
         | keep it pure and if the code is not intended to live forever
         | then this is often fine. But if the codebase is supposed to
         | stay, then there needs to be the time to clean up the hacks. Or
         | don't and then you end up scared touching anything.
         | 
         | That being said, the technologies you mentioned, I would not
         | like to touch either..
        
           | t43562 wrote:
           | I think that's a bit cruel. Usually mess happens because lots
           | of people with different ideas about the future needs and
           | best structure meet up in the code - and it's hard to develop
           | a consistent culture.
           | 
           | Also codebases get too large for any one person to refactor
           | them into shape in the time they have each day. So you end up
           | needing people who are responsible just for keeping things in
           | shape.
        
             | lukan wrote:
             | Well, that is why I said with those words it _sounds_ like
             | this to me, not that he or she _was_ in fact responsible.
             | (at least this is what I meant)
             | 
             | "Also codebases get too large for any one person to
             | refactor them into shape in the time they have each day."
             | 
             | Which is why modules, or subprojects were invented. Or
             | however you want to call it, if one person is only
             | responsible for a small part and not everyone for
             | everything. And yes, there is also the non trivial problem
             | of time management.
        
               | t43562 wrote:
               | yup, but the style of the modules starts to diverge as
               | different people maintain them.
               | 
               | IMO one does want people whose fulltime job is to looking
               | at a codebase orthogonally to those who are just
               | implementing a feature. People who make sure it builds
               | fast, is secure tries to be consistent to some degree
               | etc.
               | 
               | Many companies call teams/indivuduals like this a "cost
               | centre" and disparage it because they are dolts.
        
               | lukan wrote:
               | If those code checkers are experienced programmers, who
               | do not annoy people with arbitary guidelines and
               | standards, then yes, this might also work. But I do not
               | think they are always necessary as a seperate full time
               | role, if everything works normal otherwise. And with
               | working normal, I mean there is enough time to refactor
               | and document and clean up and is actually done.
        
           | RangerScience wrote:
           | > Modularisation is usually the key. Small modules do one
           | thing and are as seperated as much as possible with as little
           | side effects as possible.
           | 
           | 100% agree. The nuance I find is that a lot times, people
           | want to draw the lines of the module along something that's
           | more immediately intuitive, but results in longer-term
           | complexity.
        
             | lukan wrote:
             | Yes, the hard part is to define what goes inside one
             | module. There are rules of thumb, but lots of exeptions.
             | Taking any advice too literally can also make things
             | awkward.
        
           | arp242 wrote:
           | > Modularisation is usually the key. Small modules do one
           | thing and are as seperated as much as possible with as little
           | side effects as possible.
           | 
           | I think the key aspect is a bit different: understandability.
           | That is, if you open two pages of random code, then can you
           | roughly understand and follow what it roughly does?
           | 
           | Everything in a single package/module/directory can be quite
           | understandable. It doesn't necessarily _need_ to be modular,
           | but it often does help.  "Understandability" is more broad
           | though, and also includes things such as not having weird
           | hard to understand functions nested 8 levels deep, and things
           | like that.
           | 
           | It also means not over-engineering things. "Our code is hard
           | to understand, therefore, we need more architecture" is a
           | fallacy I've seen a few times. The code wasn't hard to
           | understand due to lack of modularity or architecture, but
           | because it was just badly written unclear code.
        
             | lukan wrote:
             | "It also means not over-engineering things. "Our code is
             | hard to understand, therefore, we need more architecture"
             | is a fallacy I've seen a few times. The code wasn't hard to
             | understand due to lack of modularity or architecture, but
             | because it was just badly written unclear code."
             | 
             | For sure, if I have 10 tightly separated micro modules,
             | just for the sake of it, then this is not helping clarity.
             | 
             | But if everything is an entangled mess of spaghetti code,
             | then modularisation helps, so I do not have to look into 10
             | places to get what a certain function is doing.
        
         | mrweasel wrote:
         | Something in your answer triggered a flash back to when I
         | worked for a phone company, 20 years ago. One team, under the
         | leadership of our questionable chief architect, had produced
         | our new, all encompassing backend system. This was to be the
         | corner stone of all future development and integration, dog
         | slow and complicated as it where. I worked as a .Net developer
         | and had the misfortune to be among the first to integrate with
         | this monster. Try as I might, I could not get .Net to
         | interoperate with these services. Finally I figured out that
         | the SOAP services was using some old deprecated versions. Going
         | back to the architect, I ask "did you build this on Apache
         | Axis, and please say Axis2", but no, it was just Axis, a
         | deprecated version, that would generate webservices not
         | supported by newer .Net version. That wasn't an problem,
         | because: "None of our project have upgraded to those .Net
         | versions yet"... DUDE, we've launched a brand new system based
         | on that .Net version a year ago, and what was you f-ing plan
         | for the future, to redo every single service using Axis2?
         | 
         | This guy had based a brand new system on a framework/library
         | that was no longer maintained, even before the system was
         | launched.
        
       | noelwelsh wrote:
       | My open source projects. One in particular I've been working on
       | for about 10 years. The code is consistent and always getting
       | better, even though there is a lot more work that could be done.
        
       | structural wrote:
       | Worked on a codebase for a large safety-critical system where
       | everything was 100% documented, and the development guide for the
       | project was followed so closely that you couldn't tell, across
       | millions of lines of code, that the whole thing wasn't written by
       | one person. Absolutely impressive levels of attention to detail
       | everywhere, down to not even being able to find typographical
       | errors in comments or documentation (a typo in a comment was
       | treated just as seriously as any other bug).
        
         | n_ary wrote:
         | Let me guess, it was very well funded and there were no fake
         | deadlines and cross-team dependencies, am I correct or am I
         | very correct?
        
           | contingencies wrote:
           | Cancel the rocket launch people, we have a semantic
           | incongruity! Correct is a binary state.
        
             | kridsdale3 wrote:
             | Unless you're working with Quantum Correctness.
        
             | nurettin wrote:
             | If you allow for composable statements, the whole can be
             | half correct.
        
             | Hasu wrote:
             | "When people thought the Earth was flat, they were wrong.
             | When people thought the Earth was spherical, they were
             | wrong. But if you think that thinking the Earth is
             | spherical is just as wrong as thinking the Earth is flat,
             | then your view is wronger than both of them put together."
        
           | f1shy wrote:
           | I bet you are very correct
        
         | unboxedtype wrote:
         | I am almost sure this is because the system would have to pass
         | a certification procedure somewhere, and for that they would
         | need this level of clarity. Am I right?
        
           | jamesmunns wrote:
           | Having worked in a couple of safety critical companies, there
           | are things you definitely HAVE to do, but some companies do
           | it better than others.
           | 
           | Some companies have process and development practices down
           | pat: things go smoothly, and meeting the qualification
           | process objectives is easy, because the work has been done
           | right the whole way.
           | 
           | Other companies have less established or less consistent
           | process. They generally meet the process objectives and
           | deliver a working product, but the development process is
           | often more of a struggle, and there is often a lot of
           | "cleaning up" of missed pieces at the end of the release
           | process.
           | 
           | This is just to say: companies and products in the safety
           | critical space don't necessarily have some intrinsic quality,
           | just a higher minimum bar.
        
           | f1shy wrote:
           | In my experience there is exactly 0 correlation between
           | certification needed and code Quality. Right now working in a
           | multi billion company doing SW that must pass many
           | certifications. The code is absolut trash. The tooling is
           | terrible. The people confuse make and cmake. All said. And
           | the SW gets certified, because it is all a matter of how much
           | it costs to certify. It is a kind of high level corruption,
           | that is not seen as corruption.
        
             | unboxedtype wrote:
             | Sorry, I was talking about safety critical system
             | certifications, like for avionics. The systems you describe
             | would never pass that for sure.
        
               | oasisaimlessly wrote:
               | > The systems you describe would never pass that for
               | sure.
               | 
               | I wouldn't assume that.
        
         | rr808 wrote:
         | I'd love to know if working like this is enjoyable or a chore.
        
           | simoncion wrote:
           | After way too many years of working on several sprawling Ruby
           | and Rails projects, I would enjoy the everloving fuck out of
           | working on this.
        
           | jamesmunns wrote:
           | Having worked in both kinds personally, it's _different_.
           | 
           | You spend more time on less topics: the variety of work is
           | much lower, which some people love, and some people are
           | frustrated by. There is often much less "drastic innovation",
           | or when there is, it takes much longer than you might be used
           | to in other industries. Favor is given to proven, predictable
           | technologies and choices, even if that leaves some
           | opportunities on the table.
           | 
           | That being said, it's something I often miss. The ability to
           | have such solid confidence on the things you've built, and
           | the ability to drop into nearly any piece of the system and
           | have everything be consistent and predictable is a quality in
           | itself. It makes debugging (often VERY rare) issues much more
           | tractable, both at a higher systems level, as well as digging
           | deeper into the code itself.
        
             | noisy_boy wrote:
             | Sounds like a well maintained tractor from 1950s that is
             | still chugging along.
        
             | Aeolun wrote:
             | > Favor is given to proven, predictable technologies and
             | choices
             | 
             | I feel like this goes two ways too. Sometimes people favor
             | technologies they've used before, regardless of how many
             | problems they know it causes.
             | 
             | If it's predictable that certain tech will cause you issues
             | years down the line, do not choose it again.
        
               | jamesmunns wrote:
               | Yep. Sometimes it's "known issues are avoidable issues",
               | and other times it's "hey I wish we didn't spend xx% of
               | our time avoiding (incl. spending time auditing syntax)
               | or doing mitigations for the same limitations/issues over
               | and over and over again".
               | 
               | The wheels do turn, just slowly.
        
           | JacobThreeThree wrote:
           | If lives are at stake, it would probably elicit different
           | emotions altogether.
        
       | pavlov wrote:
       | For me, the most eye-opening codebase of my career was Cocotron,
       | around 2007:
       | 
       | https://github.com/cjwl/cocotron
       | 
       | I was looking for a way to port my native Mac Cocoa apps to
       | Windows. I had been already disappointed by the aimless sprawl of
       | GNUstep.
       | 
       | This one-person project implemented all the essential APIs for
       | both Foundation and AppKit. Reading the code was a revelation:
       | can it really be this simple and at the same time this
       | effortlessly modular for cross-platform support?
       | 
       | I contributed a few missing classes, and successfully used
       | Cocotron for some complex custom GUI apps that needed the dual-
       | platform support.
       | 
       | Cocotron showed me that one person with a vision can build
       | something that will rival or even outgun large teams at the big
       | tech companies. But focus is essential. Architecture astronauts
       | usually never get down from their high orbits to ship something.
        
         | hannofcart wrote:
         | +1 to this.
         | 
         | I remember my boss asking me to build a MacOS UI we had on
         | Windows with this. I was very sceptical that it would go
         | anywhere.
         | 
         | But no, it worked. I was just open mouth shocked to see the GUI
         | worked pretty flawlessly in Windows.
        
           | cjwlloyd wrote:
           | :)
        
         | cjwlloyd wrote:
         | Thank you!
        
       | tlarkworthy wrote:
       | Google3 codebase. It's just so vast and it works. It's a miracle.
       | I feel lucky to have seen it. Everytime you change it, it reruns
       | the dependencies. Everyone has different view concurrently.
       | Commits are efficient immutable snapshots. It's just incredible
       | multiplayer. So massively beyond what can be done with GitHub.
       | Really I feel it's peak codebase. I've not seen what the other
       | big techs do but it blew my mind
        
         | asymmetric wrote:
         | What is Google3?
        
           | kqr wrote:
           | I would assume Google's monorepo.
        
             | Daub wrote:
             | This link might help...
             | https://dl.acm.org/doi/10.1145/2854146
             | 
             | The linked pdf has lots of details.
        
               | erikpukinskis wrote:
               | HTML version in case anyone prefers that to PDF:
               | https://cacm.acm.org/research/why-google-stores-billions-
               | of-...
        
           | bushbaba wrote:
           | The replacement of google2
        
         | arcade79 wrote:
         | It was wonderful to work with. :-) It's one of the things I
         | truly miss from Google.
        
         | VirusNewbie wrote:
         | Google3 codebase very consistently has _clean_ code, but some
         | of the architecture there is very much not great.
         | 
         | Some is great, some not so much.
         | 
         | Some of Verizon's code was _much_ more elegant (though much
         | smaller scope) from an API perspective, and really leaned into
         | advanced type systems in a way Google has not.
        
         | dilippkumar wrote:
         | +1
         | 
         | I can not understate how much I agree with parent comment.
         | 
         | The opposite of move fast, build a shitty prototype and iterate
         | is a deliberate problem solving approach undertaken by the
         | highest caliber of engineers. The actual challenges to be
         | addressed are effectively addressed right at the design stage.
         | 
         | The result is a thing of immense beauty and elegance.
         | 
         | I will forever be grateful for the opportunity I had to see
         | this magnificent piece of engineering in action.
        
           | robertsdionne wrote:
           | Cannot overstate.
        
             | musiciangames wrote:
             | (Aside). To expand slightly, what robertsdionne is
             | highlighting is the changing usage of this expression. In
             | its original sense, e.g an issue is so important that it is
             | impossible to overstate its importance. It is now
             | increasingly used the other way around.
             | 
             | Old me would have said it's used wrongly, but this happens
             | all the time with language. Especially things being used in
             | the opposite of their original sense, e.g. inflammable for
             | flammable.
        
               | smcameron wrote:
               | In my mind, "cannot overstate" always meant "impossible
               | to overstate", but I think some people interpret/intend
               | "cannot understate" to mean something like "must not
               | understate". I don't know if that's really what they're
               | thinking, but it is how I make sense of it. I have come
               | to just avoid such constructions.
               | 
               | Edit: reminds me of an ancient SNL skit with Ed Asner in
               | which he's a retiring nuclear engineer and as he heads
               | out the door he says to his incompetent co-workers "Just
               | remember, you can't put too much water in a nuclear
               | reactor".
        
               | deanishe wrote:
               | > opposite of their original sense, e.g. inflammable for
               | flammable
               | 
               | Inflammable was _never_ the opposite of flammable. Those
               | word have always been synonyms. The opposite was always
               | non-flammable.
        
           | semitones wrote:
           | I, like many others here, echo this sentiment. While I
           | disliked working in ads, the experience of working with that
           | repo+tooling is unmatched by anything else in my career.
        
             | bufferoverflow wrote:
             | If the tooling is so far beyond anything publicly
             | available, why don't you guys make something like that and
             | make millions?
        
               | thimabi wrote:
               | Perhaps they make more than that just by using the
               | tooling internally and creating/maintaining other stuff
               | with it -- that's a major competitive advantage
        
               | vundercind wrote:
               | I strongly suspect it's not that useful for a lot of
               | businesses.
               | 
               | So many have their code split between a two dozen clouds,
               | BA tools (... does Google put that in the monorepo too?
               | Or is that, which is a _lot_ of the code at most
               | businesses, not " _really_ " code at Google?), vendor
               | platforms, low-code tools you all hate and are trying to
               | deprecate but god damned if you aren't still spending dev
               | hours on new features in it, et c...
               | 
               | I bet achieving anywhere near the full benefits would
               | first require retooling one's entire business, including
               | processes, and bringing a whole shitload of farmed-out
               | SaaS stuff in-house at enormous expense, most places.
        
               | znkr wrote:
               | It's not just tooling; processes, infrastructure and
               | dedicated teams for centralized infrastructure are what
               | makes Google's monorepo what it is. FWIW, most of the
               | tools are publicly available or have good publicly
               | available counterparts. What's likely missing elsewhere
               | is funding for infrastructure teams.
        
               | wmanley wrote:
               | I'd love to see something like that applied to a project
               | like Debian. The tools already exist. The cost of
               | switching is too great, and you'd need everyone to learn
               | your new system before you'd see the benefit.
               | 
               | I wrote a bit about this here:
               | https://blog.williammanley.net/2020/05/25/unlock-
               | software-fr...
               | 
               | I understand that nix have made some progress in this
               | direction, but I don't know any more than that.
        
               | lesuorac wrote:
               | millions sounds like a lot but it's like just 3
               | employees.
               | 
               | To quote broccoli man, "i've forgotten how to count that
               | low" [1].
               | 
               | [1]: https://www.youtube.com/watch?v=3t6L-FlfeaI
        
           | kridsdale3 wrote:
           | I dunno man, I work at google and this comment feels like
           | circlejerk. It's not held to THAT high a standard. We
           | absolutely commit janky MVPs and iterate.
        
             | elevatedastalt wrote:
             | He is referring to the codebase itself, not individual
             | projects or CLs.
        
               | simoncion wrote:
               | As a Google outsider, that's not at all the impression I
               | got from the paragraphs.
               | 
               | This:
               | 
               | > It's not held to THAT high a standard. We absolutely
               | commit janky MVPs and iterate.
               | 
               | seems to very directly address this:
               | 
               | > The opposite of move fast, build a shitty prototype and
               | iterate is a deliberate problem solving approach
               | undertaken by the highest caliber of engineers. The
               | actual challenges to be addressed are effectively
               | addressed right at the design stage.
               | 
               | If your claim is that "Well, if you look at the codebase
               | AS A WHOLE, there's absolutely no iteration and shitty
               | prototypes, it's all designed right from the start."...
               | well, I can't see any way that codebase came into
               | existence without being built up by individual projects
               | and changesets.
               | 
               | So, yanno, when folks on the ground report that
               | individual projects and changesets/PRs/whatever ARE using
               | the "Commit something barely serviceable, test it out,
               | and iterate." process, statements like "We always get the
               | design right before we write even a single line of code!"
               | definitely come off as a circlejerk.
               | 
               | Google's a very large software house, it has a very
               | exclusive hiring process, and it (reportedly) has
               | internal tooling that's very well-adapted for the
               | problems a company of its size faces. But Google is still
               | hiring from the same pool of programmers as everyone
               | else, and the odds that zero of those that it hires will
               | work best with the "Get something out there to field-
               | test, and use the test results to make it more suited for
               | field use." method of development are absolutely zero.
               | Given Google's size, the odds that zero of _those_ will
               | never be able to negotiate to work in the way they work
               | best are ALSO zero.
               | 
               | (And -frankly- I expect that this method of development
               | gets used a lot in the company. You can burn an assload
               | of time on simulators (and what is the "What if?" game,
               | but a in-brain simulator?), but in the realm of software,
               | it's not-infrequently the case that the simulator with
               | the best ROI is real-world deployment.)
        
               | remram wrote:
               | He is referring to the codebase not the code?
               | 
               | What does that mean?
        
           | kichimi wrote:
           | I think skill is also an issue here, in both directions. I
           | have worked with a company that followed the opposite of move
           | fast and it just turned into a 'who is the most correct' and
           | 'what is the most elegant code' competition. We didn't push
           | out a single new feature in the three years i'd worked there
           | at that point. There was so much focus on code that we gave
           | almost no time to business requirements.
           | 
           | The wider implication of this was that the number of tickets
           | we got dropped dramatically, because users knew they'd never
           | be resolved anyway.
           | 
           | Balance is key.
        
         | inoffensivename wrote:
         | I honestly can't tell if this comment is sarcastic...
         | 
         | google3 is a quagmire and it's getting worse by the day
        
           | ignoramous wrote:
           | Well, first impressions:                 It is part of our
           | human condition to long for hard lines and clear concepts.
           | When we have them we have to either face the fact that some
           | realities elude them, or else bind ourselves to the
           | inadequacy of the concepts.
           | 
           | https://fs.blog/impressions-are-schematically-determined /
           | https://archive.is/F8uPE
        
           | thwd wrote:
           | Agreed. Once long (10+ years) ago, google3 may have been peak
           | codebase. Nowadays, not so much.
           | 
           | Edit: I guess it also depends at what level of abstraction
           | you work. High: can be easy breezy. Low: oh boy.
        
         | phreeza wrote:
         | I agree but would add that within the codebase there are also
         | vastly different experiences. The search ads serving stack for
         | example is phenomenal, it is >10 years old, very complex and
         | gets contributions from hundreds of engineers every week, yet
         | it is quite easy to understand what is going on anywhere and it
         | has great performance. But I also had to work with other code
         | which had been through 3 rewrites, none of which had ever been
         | totally completed, so there were overlapping paradigms
         | everywhere and working with it was just a nightmare. Tests
         | never caught anything leading to several post mortems.
        
         | lokar wrote:
         | It was certainly great up to like 2015, perhaps later?
         | 
         | You could spend all day finding fun code to read.
         | 
         | I always like the main signal (eg sigsev) handler.
        
       | cpojer wrote:
       | https://github.com/nkzw-tech/athena-crisis
        
       | joeshachaf wrote:
       | I worked with a very experienced engineer who created his own WAF
       | (software-based) using Java. From a purely architectural
       | perspective, it might not be the best (I might have used an ISAPI
       | filter back then), but the code itself was very efficient, well-
       | written and documented. I used it several times as a teaching
       | example.
        
       | Agingcoder wrote:
       | I haven't contributed code to it, but I've read lots of code in
       | it ( for work reasons ) - I really like the golang compiler and
       | library codebase.
       | 
       | About codebases I've written code for, the best one strived for
       | simplicity, and was driven by very strong engineers who actively
       | considered code hygiene ( in the broadest possible sense ) a
       | first class citizen.
        
         | christophilus wrote:
         | Go, SQLite, Redis are the three that came to mind for me.
         | Simple, readable, fairly flat hierarchy.
        
       | prinny_ wrote:
       | The one in my previous job, which was an admin board for a market
       | intelligence application. Ultimately, the reason it was good was
       | because the engineers had zero ego on top of having excellent
       | skills. The team that set the codebase were basically 4 seniors
       | and 3 principals (the client actually did pay for top talent in
       | this case) so not only everything was based on industry
       | standards, written elegantly and organized perfectly, but every
       | time some new requirement came up, these senior / principal
       | engineers would discuss it in the most civilized matter I have
       | ever seen.
       | 
       | E.g, "we need to come up with a way to implement X". Person A
       | gives their idea, person B gives another idea and so on until
       | everybody shared their thoughts. Then someone would say "I think
       | what person C said makes the most sense" and everybody would
       | agree and that was it. 30 minutes to hear everybody out, 3
       | minutes to discuss who will do it and when and the meeting was
       | over.
       | 
       | I think the biggest testament to this code base was that when
       | junior members joined the team, they were able to follow the
       | existing code for adding new features. It was that easy to
       | navigate and understand the big picture of.
        
         | brailsafe wrote:
         | This is the approach I tried to take as an IC frontend on a
         | building management system dashboard about a year ago. It was
         | basically me and one other frontend, one full-stack, and a QA
         | or 2, plus a manager. My manager I guess had written a lot of
         | the code we'd be interacting with, and I guess was somewhat
         | protective and reluctant to delegate authority over decisions
         | around it. It was a big refactoring project, and I just
         | encouraged my colleagues to take the things they wanted, like
         | rebuilding the state management system. We'd discuss the
         | requirements and why it was necessary, and I'd look for reasons
         | to agree rather than disagree, then we'd be off. Something
         | burnout has taught me is that the marginal differences between
         | one implementation detail or another are not worth getting hung
         | up on unless they pose a real problem, especially when someone
         | else gets to decide that doing it fast is priority (sprints).
        
           | robertlagrant wrote:
           | > Something burnout has taught me is that the marginal
           | differences between one implementation detail or another are
           | not worth getting hung up on unless they pose a real problem,
           | especially when someone else gets to decide that doing it
           | fast is priority (sprints).
           | 
           | I think this is true. And a self-imposed problem (should a
           | problem arise) is much less frustrating to fix than one that
           | came from a decision imposed by someone else, even if the
           | latter avoided loads of other problems. Sometimes it's better
           | to let people make mistakes (as you believe them to be) and
           | correct them later.
        
             | konschubert wrote:
             | And often, the cost of indecision is much bigger than the
             | margin in utility between the optimal and the second-to-
             | optimal solution.
        
         | psychoslave wrote:
         | Great example as it shows that code base really is a side
         | consideration and a proxy to team quality.
        
         | cainxinth wrote:
         | Another reminder that soft skills multiply the effectiveness of
         | hard skills.
        
           | alickz wrote:
           | Exactly.
           | 
           | What is development if not communication? Between the
           | clients, the coworkers, the users, and the computer
           | 
           | I'd take a mediocre dev with no ego and ok social skills on
           | my project any day over an egotistical genius who can't work
           | with other people or effectively communicate their ideas
        
         | zamalek wrote:
         | Otherwise known as "you are not your code" and "you are not
         | your ideas." Essentially, criticism against your code is not a
         | personal attack (and other corollaries). Its definitely
         | something I aspire to, but its surprisingly hard to get right.
        
       | BiteCode_dev wrote:
       | Last year I worked for a client that gave me a lot of time, money
       | and autonomy to lead dev on a critical software rewrite.
       | 
       | We got a small team of competent people, with domain experts to
       | peer code with the devs.
       | 
       | It was wonderful. We could test, document and clean up. Having
       | people who knew the trade and users at hand removed second
       | guessing.
       | 
       | The result was so good we found bugs even in competitors'
       | implementations.
       | 
       | We also got x5 in perfs compared to the system it was replacing
       | and more features.
        
         | leetrout wrote:
         | Good for you. Magical moments in careers are hard to find in my
         | experience but they are so satisfying when you get there.
         | 
         | Glad whomever was over this didnt just drop the "dont rewrite"
         | joel spolsky article and fight making it happen.
        
           | BiteCode_dev wrote:
           | I actually was the one telling them not to rewrite, lol.
           | 
           | But the original code was a mess of matlab spaghetti, they
           | couldn't find a way to hire for that. Not to mention turning
           | it into a web service was already a big hack of java parsing
           | a raw dump of matlab datastructures that nobody dared to
           | touch.
           | 
           | I had to read the matlab code, and it tooks hours to decypher
           | a few lines. Plus the language doesn't have great debugging
           | and error handling capabilities and the tooling is quite
           | terrible.
           | 
           | So rewriting to python won, and for once, I must say it was a
           | good call.
        
           | junon wrote:
           | Joel has since said that that he doesn't really agree with
           | that advice anymore, at least not in the same way. Super
           | annoying that it gets parroted over and over again as though
           | it's the word of the lord.
        
             | kevin_nisbet wrote:
             | I agree it shouldn't be parroted as though it's the word of
             | the lord, like any advice it will be more or less
             | applicable on specific situations. I've been on both sides
             | of the let's redesign a thing plenty of times.
             | 
             | The balance is somewhere in the middle, it's valuable
             | advice for what can go wrong when you don't think through
             | the implications of the decision making process, understand
             | how and why the system works the way it does, and what
             | risks exist for what can go wrong with a big redesign. But
             | like anything, if the risks are understood, then those
             | risks can be accepted, mitigated, or rejected if
             | appropriate, or provide guidance on why the redesign
             | investment isn't worth the associated risks.
        
         | thecupisblue wrote:
         | Similar thing.
         | 
         | Had time and autonomy from a client, so took sweet time
         | examining the domain, the existing systems et al. Spent a few
         | months writing the basis and the framework around what will be
         | done, based on years and years of experience I had with bad
         | frameworks and codebases, combined with working on the same
         | domain for their parent company years ago.
         | 
         | And it worked. We delivered features insanely fast, hundreds of
         | forms were migrated, feature generators would create 90% of the
         | boilerplate code and the code was small, readable and neatly
         | clustered. Maintaining it was a piece of cake, leading to us
         | not having enough work after a while so I negotiated our time
         | to half a week for the same money.
         | 
         | After a while, client deemed us too expensive to pay for only
         | 2.5 days of work - after all, how does it make sense - if we
         | are paying them that much, they should work 5 days!
         | 
         | So they cut us out. Two things happened:
         | 
         | 1. Devs that got moved to other projects in the company told me
         | they didn't know development could be so smooth and tried to
         | replicate it in future projects, even tho they say it failed a
         | lot of lessons they picked up from the framework were highly
         | relevant in their future careers.
         | 
         | 2. The company found a cheaper developer, he said "this is
         | unusable and has to be rewritten" and rewrote it into "clean
         | code", taking longer than the original project took. At least
         | he works 5 days a week now.
        
           | Aeolun wrote:
           | We were going to do this. Get several months to set up a
           | clean base for our new system with only the four most
           | competent people working on it.
           | 
           | Then I went on vacation for a week.
           | 
           | Come back to a system that needs to be delivered to prod next
           | month with 20 randos submitting PR's...
           | 
           | WTF happened. Did you learn _nothing_ from previous failures?
        
             | bboygravity wrote:
             | What was the lesson they where supposed to have learned?
        
               | Aeolun wrote:
               | 9 women cannot have a baby in a month.
               | 
               | There's a limited amount of people that can work on a
               | system at the start of a project if you want it to be a
               | coherent whole, so that you can have everyone iterate on
               | it in the same style later. If you start out with 20
               | (junior) engineers, you get a kind of frankenstein where
               | all issues are waved away because 'we need to deliver in
               | a week'
        
       | simfoo wrote:
       | My past three employers code bases: mono-repos, Bazel, lots ot
       | C++ and Python, thousands of libraries and tools, code generation
       | and modeling tools that are fully integrated into the build, easy
       | cross compilation, large integration tests just one bazel test
       | invocation away, hermetic and uniform dependencies...
        
         | fire_lake wrote:
         | Bazel Python does not cross compile IIRC. How was this
         | achieved?
        
           | simfoo wrote:
           | Python is only used for build-time tooling (modeling, code
           | generators) and developer tooling. All the on-target code is
           | C and C++, and only that is cross compiled (Linux, QNX,
           | various RTOS and x86+aarch64)
        
       | Claudus wrote:
       | Laravel, once you get the hang of it everything just works, and
       | using a debug bar to optimize database calls is very satisfying.
        
       | neilv wrote:
       | One of them stands out, due to being super-productive, over
       | years, and then decades.
       | 
       | A large system that was originally written by only two super-
       | productive engineers (I mean real engineers, both with PhDs in an
       | area of Engineering). And a comparably capable and essential IT
       | person.
       | 
       | The reasons for the super-productivity include one of the
       | developers choosing great technology and using it really well, to
       | build a foundation with "force multiplier" effects, and the other
       | developer able to build out bulk with that, while understanding
       | the application domain.
       | 
       | Another reason was understanding and being pretty fully in
       | control of the code base, so that, as needs grew and changed,
       | over years, someone could figure out how to do whatever was
       | needed.
       | 
       | One of the costs was that most things had to be built from
       | scratch. Over time that also proved to be an advantage, because
       | whenever they needed (put loosely) a "framework" to something it
       | couldn't do, they effectively owned the framework, and could make
       | dramatic changes.
       | 
       | When I said "costs", I mean things like, many times they needed
       | to make a component from scratch that would be an off-the-shelf
       | component in some other ecosystem. So if someone looked closely
       | at how time was sometimes spent, without really understanding it
       | or knowing how that panned out, it would look like a cost that
       | they could optimize away. But if they looked at the bigger
       | picture, they'd see a few people consistently, again and again,
       | accomplishing what you'd think would take a lot more people to
       | do.
       | 
       | It helped that the first programmer also became the director for
       | that area of business, and made sure that smart engineering kept
       | happening.
       | 
       | Someone might look for a reason this couldn't work, and think of
       | bus factor. What I think helped there was the fact that the work
       | involved one of those niche languages that attract way more super
       | programmers than there are jobs. "Gosh, if only we had access to
       | a secret hiring pool of super programmers who were capable of
       | figuring out how to take up where the other person left off, and
       | we had a way to get them to talk with us...")
       | 
       | It was easy to imagine a competitor with 100 developers, not able
       | to keep up, and at many points getting stuck with a problem that
       | none of them were able to solve.
        
         | klibertp wrote:
         | I assume you avoided identifying (or even hinting at) this
         | "great technology" on purpose, but could you persuaded to
         | divulge what it was?
        
           | neilv wrote:
           | It started as a bespoke structured data and Web architecture,
           | which did somewhat complex things that would've been a huge
           | headache to build and maintain in the dominant languages of
           | the time, but were viable for one person to figure out and
           | implement in Scheme and PostgreSQL.
           | 
           | That bespoke architecture and implementation language lent
           | itself to a lot of rapid domain-specific functionality, as
           | well as doable architecture changes over time.
        
             | klibertp wrote:
             | Thanks! :) I hoped it'd be a Lisp or Smalltalk - great to
             | hear I almost guessed right, though I never imagined an
             | RDBMS to be in the mix :)
        
               | neilv wrote:
               | There's actually a bespoke object metamodel mapping atop
               | the RDBMS, and it permits customers who want to use SQL
               | direct to the database to do so.
               | 
               | And, IMHO, PostgreSQL is the default choice for most
               | storage backend purposes, unless one has figured out a
               | good reason something else is better. :)
               | 
               | There were also multiple kinds of large blob storage
               | needs, and changing architecture needs over time (e.g.,
               | multi-tenant, changing authn implications of things like
               | browser plugins, scalability, move to cloud), so systems
               | programming skills and controlling your framework comes
               | in handy, for not getting stuck or blocking on vendors,
               | but just solving the problem.
        
         | bruce343434 wrote:
         | What is this niche language attracting super programmers?
        
           | neilv wrote:
           | Scheme, Common Lisp, Haskell, [edit] Smalltalk, probably more
           | obscure ones...
           | 
           | Also, Erlang, Rust, and Clojure, though those have been
           | rumored to be employable, so no longer get as much of the
           | filter as you get when it was just people caring strongly
           | enough to want to use a particular language _despite_ the
           | unemployability.
           | 
           | Whether that caring happened because of skill (to identify
           | something good), or skill happened because of caring (to
           | invest the time in exploration and thinking and practice),
           | these communities seem to get more than their share of great
           | programmers.
           | 
           | And so you know where to look for them, and you have a reason
           | that they might want to talk to you.
        
             | bear8642 wrote:
             | >probably more obscure ones.
             | 
             | Apl/BQN worth adding for the recent CUDA/vector language
             | explosion, and has it's fair share of extraordinary
             | programmers!
             | 
             | Forth an interesting one too look, although never properly
             | used.
             | 
             | miniKanren/Prolog another vein to branch down
             | (unification/logic programming)
        
       | nonameiguess wrote:
       | No name because the project code name changes with each new
       | contract, but the first-level ground processing system for the
       | NRO that turns raw satellite downlinks into some kind of human-
       | intelligible files.
       | 
       | I think a lot was just the restrictions built into the way
       | development happens required a high level of discipline, care,
       | and planning. Also, requirements were pretty tightly coupled to
       | sensor platform capabilities, which are known well in advance and
       | don't change unexpectedly, so waterfall development actually
       | works and you don't have to deal with the chaos of not really
       | knowing what will and won't work and customers constantly
       | changing their minds.
       | 
       | Code base was overwhelmingly C++, some Fortran, a lot of it was
       | very old. It was all developed on airgapped networks, and the
       | difficulty of bringing in external dependencies meant there
       | largely were not any. All of the library functionality we
       | required was mostly developed in-house, so we had extremely well-
       | documented and stable functions available to do damn near
       | anything you could want, with a good chance that whoever first
       | wrote it was still there. All development had always been tied to
       | a ticketing system of some sort that included all of the original
       | discussion, design documents, and that kind of thing might add
       | process overhead upfront, but it means that forever new
       | developers can simply read the history and learn exactly why
       | everything works the way it works.
       | 
       | The system itself was very Unixy. In production, it was meant to
       | be run as a server with many instances striped across high-
       | performance compute nodes, but it did not have to be run that
       | way. Every individual product flow could also be built as its own
       | transient executable, so that working on a single component could
       | easily be done locally. You didn't have to rebuild the world or
       | spin up your own server. Performance requirements were enough
       | that we had our own customized networking stack and filesystem in
       | production, but nothing depended on this for function, so you
       | could still develop and test on plain Linux with ext4.
       | 
       | The culture was terrific. We were part of one of the big five
       | defense contractors, but an acquisition and this program was
       | largely still staffed by employees of the original company that
       | had been acquired. We were de facto exempted from most of the
       | bullshit any other program had to deal with. I don't know if that
       | was part of the original terms of being acquired or just a
       | consequence of having so many long-time developers that couldn't
       | afford to be lost if you subjected them to nonsense. This was the
       | kind of project that people intentionally stayed on and retired
       | from because the experience was so much better than any other
       | project you could get assigned to.
       | 
       | Ironically, it had none of the characteristics that high-
       | performing companies often tout. You work in private. The rest of
       | the company, including your own management chain, doesn't even
       | know what you're working on. You'll never get any recognition or
       | publicity. The pay is mediocre. We weren't attracting the best
       | and brightest from all of the world. You had to be American, have
       | a top-secret clearance, and be geographically close enough to the
       | facility to get there every day, so this was a pretty constrained
       | hiring pool. I still worked with some of the smartest people and
       | best engineers I've ever known. The upside of this kind of
       | environment is you have no mercenaries or publicity hounds.
       | Everyone who sticks around is a person who really loves and cares
       | about what they're working on, and a lot of people did stick
       | around. The sanity and organization of the code was heavily
       | facilitated by having a whole lot of people working on it who'd
       | been working on it for 30+ years.
        
       | rthnbgrredf wrote:
       | The best codebase is the one you fully understand. I prefer
       | codebases that are small enough to understand within a week. This
       | is why I like Microservices. Large codebases can be overwhelming
       | and even senior developers working a decade in the company of
       | might not fully understand them. Instead, I prefer maintaining a
       | few Microservices that our team fully comprehends, where the
       | entire codebase fits into a clear mental model. We then interact
       | with other codebases, that have active mental models in other
       | teams, via APIs.
        
         | 01HNNWZ0MV43FF wrote:
         | I'm skeptical of this because I think that a well-architected
         | large codebase should be as good as multiple microservices
         | stapled together at the seams. But then I don't do much web
         | dev.
        
       | _kb wrote:
       | I don't think the 'code' side of code base can be considered in
       | isolation.
       | 
       | What makes a _project_ objectively good (from subjective
       | experience) is a combination of code, design, documentation, and
       | often the humans involved.
        
       | fer wrote:
       | One that had a sort of improvised facade/adapter pattern (it
       | didn't really follow either) in a clearly cut multilayered and
       | pipelined structure, with actor model bits where it made sense.
       | 
       | The code wasn't simple, at all. It took active training of new
       | arrivals for them to understand it. But it was very well thought
       | out, with very few warts given the complexity, and extremely easy
       | to extend (that was the main requirement, given constant changes
       | in APIs and clients).
       | 
       | We had an API, with multiple concurrent versions, that
       | transformed requests into an intermediate model, on which our
       | business logic operated, later targetted external APIs (dozens of
       | them, some REST, some SOAP, some under NDAs, some also with
       | multiple versions), whose responses turned again into the
       | intermediate model, with more business logic on our end, and a
       | final response through our API. Each transaction got its context
       | serialized so we could effectively have what was an, again
       | improvised, "async/await"-like syntax in what was (trigger
       | warning) C++03 code.
       | 
       | The person who engineered it didn't have formal CS background.
        
       | Maledictus wrote:
       | Ruby on Rails.
       | 
       | It is the only framework I have read top to bottom.
       | 
       | Also the FreeBSD kernel, if you want to see a C code base that's
       | quite beautiful (for C).
        
         | sgt wrote:
         | In this category I would nominate Django as well. It's very
         | well designed (opinionated, but usually for good reasons).
         | 
         | In terms of large C code bases I enjoy reading the PostgreSQL
         | source code.
        
           | rc_mob wrote:
           | whenever I'm stuck on how to structure some code, I ask
           | myself how would Laravel do it? and look up their code and
           | structure mine similarly
        
             | chuckadams wrote:
             | My story is similar, but using Laravel as the canonical
             | example of how not to structure things. Even without
             | Facades, the internals are an impenetrable maze of magic
             | methods, with things like Macroable adding methods from
             | dog-knows-where. Even adding phpdoc types doesn't work when
             | dealing with things like scopes.
        
       | t43562 wrote:
       | I just cannot face the thought of calling any of them "best".
       | 
       | Every one had good and bad features though. One or two were OS-
       | sized and I think a codebase that compiles and links to 85GB of
       | output for 20+ devices without being a total disaster inside is
       | harder to do than a neat small python module or whatever.
       | 
       | GOOD FEATURES:
       | 
       | Maintenance of the build and test: I worked on tools that helped
       | builds go faster so I saw a lot of codebases where people were
       | not maintaining the build partly because nobody had that a s a
       | responsibility. There was bad management of dependencies leading
       | to build failures, poor performance, incorrect build output.
       | Android would be a counter example to that - I don't know if
       | people like developing in it but it was always hard to accelerate
       | it as the maintainers fixed performance problems regularly
       | leaving our tools with little to improve.
       | 
       | Using appropriate languages. Writing everything in C++ was a fad
       | at one time. All projects work better, port better, have faster
       | build times, are easier to test etc if they use memory safe
       | "build once" languages to a maximum (e.g. java) and unsafe ones
       | (e.g. C/C++ which have to be rebuilt and tested for each
       | device/os) to a minimum. IMO Android beat Symbian amongst other
       | reasons because it wasn't all C/C++ and that meant a lot of code
       | didn't have to be rebuilt for different devices. This made builds
       | faster and fast builds lead to better quality because of a short
       | dev-test cycle.
       | 
       | Use of faster compilers over "better" compilers. Ultimate code
       | performance and quality depends on a fast development cycle more
       | (IMO) than on having the most optimizing compiler. GCC versus the
       | older ARM compilers for example. Now the ARM compiler is based on
       | LLVM and I know that happened indirectly from a suggestion I made
       | to someone who then made it to ARM who then did it.
       | 
       | The setup and build of one codebase I worked on was as easy as
       | one could expect, the build errored out if you tried to use the
       | wrong tools so you never ended up debugging weird failures
       | because of an incorrect tool in your PATH somewhere. I made this
       | feature happen :-D. With big codebases the tools could be
       | included in the version control system so you knew you had the
       | right compiler, right object dumper etc. This is another strength
       | of Android and yet I was in a project for Symbian to do the
       | opposite because of some utter bonehead who never touched a build
       | in his life who was trying to make a name for himself with his
       | slimy bosses as a "doer" and "reformer."
       | 
       | Codebases (especially big ones) benefit a lot from some kind of
       | codesearch/index where you could find out where some
       | function/class/variable was defined and what version of the
       | source base it was introduced in.
       | 
       | BAD FEATURES:
       | 
       | Exclusively Owned code - we need to know who understands code
       | best and who is best to review it but I don't think anyone should
       | have totally exclusive control. It was a nightmare for me at one
       | job - trying to get another team to make some needed change (like
       | fixing their stupid makefiles to work properly in parallel). We
       | (build team) should have been able to do it ourselves - maybe
       | including them in the PR. Sometimes ownership is entirely
       | theoretical - nobody who wrote it is still employed and nobody
       | among the notional owners understands it and none of them want to
       | approach it within 100 metres in case it blows up and becomes
       | their problem. I simply _had_ to approach such code - no choice -
       | but I kept having to send diffs to people who didn 't want to
       | bother to look at them. It was a case of pushing wet spaghetti
       | and took forever to do very simple changes.
       | 
       | Insufficient tests that run infrequently. What else is there to
       | say?
       | 
       | Complicated code with no "why" or "what this is for" type
       | comments. The kind of thing you trawl around in for weeks and
       | cannot make head nor tail of what is going on overall.
       | 
       | Code with so much dependency injection and general SOLID that you
       | have to bounce all over the place to understand a very simple
       | action.
       | 
       | Code where writing tests is an enormous ballache. In one Go
       | codebase the reason was because somone decided that the standard
       | Go practise of an array of test data being run through a kind of
       | "test engine" was the only way anyone should be allowed to write
       | tests. Hence you had to do lots of weird things to make your test
       | cases into data. Generally we use a kind of "religious" approach
       | to try to get consistency out of a group of people but then take
       | it much too far.
       | 
       | codebases without automated reformatting - so everyone wastes
       | time arguing about line spacing or camel-case names or whatever
       | in their PRs.
        
         | gwbas1c wrote:
         | > Code with so much dependency injection and general SOLID that
         | you have to bounce all over the place to understand a very
         | simple action.
         | 
         | I find that happens when people get religious about patterns
         | and methodology; without understanding the "why", the language,
         | and how a computer works.
         | 
         | Case in point: I once worked on a C# project that used a port
         | of Spring for dependency injection: Ultimately, it was near
         | impossible to know when something was constructed, and what was
         | calling what. There were classes that couldn't call themselves
         | through "this" because of certain weird dependency injection
         | features used.
         | 
         | Later, I decided to use dependency injection as a design
         | pattern: Instead of a complicated DI framework, there was just
         | a few files of code. It was very easy for newcomers to
         | understand. It was also easy to swap in mock objects, and easy
         | to swap dependencies based on the target platform. It was also
         | easy to see when a dependency was constructed; because it
         | wasn't hidden behind a giant framework.
        
       | nijave wrote:
       | I was debugging some issues with Thanos and had pretty good
       | success tweaking the codebase to add additional telemetry.
       | 
       | The code was fairly well organized and more importantly worked
       | out of the box with a Makefile and IDE integration (GoLand). All
       | it took was `git clone` and opening GoLand to get started.
       | 
       | For C (maybe it's C++), fluentbit seemed pretty straight forward
       | (I don't have much experience in C though)
        
       | sibit wrote:
       | Pretty much any internal tool/TUI/CLI/library I've created. If I
       | had to guess I'd say at most 25% of the company projects I've
       | worked on have launched AND have consistent usage. Working hard
       | on something just for it to wither crushes my soul but internal
       | projects are different. They're all skunk works projects. No
       | tickets. No project/board. No PM pushing back on how many points
       | (read: hours) something should be. I'm solving real problems that
       | directly impact the quality of life for myself and my coworkers.
       | The best part is getting real, genuine, feedback. If something
       | sucks they'll tell you and they won't sugarcoat it.
        
         | jftuga wrote:
         | I love this take. What language(s) do you typically use to
         | write CLI programs? I'm also interested in learning about what
         | types of internal TUI tools you have created.
        
       | wruza wrote:
       | The ones that were straightforward and close to the business. It
       | starts at the obvious, works in an obvious way and has comment
       | blocks at hard parts.
       | 
       | For this reason I despise most modern [web] projects, which have
       | a weak start, immediately drop into "services" and "components",
       | do one action per source file per 30-50 lines of code, which are
       | mostly imports and boilerplate, and have hundreds of these files.
       | You can never tell what it does because it does almost nothing
       | except formalities.
       | 
       | I also noted a tendency to use wrong paradigms for a language.
       | E.g. it may have no closures (imagine that in 202x) so they use
       | events as continuations for asynchronicity, which results in a
       | mess. Or it isn't immutable/functional, but they pretend it is,
       | which results in fragility.
       | 
       | The best projects are both close to their business and written in
       | a paradigm of the language used.
       | 
       |  _Was there someone enforcing good practices top down?_
       | 
       | Natural time pressure is the best bs cleaner, imo. You write
       | effing code, maybe have few hours a week to refactor strange
       | parts. With no time pressure a project naturally becomes massaged
       | by all members into the "likeable" form of their age.
        
       | 8474_s wrote:
       | I can't name a good one, but i have strong dislike for big
       | project codebases where even finding one file out of thousands
       | where the relevant code resides is a challenge: its never
       | something isolated but acts like some "component". The best i can
       | think of is one-person projects where organization is streamlined
       | as its actually has to be used by the author, not like a cog in a
       | giant project.
        
       | 999900000999 wrote:
       | Realistically any code base where the engineers had at least a
       | basic understanding of programming. You do not know suffering
       | until you've seen someone hard code basic variables, we're
       | talking about strings all over the place, and then they just copy
       | the function again to replace the strings .
       | 
       | I've legitimately left jobs over bad code. We're talking about
       | code that did nothing in reality. The best code bases have been
       | ones where I've been able to lead the direction. I get to know
       | exactly how things work. I'm privileged to have a job where I
       | essentially created the initial framework right now .
       | 
       | Plus I'm fully remote, life is pretty good.
        
         | naikrovek wrote:
         | > any code base where the engineers had at least a basic
         | understanding of programming
         | 
         | I felt this in my soul as soon as I read it. The number of
         | people writing code who should not even be speaking to others
         | because of how clearly they don't understand _anything_
         | technical is unfathomably high.
         | 
         | Our industry hires seat fillers and tells them to write
         | software. _And then they do_ , and it's every bit as bad as it
         | sounds.
        
       | karmakaze wrote:
       | Probably the microservices-based one for async video messaging
       | (i.e. Slack for video) for workers in the field. Each service was
       | small enough that we could do a blue-green deploy to prod in
       | about 2 minutes running only the service's tests and a tiny
       | (intentionally limited) set of about 8 system journey tests _(can
       | onboard a new user, user can create a contact /group, user can
       | send a common content-type message, user can receive messages,
       | user can react/respond to a message)_. Every commit to
       | main/master automatically either deployed to prod or broke the
       | CI/CD pipeline and needed to be fixed ASAP. Each service was also
       | well-known by team members that it literally could be rewritten
       | in a week or two if desired to change a key part of its design.
        
         | karmakaze wrote:
         | Thinking about what I wrote, I suppose my criteria for a good
         | codebase is one that has the lowest friction to change: a fast
         | edit/run/test/debug loop, a meaningful sense of security
         | (without dogma), and fast automated deployment/revert (via
         | blue/green). Given those a bad codebase can become a good one
         | again (by uncoordinated action of individuals) without
         | _everyone having to buy into_ a large investment.
        
       | llmblockchain wrote:
       | Any code I have ever touched.
        
       | ttfkam wrote:
       | Postgres. I don't code in C if I can avoid it, since it often
       | feels like an awful lot of extra typing while still having to
       | worry about memory safety. But the Postgres codebase is
       | extraordinarily well organized and respects the humans that work
       | with it with its intelligent handling of memory and judicious use
       | of macros.
       | 
       | I consider the core Postgres codebase to be the gold standard in
       | development even though it's in a language I do not prefer to
       | write in if given the choice.
       | 
       | Shout out to the pgrx folks. You're awesome!
       | https://github.com/pgcentralfoundation/pgrx
        
         | rurp wrote:
         | That is nice to hear, albeit unsurprising. Their public
         | documentation is some of the best that I have worked with.
         | Postgres is such an impressive project overall.
        
         | shutupnerd0000 wrote:
         | This codebase is 99.6% Rust according to GitHub.
        
           | radimm wrote:
           | That's PGRX - not PostgreSQL
        
           | ttfkam wrote:
           | As I said, I don't like coding in C.
        
           | rrr_oh_man wrote:
           | https://github.com/postgres/postgres
           | 
           | Postgres (core?) seems to be 85% C.
        
             | HaZeust wrote:
             | He's referencing pgrx:
             | 
             | https://github.com/pgcentralfoundation/pgrx
             | 
             | To be fair pgrx _was_ mentioned in OP under a positive
             | light, and OP _did_ say he likes avoiding coding in C, so
             | it was something to bring up.
        
           | jokoon wrote:
           | I laughed at this comment
           | 
           | sorry
        
       | mikewarot wrote:
       | The run time library for Turbo Pascal/Delphi for Windows was
       | completely documented, sane, and very easy to work with. The
       | working examples really helped.
       | 
       | The free Pascal RTL seems opaque in comparison. Their reliance on
       | and archaic help file build system keeps contributors away. Thus
       | it's poorly documented at best.
        
         | nurple wrote:
         | Was always an Anders fan because of his work in the
         | turbo/delphi community.
        
       | tiffanyh wrote:
       | I've seen a few people say 'google3'.
       | 
       | Q: is it actually the code that you loved, or simply the tooling
       | that exists?
       | 
       | (and if it's tooling, why can't that type of tooling be
       | replicated for other codebases outside of google?)
        
         | okdood64 wrote:
         | > (and if it's tooling, why can't that type of tooling be
         | replicated for other codebases outside of google?)
         | 
         | The elegance of the tooling from what I hear is that there's
         | tons of different tools maintained by different teams that work
         | seamlessly (and fast) together to produce google3 and all of
         | its supporting pieces.
         | 
         | But to answer your question, sure it can. But good luck
         | building your own. Google has been doing this since the 2000s.
         | 
         | And if you're a big company already, you've already bought into
         | your existing patterns & design choices; things like that are
         | VERY hard to change.
        
         | aleksiy123 wrote:
         | Its both. The tooling has a very direct impact on the quality
         | of the code.
         | 
         | I think the reason its not easy replicable is:
         | 
         | 1. It takes a ton of initial investment and ongoing maintenance
         | but its worth it when your code base is gigantic.
         | 
         | 2. There is a consistent set of top down enforced rules. With
         | the consistency it becomes much, much easier to build tight
         | integrations between tools.
         | 
         | (almost?) everything is buildable by a single build system
         | (blaze). When anyone can consistently build/test/run anything
         | in your codebase it becomes a lot easier to build a whole host
         | of potential tools like code search.
         | 
         | Probably someone can dive deeper than I can. But one thing I
         | learned the most important property for a code base to be
         | maintainable/scalable is consistency.
        
           | aleksiy123 wrote:
           | One more thought, is also how much other systems can utilize
           | the same tooling/workflows by just storing things in source
           | code. Things that would probably traditionally stored as
           | application state in a database are often stored instead in
           | google3 as config.
           | 
           | Things like on team rosters, on call rosters, code review
           | permissions, service permissions, feature flags.
           | 
           | All of it stored in google3 and can all utilize the same
           | consistent set of tooling like presubmit test, integration
           | tests, deployment automation, permissions, code search.
           | 
           | Its sort of like Infrastructure as Code but more.
        
             | kridsdale3 wrote:
             | I also love the built-in history diffing that this gives
             | you on those files.
        
             | nurple wrote:
             | Org as code
        
           | ryukafalz wrote:
           | > When anyone can consistently build/test/run anything in
           | your codebase it becomes a lot easier to build a whole host
           | of potential tools like code search.
           | 
           | As someone who hasn't worked for Google, how does Google's
           | implementation of this differ from e.g. Guix/nixpkgs? Being
           | able to easily build/test any available package is a big
           | reason I like using tools like those.
        
             | aleksiy123 wrote:
             | I'm unfortunately not that familiar with the internals.
             | 
             | I think the public bazel (public version of blaze) docs is
             | probably the best place to look.
             | 
             | https://bazel.build/basics/hermeticity
        
         | fragmede wrote:
         | That type of tooling _can_ be replicated. That 's why every
         | Xoogler tries so hard to get Bazel adopted on everything they
         | touch. Sometimes that's appropriate, sometimes it's not, but
         | that's why.
         | 
         | Bazel isn't the whole of everything though, the other piece
         | being exported is kubernetes, which isn't Google's borg, but
         | that's its roots. There's Apache Airflow if you need a workflow
         | engine like Sisyphus, no shortage of databases to choose from,
         | though now we're drifting into the operations side of things.
         | 
         | But basically, Google has invested untold millions of dollars
         | in the form of SWE and other engineering hours into making
         | Google3 operate. If anyone else invested that kind of dough,
         | with enough smarts, they'd also be able to make it a good
         | experience. The problem is few people have that kind of budget,
         | and even fewer invest in that thing, preferring to use free
         | tools instead.
         | 
         | What tool do you use to edit code, and how much did your
         | employer spend on that for you?
        
           | tiffanyh wrote:
           | So the development experience Github is marching towards
           | doesn't compare?
           | 
           | E.g., Codespaces, CoPilot, Actions, etc?
           | 
           | (genuinely curious, not trolling)
        
             | fragmede wrote:
             | It does! Working like Google is like that but having had
             | decades to build that dream, using the resources that only
             | a megacorp could bring to bear in the problem, and in a
             | monorepo. The fact that it's a monorepo is not to be
             | dismissed. git doesn't work for monorepos.
             | 
             | Codespaces is like CitC; I haven't been a Google since the
             | rise of AI so I can't comment on how the internal
             | equivalent to Copilot is; Actions is very primitive
             | compared to what Google has, but yeah, you can see where
             | it's going. it doesn't compare right now but it could,
             | eventually.
        
               | kridsdale3 wrote:
               | The Gemini-powered copilot is fine, but has so far, for
               | me, only done automated like "heres a simple method based
               | on the comment you wrote", or "I patched in a change to
               | the other callsites when you renamed this thing". I have
               | also not used it in 2 months so it's probably better
               | already.
               | 
               | I still mainly do my LLM assisted coding in chat-style
               | interfaces that are more like pair programming by mail.
               | 
               | I think the Gemini code assist is coming out as a
               | consumer product, I forgot the name, because Google is
               | awful at branding.
        
               | pcatach wrote:
               | > git doesn't work for monorepos.
               | 
               | Why not? Is it related to partial checkouts?
        
           | simoncion wrote:
           | > That's why every Xoogler tries so hard to get Bazel adopted
           | on everything they touch.
           | 
           | After burning an entire weekend like five years back trying
           | to get a released version of tensorflow to build from source
           | [0], I'm catastrophically disinclined to use Bazel for
           | anything.
           | 
           | I found its diagnostics to be utterly unhelpful, its
           | documentation to say nearly nothing I needed to know, and the
           | various Internet resources for the build system to be
           | somewhere between "as confused and lost as I was" and "total
           | fanboy who is so expert in the system that they are incapable
           | of speaking like anything other than an architecture
           | astronaut".
           | 
           | I'm sure it's legitimately fuckin amazing when you learn it
           | at a company that's big enough to have one or more entire
           | teams dedicated to internal developer tooling (and training
           | for and documentation of the same), but (at least in my
           | experience) for those of us on the outside, it's just bad,
           | bad, bad.
           | 
           | [0] The fucking thing wouldn't even build in the officially-
           | supplied "build tensorflow" Docker image. I was utterly
           | unable to find out why. I get that this indicates that the
           | tensorflow folks fucked up somehow, but the fact that I was
           | utterly unable to figure out how to understand WHAT they
           | fucked up is pretty damning.
        
             | dieortin wrote:
             | I drove the adoption of Bazel for my C++ project, which
             | certainly does not have an entire team (nor even an entire
             | person) dedicated to maintaining the build system.
             | 
             | We're having incredible build performance with it, and I
             | find its language much more readable than CMake or Make.
             | We're also using Bazel for things like packaging.
             | 
             | I would encourage you to give it another go.
        
       | pradn wrote:
       | Google's monorepo, and it's not even close - primarily for the
       | tooling:
       | 
       | * Creating a mutable snapshot of the entire codebase takes a
       | second or two.
       | 
       | * Builds are perfectly reproducible, and happen on build
       | clusters. Entire C++ servers with hundreds of thousands of lines
       | of code can be built from scratch in a minute or two tops.
       | 
       | * The build config language is really simple and concise.
       | 
       | * Code search across the entire codebase is instant.
       | 
       | * File history loads in an instant.
       | 
       | * Line-by-line blame loads in a few seconds.
       | 
       | * Nearly all files in supported languages have instant symbol
       | lookup.
       | 
       | * There's a consistent style enforced by a shared culture, auto-
       | linters, and presubmits.
       | 
       | * Shortcuts for deep-linking to a file/version/line make sharing
       | code easy-peasy.
       | 
       | * A ton of presubmit checks ensure uniform code/test quality.
       | 
       | * Code reviews are required, and so is pairing tests with code
       | changes.
        
         | pjungwir wrote:
         | > Entire C++ servers with hundreds of lines of code can be
         | built from scratch in a minute or two tops.
         | 
         | Hundreds, huh? Is this a typo? It makes me wonder if the whole
         | comment is facetious. Or do C++ programmers just have very low
         | expectations for build time?
        
           | jbyers wrote:
           | I suspect they meant "hundreds of thousands"
        
             | pradn wrote:
             | Yes, oops - fixed!
        
           | stefan_ wrote:
           | That's the beauty of C++, an absurdly slow build is just an
           | include away.
        
             | thfuran wrote:
             | Boost always helps prop up falling compile times.
        
               | spease wrote:
               | Has anyone done LLM training with template
               | metaprogramming? That seems like another excellent way to
               | keep Google's build servers warm.
        
               | mianos wrote:
               | Specially when your juggling is getting out of practice.
        
             | robodan wrote:
             | The public version of Google's build tool is Bazel (it's
             | Blaze internally). It has some really impressive caching
             | while maintaining correctness. The first build is slow, but
             | subsequent builds are very fast. When you have a team
             | working on similar code, everyone gets the benefit.
             | 
             | As with all things Google, it's a pain to get up to speed
             | on, but then very fast.
        
             | citizen_friend wrote:
             | Just wait until you try a "modern" language
        
         | dskloet wrote:
         | Code search not just across the entire code base but across all
         | of time.
        
         | ok_dad wrote:
         | I'm surprised they didn't turn that into a product, it sounds
         | great.
        
           | twunde wrote:
           | Parts have been. Sourcegraph is basically the code search
           | post built by ex-Googlers originally. Bazel is the open
           | source build tool. Sadly, most of these things require major
           | work to set up yourself and manage, but there's an alternate
           | present where Google built a true competitor to GitHub and
           | integrated their tooling directly into it.
        
             | snotrockets wrote:
             | Building tools for others is a competency that is under
             | rewarded at Google. They would never.
        
             | kridsdale3 wrote:
             | I've published my Google proprietary stuff (when we decided
             | to open source it) on GitLab, but they wouldn't let me do
             | it on GitHub.
        
         | tallowen wrote:
         | I always find these comments about interesting, having worked
         | at Facebook and Google, I never quite felt this way about
         | Google's Monorepo. Facebook had many of the features you listed
         | and quite performantly if not more so. Compared with working at
         | Facebook where there are no owners owners files and no
         | readability requirements, I found abstraction boundries to be
         | much cleaner at FB. At google, I found there was a ton of cruft
         | in Google's monorepos that were too challenging / too much work
         | for any one person to address.
        
           | pradn wrote:
           | OWNERS files rarely get in the way - you can always send a
           | code change to an OWNER. They are also good for finding
           | points of contact quickly, for files where the history is in
           | the far past and changes haven't been made recently.
           | 
           | Readability really does help new engineers get up to speed on
           | the style guide, and learn of common libraries they might not
           | have known before. It can be annoying - hell, I'll have to
           | get on the Go queue soon - but that's ok.
        
             | sawyna wrote:
             | This isn't true at all for OWNERS files. If you try
             | developing a small feature on google search, it will
             | require plumbing data through at least four to five layers
             | and there is a different set of OWNERS for each layer.
             | You'll spend at least 3 days waiting for code reviews to go
             | through for something as simple as adding a new field.
        
               | okdood64 wrote:
               | 3 days for a new change on the biggest service on the
               | planet? Not bad.
        
               | tallowen wrote:
               | I agree that it could be worse! Facebook has significant
               | (if not more) time spent and I found adding features to
               | news feed a heck of a lot easier than adding features
               | that interacted with google search. Generally a lot of
               | this had to do with the number of people needed to be
               | involved to ensure that the change was safe which always
               | felt higher at Google.
        
               | kridsdale3 wrote:
               | I have the same background: I find the code quality at G
               | to be quite a lot higher (and test pass-rate, and bug
               | report-rate lower) than News Feed, which was a total
               | shit-show of anything-goes. I still hold trauma from
               | being oncall for Feed. 70 bugs added to my queue per day.
               | 
               | The flip side is of course that I could complete 4 rounds
               | of QuickExperiment and Deltoid to get Product Market Fit,
               | in the time it takes to get to dogfooding for any feature
               | in Google.
        
               | codethief wrote:
               | I'm only an outside observer in this conversation but
               | could it be that the review process (or the lack thereof)
               | and the ease with which you can add new features has had
               | an impact on the quality of the software?
               | 
               | The thing is, in my experience as a user Facebook (the
               | product, not the former company) is absolutely riddled
               | with bugs. I have largely stopped using it because I used
               | to constantly run into severe UI/UX issues (text input no
               | longer working, scrolling doing weird things, abysmal
               | performance, ...), loading errors (comments & posts
               | disappearing and reappearing), etc. Looking at the
               | overall application (and e.g. the quality of the news
               | feed output), it's also quite clear that many people with
               | many different ideas have worked on it over time.
               | 
               | In contrast, Google search still works reasonably well
               | overall 25 years later.
        
               | esprehn wrote:
               | There are pretty different uptime and stability
               | requirements for a social product and web search (or
               | other Google products like Gmail). When news feed is
               | broken life moves on, when those products break many
               | people can't get any work done at all.
               | 
               | One of Google's major cultural challenges is imposing the
               | move slow and carefully culture on everything though.
        
               | zeroonetwothree wrote:
               | It's not considered ok for newsfeed to break. It would be
               | a massive issue that would command the full attention of
               | everyone.
        
               | esprehn wrote:
               | And yet folks who are on call for it say things like
               | this: https://news.ycombinator.com/item?id=40826497
        
             | tallowen wrote:
             | I think I have heard similar things from other googlers and
             | I think there might be two factors on why I think this:
             | 
             | - I worked on Google Assistant which was responsible for
             | integrating many services. This meant I had to work with
             | other peoples code way more regularly that many at google.
             | 
             | - I moved from FB to google - I'm not really sure how many
             | people have had this experience. I think many of my
             | colleagues at google found it surprising how many of the
             | things they thought were unique to google actually also
             | existed at FB.
             | 
             | At the end of the day, any of these processes have
             | pros/cons but I think the cruft of having APIs that are a
             | couple steps harder to evolve due to finding
             | Readability/Owners for everything you touch just makes
             | things slightly less cohesive and a trickier place to have
             | a "good" codebase.
             | 
             | When I worked at FB, I would frequently rebase my code on
             | Monday and find that, for example, the React framework
             | authors or another smaller infra team had improved the API
             | and had changed *every* callsite in the codebase to be
             | improved. This type of iteration was possible in certain
             | situations but was just much less common at google than at
             | fb.
        
               | snotrockets wrote:
               | > I think many of my colleagues at google found it
               | surprising how many of the things they thought were
               | unique to google actually also existed at FB.
               | 
               | Google workers are groomed to believe Google is the best,
               | and hence they are too. A corollary of that, then, is
               | that nobody else has it that good, when in fact, others
               | sometimes have it better.
        
               | kridsdale3 wrote:
               | I also made the move from FB to G and echo everything
               | said above. Googlers have a massive superiority complex.
               | In reality, it's naivete.
               | 
               | My 2 cents: OWNERS is fairly useful, if only as a form of
               | automating code reviewer selection. Readabilty is a
               | massive drag on org-wide productivity. I have had
               | diffs/CLs take MONTHS to be approved by every Tom Dick
               | and Harry whose claws were added to my code and made me
               | re-design whole project approaches, and they were only
               | there because they're supposed to check if my new-lines
               | are in the right spot for that language. I thought about
               | quitting.
        
               | wavemode wrote:
               | People really underestimate how much productivity drain
               | there is in having a bad code review culture. One of the
               | worst things about working at Amazon was that any
               | feedback on a merge request, no matter how small,
               | required you to request a re-review.
        
               | ynx wrote:
               | +1.
               | 
               | Going from FB to $REDACTED to Oculus was a pretty wild
               | ride, there were a lot of different cultures, though I
               | think generally speaking the best qualities filtered
               | through.
               | 
               | (also, howdy former teammate)
        
               | kridsdale3 wrote:
               | (trying to recall unixname to human name mapping.....)
        
               | elevatedastalt wrote:
               | Huh? Facebook has a lot of that infra because ex-Googlers
               | built it there. It takes an insane amount of delusion to
               | notice something common between a father and a son and
               | say that the dad inherited it.
        
               | pradn wrote:
               | No, we're quite aware the world outside has been catching
               | up. There's even a famous doc by a senior director about
               | it...
        
           | AnotherGoodName wrote:
           | Same and another vote for meta. Meta made the language fit
           | their use case. Go into bootcamp change the search bar text
           | to 'this is a search bar!' press F5 and see the change (just
           | don't ship that change ;D). It's incredibly smooth and easy.
           | 
           | Googles a mess. There's always a migration to the latest
           | microservices stack that have been taking years and will take
           | many more years to come.
           | 
           | Like meta just changed the damn language they work in to fit
           | their needs and moved on. Google rewrites everything to fit
           | the language. The former method is better in a large
           | codebase. Meta is way easier to get shit done to the point
           | that google was left in the dust last time they competed with
           | meta.
        
             | quicklime wrote:
             | I think what you're saying is true for www, but not fbcode,
             | and the later starts to look a lot like google3. I agree
             | though, Meta's www codebase has the best developer
             | experience in the industry.
        
           | codethief wrote:
           | For non-Googlers like me, here's some background about
           | "readability":
           | 
           | https://www.moderndescartes.com/essays/readability/
        
           | mattnewton wrote:
           | Huh, also having worked at both I had exactly the opposite
           | experience. Google's tools looked ugly but just worked. At
           | meta there were actually multiple repos you might have to
           | touch and tools worked unreliably across them. Owners files
           | made sure there was less abandoned code and parent owners
           | woild be found by gwsqueue bots to sign off on big changes
           | across large parts of the repo by just reading these files.
        
           | tantalor wrote:
           | > no owners owners files and no readability requirements
           | 
           | Move fast and break things, right?
        
             | zeroonetwothree wrote:
             | I don't think that owners files are the best way to ensure
             | things don't break.
        
         | robertsdionne wrote:
         | * https://abseil.io/resources/swe-book/html/ch16.html
         | 
         | * https://abseil.io/resources/swe-book/html/ch17.html
        
         | WWWMMMWWW wrote:
         | Google's code, tooling and accompanying practices are
         | developing a reputation for being largely useless outside
         | Google ... and many are starting to suspect it's alleged value
         | even inside Google is mostly cult dogma.
        
           | wiseowise wrote:
           | > Google's code, tooling and accompanying practices are
           | developing a reputation for being largely useless outside
           | Google ...
           | 
           | Not that I don't believe you, but where do you see this?
        
             | matthewfcarlson wrote:
             | I haven't worked at google, but this is something I have
             | heard from a few people. Reputation is largely word of
             | mouth, so it checks out for me. I suspect the skills/tools
             | at most large companies are increasingly less transferrable
             | as they continue to grow in scale and scope.
        
             | robertakarobin wrote:
             | I can vouch for it. It's the main reason I quit: none of
             | the "hard" skills necessary to code at Google were
             | transferrable anywhere outside of Google. It would have
             | been easy enough to skate and use "soft" skills to move up
             | the management ladder and cash big checks, but I wasn't
             | interested in that.
             | 
             | The reason it's not transferrable is that Google has its
             | own version of EVERYTHING: version control, an IDE, build
             | tools, JavaScript libraries, templating libraries, etc,
             | etc. The only thing I can think of that we used that wasn't
             | invented at Google was SCSS, and that was a very recent
             | addition. Google didn't even use its own open-source
             | libraries like Angular. None of the technologies were
             | remotely usable outside Google.
             | 
             | It might sound cool to use only in-house stuff, and I
             | understand the arguments about licensing. But it meant that
             | everything was poorly-documented, had bugs and missing
             | features that lingered for years, and it was impossible to
             | find a SME because whoever initially built a technology had
             | moved on to other things and left a mess behind them.
             | 
             | Some people may be able to deal with the excruciating
             | slowness and scattered-ness, and may be OK with working on
             | a teeny slice of the pie in the expectation that years
             | later they'll get to own a bigger slice. But that ain't me
             | so I noped out as soon as my shares vested.
        
               | Banditoz wrote:
               | Their own IDE? Is it web based? So they don't use, say,
               | IntelliJ/Eclipse for Java projects?
        
               | robertakarobin wrote:
               | It's basically a fork of VSCode.
        
               | makeramen wrote:
               | That's a recent development, used to be something else
               | altogether.
        
               | audiodude wrote:
               | 12 year current Googler here. You are absolutely correct
               | about "Google has its own version of EVERYTHING". Midway
               | through my current career, I started to get existential
               | dread about the fact that I wasn't "up to date" on any
               | current development practices or frameworks.
               | 
               | Partly, this was assuaged through participating in open
               | source projects in my free time. That's how I learned
               | Docker, Github workflow, React, Vue, Bootstrap, Tailwind,
               | etc.
               | 
               | But at the same time, I think it is a mistake to consider
               | working with tools/languages/frameworks to be the only
               | "hard" skills. Galaxy brain is realizing that anyone can
               | learn a language/framework/workflow in a month or so. The
               | real work is applying sound principles to the design and
               | production of meaningful artifacts within those systems.
        
               | kaba0 wrote:
               | Though credit where it's due, some of their tools really
               | have been years ahead of anything outside of google, e.g.
               | the closure compiler that made javascript development
               | scalable.
        
             | openmajestic wrote:
             | I have seen this discussed in hiring decisions. I don't
             | know that it played a large factor in a decision, but lack
             | of experience in the standard tools/practices/terms of
             | software development because of a career at Google was
             | definitely a discussion point.
        
             | makeramen wrote:
             | I had a bunch of very tenured teammates that didn't really
             | know how to use git, so there were only a few of us
             | comfortable enough integrating and interacting with an open
             | source dependency repo.
        
           | worik wrote:
           | > Google's code, tooling and accompanying practices are
           | developing a reputation for being largely useless outside
           | Google .
           | 
           | It is almost a tautology
           | 
           | Why would they be useful for domains they are not designed
           | for?
        
             | tonynator wrote:
             | Most companies don't use proprietary tools for everything.
        
           | jart wrote:
           | Google used to have a near monopoly on the most expensive,
           | educated, devoted, and conscientiously willful people and
           | imposed very few demands on their time. The lengths to which
           | they were willing to go, to make everything they did with the
           | tools pleasant and elegant, was orders of magnitude beyond
           | anything I'd seen.
           | 
           | Some of us thought that the magic of these people would be
           | imbued in the dev tools that they created, so if enterprises
           | adopted the tools, then they'd reap the benefits of that same
           | magic too. But this simply wasn't true. The tools didn't
           | actually matter; it was the way they used them.
           | 
           | For example, when other companies started adopting tools like
           | Bazel (open source Blaze) they wanted features like being
           | able to launch ./configure scripts inside Bazel, which
           | totally violates the whole point of Bazel, and never would
           | have been allowed or even considered inside Google. The Bazel
           | team was more than happy to oblige, and the users ended up
           | with the worst of all worlds.
        
             | WWWMMMWWW wrote:
             | Google's systems were designed to index mountains of low
             | value data at hitherto unseen scale, and they're good at
             | that. But, to-the-second system-wide precision with full
             | audit trails ... not so much.
             | 
             | You keep seeing startups with ex-Googlers that think they
             | can "disrupt" Fintech with Google's "secret sauce" ... this
             | tends to go badly.
             | 
             | I've had to clean up one of these messes where, in all
             | seriousness, even a pre-2000 LAMP stack (never mind Java)
             | implemented by people who understood the finance domain
             | would have worked better.
        
             | dieortin wrote:
             | If Google open sourced their BUILD files for public
             | libraries, we wouldn't be able to use workarounds...
             | Migrating something complex like ffmpeg to Bazel is not
             | something trivial.
             | 
             | Bazel is an awesome tool though, I'm very glad it was open
             | sourced and receives constant attention from Google.
        
         | bbor wrote:
         | I'm just one incompetent dev, but I'll throw this in the convo
         | just to have my perspective represented: every individual part
         | of the google code experience was awesome because everyone
         | cared a ton about quality and efficiency, but the overall
         | ecosystem created as a result of all these little pet projects
         | was to a large extent unmanaged, making it difficult to operate
         | effectively (or, in my case, basically at all). When you join,
         | one of the go-to jokes in their little intro class these days
         | is "TFW you're told that the old tool is deprecated, but the
         | new tool is still in beta"; everyone laughs along, but
         | hopefully a few are thinking "uhhh wtf".
         | 
         | To end on as nice of a note as possible for the poor Googs: of
         | all the things you bring up, the one I'd highlight the biggest
         | difference on is Code Search. It's just _incredible_ having
         | that level of deep semantic access to the huge repo, and people
         | were way more comfortable saying "oh let's take a look at that
         | code" ad-hoc there than I think is typical. That was pretty
         | awesome.
        
           | aleksiy123 wrote:
           | Imho the reason for the deprecated and beta thing is because
           | there is a constant forward momentum.
           | 
           | Best practices, recommendations and tooling is constantly
           | evolving and requires investment in uptake.
           | 
           | I sometimes feel like everything is legacy the moment it's
           | submitted and in a constant state of migration.
           | 
           | This requires time and resources that can slow the pace of
           | development for new features.
           | 
           | The flips side is this actually makes the overall codebase
           | less fractured. This consistency or common set of assumptions
           | is what allows people to build tools and features that work
           | horizontally across many teams/projects.
           | 
           | This constant forward momentum to fight inconsistency is what
           | allows google3 to scale and keep macro level development
           | velocity to scale relative to complexity.
        
             | bbor wrote:
             | That's all well said, thanks for sharing your perspective!
             | Gives me some things to reflect on. I of course agree
             | re:forward momentum, but I hope they're able to regain some
             | _grace_ in that momentum with better organization going
             | forward. I guess I was gesturing to people "passing the
             | buck" on hard questions of team alignment and mutually
             | exclusive decisions. Obviously I can't cite specifics bc of
             | secrecy and bad memory in equal amounts, so it's very
             | possible that I had a distorted view.
             | 
             | I will say, one of the things that hit me the hardest when
             | the layoffs finally hit was all the people who have given
             | their professional lives to making some seriously
             | incredible dev tools, only to be made to feel disposable
             | and overpaid so the suits could look good to the
             | shareholders for a quarter or two. Perhaps they have a
             | master vision, but I'm afraid one of our best hopes for an
             | ethical-ish megacorp--or at least vaguely pro social--is
             | being run for short term gain :(
             | 
             | However that turns out for society, hopefully it ends up
             | releasing all those tools for us to enjoy! Mark my words,
             | colab.google.com will be shockingly popular 5y from now, if
             | they survive till then
        
               | kridsdale3 wrote:
               | I guarantee you that the master vision is exactly what
               | you wrote.
               | 
               | Google is not a software company, it is an advertising
               | system providing cash-flow to a hedge fund that comprises
               | a large part of every pension and retirement fund in
               | America. It's far too important as simply a financial
               | entity to risk anything on .... _product development_.
        
         | teaearlgraycold wrote:
         | My experience with google3 was a bit different. I was shocked
         | at how big things had gotten without collapsing, which is down
         | to thousands of Googlers working to build world-class internal
         | tooling. But you could see where the priorities were. Code
         | Search was excellent - I'd rate it 10/10 if they asked.
         | 
         | The build system always felt more like a necessary evil than
         | anything else. In some parts of google3 you needed three
         | _separate_ declarations of all module dependencies. You could
         | have Angular 's runtime dependency injection graph, the
         | Javascript ESM graph, and the Blaze graph which all need to be
         | in sync. Now, the beautiful part was that this still worked.
         | And The final Blaze level means you can have a Typescript
         | codebase that depends on a Java module written in a completely
         | unrelated part of google3, which itself depends on vendored C++
         | code somewhere else. Updating the vendored C++ code would cause
         | all downstream code to rebuild and retest. But this is a multi
         | billion dollar solution to problems that 99.99% of companies do
         | not have. They are throwing thousands of smart people at a
         | problem that almost everyone else has "solved" by default
         | simply by being a smaller company.
         | 
         | The one tooling I think every company could make use of but
         | doesn't seem to have were all of the little hacks in the build
         | system (maybe not technically part of Blaze?). You could
         | require a developer who updates the file at
         | /path/to/department/a/src/foo.java to simultaneously include a
         | patch to /path/to/department/b/src/bar.java. Many files would
         | have implicit dependency on each other outside of the build
         | graph and a human is needed to review if extra changes are
         | needed. And that's just one of a hundred little tricks project
         | maintainers can employ.
         | 
         | The quality of the code was uniformly at least "workable" (co-
         | workers updating parts of the Android system would probably not
         | agree with that - many critical system components were written
         | by one person poorly who soon after quit).
        
           | SR2Z wrote:
           | > But this is a multi billion dollar solution to problems
           | that 99.99% of companies do not have.
           | 
           | I know it's trendy for people to advocate for simple
           | architectures, but the honest-to-god truth is that it's
           | insane that builds work ANY OTHER WAY. One of the highest
           | priorities companies should have is to reduce siloing, and I
           | can barely think of a better way to guarantee silos than by
           | having 300 slightly different build systems.
           | 
           | There is a reason why Google can take a new grad SWE who
           | barely knows how to code and turn them into a revenue
           | machine. I've worked at several other places but none of them
           | have had internal infrastructure as nice as the monorepo; it
           | was the least amount of stress I've ever felt deploying huge
           | changes.
           | 
           | Another amazing thing that I don't see mentioned enough was
           | how robust the automatic deployments with
           | Boq/Annealing/Stubby were. The internal observability library
           | would automatically capture RPC traces from both the client
           | and server, and the canary controller would do a simple
           | p-test on whether or not the new service had a higher error
           | rate than the old one. If it did? The rollback CL would be
           | automatically submitted and you'd get a ping.
           | 
           | This might sound meh until I point out that EVEN CONFIG
           | CHANGES were versioned and canaried.
        
             | kridsdale3 wrote:
             | I've worked at the majority of FAANG.
             | 
             | Facebook's build system works the same as Googles, because
             | most of FB's infra was made by ex-Googlers around 10-15
             | years ago. The worst thing I can say about Blaze is
             | basically already pointed out above, sometimes you need to
             | write little notes to the presubmit system to ensure cross-
             | boundary updates. Whatever, it's all text files in the end.
             | 
             | The wildest was at Apple. It's just as you said, 300 build
             | systems. Not only that, but 300 source code repositories!
             | Two teams in the same hall that hang out all the time could
             | be using git and svn, for no good reason besides what
             | someone wrote "init" in 20 years ago. There was no cross
             | team communication, by design, because Steve Jobs was
             | paranoid. Their sync mechanism was to build the entire
             | stack once a night, and force everyone to full-reinstall
             | their OS and toolchain to "rebase". Insane.
        
             | teaearlgraycold wrote:
             | I definitely agree most companies should use a monorepo.
             | Most companies don't need Blaze, though.
             | 
             | And the whole rollout system was excellent. I wish that
             | tech was standard but I have a vague idea of how much work
             | that would be to implement and few companies will be able
             | to afford to get that right.
             | 
             | Edit: I forgot to mention - I absolutely hated Cider and
             | all of the included plugins. Sure the code at Google was
             | fine but the code editing experience destroyed all of the
             | fun of coding. Is that function signature correct? You'll
             | find out in 45 seconds when the Intellisense completes! And
             | when I was there Copilot was a thing outside of Google but
             | we were not allowed to use any AI (even Google's own AI) to
             | write code. The whole situation was so bad I wrote a few
             | paragraphs about it in my offboarding survey.
        
           | codethief wrote:
           | > You could require a developer who updates the file at
           | /path/to/department/a/src/foo.java to simultaneously include
           | a patch to /path/to/department/b/src/bar.java.
           | 
           | Could you elaborate on how this worked exactly?
        
             | teaearlgraycold wrote:
             | There's a configuration directive you put in a plain text
             | file in the monorepo which lets you configure:
             | 
             | * File A's path
             | 
             | * File B's path
             | 
             | * The message shown on the commit's review page if B isn't
             | updated when A is updated
             | 
             | * How a developer can override the alert (Which would be a
             | custom named directive added to a commit message, like
             | "SKIP_FILE_SYNC_ALERT=true")
             | 
             | You then need to either commit a diff to file B when file A
             | is changed or override it in order to get the commit added
             | to HEAD. This is just one of many different "plugins" for
             | the CI system that can be configured with code.
        
               | codethief wrote:
               | Thanks!
        
         | scubbo wrote:
         | Interesting to note that almost all of these are to do with
         | tooling _around_ the codebase, not the contents _of_ the
         | codebase!
        
           | kridsdale3 wrote:
           | Just like the man is the product of his genetic code, the
           | codebase is invariably the product of the constraints on its
           | edits enforced by tooling.
        
             | nathan_douglas wrote:
             | so we beat on, commits against the tooling, borne back
             | ceaselessly into the technical debt
        
             | remram wrote:
             | Is that true, though? Is the code itself good? Because it
             | is sorely absent from GP's list...
             | 
             | If you are trying to say that people can't make bad code
             | with good tools, I don't agree.
        
         | dheera wrote:
         | Why do so many people like monorepos?
         | 
         | I tend to much prefer splitting out reusable packages into
         | their own repos with their own packaging and unit tests and
         | tagging to whatever version of that package. It makes it MUCH
         | easier for someone to work on something with minimal overhead
         | and be able to understand every line in the repo they are
         | actually editing.
         | 
         | It also allows reusable components to have their own
         | maintainers, and allows for better delegation of a large team
         | of engineers.
        
           | Tyr42 wrote:
           | I can change a dependency and my code at the same time and
           | not need to wait for the change to get picked up and deployed
           | separately. (If they are in the same binary. Still need cross
           | binary changes to be made in order and be rollback safe and
           | all that.)
        
           | radicality wrote:
           | Have you ever worked at FB / Google / whatever other company
           | has huge mono repo with great tooling?
           | 
           | I went from many years at FB, to a place like you describe -
           | hundreds of small repos, all versioned. It's a nightmare to
           | change anything. Endless git cloning and pulling and rebase.
           | Endless issues since every repo ends up being configured
           | slightly differently, and very hard to keep the repo metadata
           | (think stuff like commit rules, merge rules, etc) up to date.
           | It's seriously much harder to be productive than with a well-
           | oiled monorepo.
           | 
           | With a monorepo, you wanna update some library code to
           | slightly change its API? Great, put up a code change for it,
           | and you'll quickly see whether it's compatible or not with
           | the rest of the whole codebase, and you can then fix whatever
           | build issues arise, and then be confident it works everywhere
           | wherever it's imported. It might sound fragile, but it really
           | isn't if the tooling is there.
        
             | dheera wrote:
             | I have worked at a company that has huge monorepos and bad
             | tooling.
             | 
             | Tooling isn't the problem though, the problems are:
             | 
             | - multiple monorepos copying code from each other, despite
             | that code should be a library or installable python package
             | or even deb package of its own
             | 
             | - you will never understand the entire monorepo, so you
             | will never understand what things you might break. with
             | polyrepos different parts can be locked down to different
             | versions of other parts. imagine if every machine learning
             | model had a copy of the pytorch source in it instead of
             | just specifying torch==2.1.0 in requirements.txt?
             | 
             | - "dockerize the pile of mess and ship" which doesn't work
             | well if your user wants to use it inside another container
             | 
             | - any time you want to commit code, 50000 people have
             | committed code in-between and you're already behind on 10
             | refactors. by the time you refactor so that your change
             | works, 4000 more commits have happened
             | 
             | - the monorepo takes 1 hour to compile, with nothing to
             | compile and unit test only a part of it
             | 
             | - ownership of different parts of the codebase is difficult
             | to track; code reviews are a mess
        
               | joshuamorton wrote:
               | All of these are solved with tooling.
               | 
               | - dont have multiple monorepos
               | 
               | - use blaze or similar and run all downstream tests, the
               | binary for your ml model includes pytorch whether you
               | build from source or requirements.txt.
               | 
               | - other people committing doesn't matter if you are
               | isolating presubmits and similar.
               | 
               | - using a blaze-like you never compile the whole monorepo
               | 
               | - code owners etc. makes this straightforward.
               | 
               | Like, as someone whose career has been mostly at Google,
               | these are not problems I encounter at all, or only in
               | cases where you'd have similar scope of problems no
               | matter the repo structure.
        
               | hughesjj wrote:
               | > other people committing doesn't matter if you are
               | isolating presubmits and similar.
               | 
               | Could you elaborate on this one?
        
               | joshuamorton wrote:
               | If you only need to run the tests you affect, and only
               | need to sync and update files touched in your CL,
               | external changes are generally not impactful, sync &
               | submit is a quick process even if people are submitting
               | things elsewhere.
               | 
               | It's only a problem if someone submits a file you're
               | touching, in which case you just have standard merge
               | conflict issues.
        
               | audiodude wrote:
               | I think all of your problems are root caused by the
               | phrase "multiple monorepos". This sounds more like
               | "polyrepos" or something, multiple siloed repos that
               | themselves might contained disparate projects, languages
               | and tooling.
               | 
               | Google3 is a true monorepo. 99% of code in the company is
               | one repo, with minor exceptions for certain open source
               | projects and locked down code.
               | 
               | Edit: for example, you can change YouTube code, Google
               | Assistant code, a Maps API, some config for how URLs are
               | routed at top level load balancers, etc all in one CL if
               | you really wanted to/needed to.
        
             | Thaxll wrote:
             | How do you handle tagging, the whole repo gets a new tag /
             | version every time something change?
        
               | audiodude wrote:
               | Yes, all changes (CLs) are atomic operations. The CL
               | number reflects a specific moment in time for the entire
               | repo.
        
           | sghill wrote:
           | Up to what scale?
           | 
           | This works well for a couple dozen repos per team in my
           | experience. It's also my preferred way to work.
           | 
           | It doesn't scale so well to hundreds of repos per team
           | without significant tooling. At some point anything cross-
           | cutting (build tool updates, library updates, etc) becomes
           | hard to track. Repos are left behind as folks change teams
           | and teams are reorg'd.
           | 
           | I've never worked in a monorepo, but I can see the appeal for
           | large, atomic changes especially.
        
           | zerd wrote:
           | If you have a reusable component in a separate repository and
           | need a change, you have to submit that, merge, release, then
           | bump the version in the downstream project to use it. Then if
           | someone else uses the project updates but hits an issue that
           | you introduced they have to go fix it, perhaps a month later,
           | with no context of what changed. Or they just don't upgrade
           | the version, and reimplement what they need. With a monorepo
           | it would be one change, and your change breaking someone
           | else's would get flagged and fixed with the code change. I've
           | seen the amount of shared code get less and less and more
           | stale with polyrepo.
        
         | rkagerer wrote:
         | Question I've always wondered: Does Google's monorepo provide
         | all its engineers access to ALL its code?
         | 
         | If yes, given the sheer number of developers, why haven't we
         | seen a leak of Google code in the past (disgruntled employee,
         | accidental button, stolen laptop, etc)?
         | 
         | Also how do they handle "Skunkworks" stlye top-secret projects
         | that need to fly under the radar until product launch?
        
           | rkagerer wrote:
           | Edit - I guess there hasn't been zero leaks:
           | https://searchengineland.com/google-search-document-leak-
           | ran...
        
           | pradn wrote:
           | The very very important stuff is hidden, and the only two
           | examples anyone ever gives are core search ranking algorithms
           | and the self-driving car.
           | 
           | Even the battle-tested hyper-optimized, debugged-
           | over-15-years implementation of Paxos is accessible. Though
           | I'm sure folks could point out other valuable
           | files/directories.
        
             | kccqzy wrote:
             | Former employee here. I remember a third example: the anti-
             | DoS code is hidden. I remember this because I needed to do
             | some very complicated custom anti-DoS configuration and as
             | was my standard practice, I looked into how the
             | configuration was being applied. I was denied access.
             | 
             | Fourth example: portions of the code responsible for
             | extracting signals from employees' computers to detect
             | suspicious activity and intrusion. I suspect it's because
             | if an employee wants to do something nefarious they
             | couldn't just read the code to figure out how to evade
             | detection. I only knew about this example because that
             | hidden code made RPC calls to a service I owned; I changed
             | certain aspect of my service and it broke them. Of course
             | they fixed it on their own; I only got a post-submit
             | breakage notification.
        
           | robodan wrote:
           | Partial check outs are standard because the entire code base
           | is enormous. People only check out the parts they might be
           | changing and the rest magically appears during the build as
           | needed.
           | 
           | There are sections of the code that are High Intellectual
           | Property. Stuff that deals with spam fighting, for example. I
           | once worked on tooling to help make that code less likely to
           | be accidentally exposed.
           | 
           | Disclaimer: I used to work there, but that was a while back.
           | They probably changed everything a few times since. The need
           | to protect certain code will never go way, however.
        
           | cmrdporcupine wrote:
           | Google3 monorepo source isn't, by policy, supposed to leave
           | the corp network workstations, and can't even be on your
           | corporate provided laptop (except for some edge cases in
           | mobile apps dev). Even during full COVID lockdown WFH we had
           | to remote into our machines. (I worked on embedded stuff and
           | had to compile on my office workstation, scp the binaries
           | home, and flash my device, and repeat. Super slow cycle.)
           | 
           | So, anyways, source code being basically on-premise only and
           | on machines that they can fully audit and control... Would
           | you be stupid enough to "cp -r srccheckout /media/MYUSBSTICK"
           | on such a box?
           | 
           | Also believe it or not they used to have a very open internal
           | culture at Google because the bulk of employees genuinely
           | liked the company and its stated mission and there was a bit
           | of a social contract that seemed to be upheld. Stuff didn't
           | generally leak out of the wide open all hands, even. Past
           | tense.
        
           | tfsh wrote:
           | Within the monorepo there is the notion of "silos" where
           | access to directories can be restricted to groups of
           | people/bots. Though I believe that's exceedingly rare, I've
           | never come across one.
        
           | kccqzy wrote:
           | It's extremely easy to detect a disgruntled employee making a
           | copy of source code. There's no accidental button to leak.
           | There's no source code on laptops as policy doesn't allow it,
           | with limited exceptions only.
           | 
           | But there was a giant leak a long time ago. It was called
           | Operation Aurora done by China. Legend has it that to this
           | date the Chinese search engine Baidu still uses stolen code
           | from Google.
        
           | bhawks wrote:
           | Google laptops and workstations (anything that can actually
           | access srcfs to get this data) are extremely monitored and
           | controlled.
           | 
           | Very critical stuff (ranking, spam/abuse, etc) can be further
           | protected via silos which lock down sections of the code base
           | (but still allow limited interactions with the build).
           | 
           | Google spent significant engineering $$$ into its development
           | tools and policies (generally building custom with no intent
           | to ever monetize vs buying). I don't see a company today, in
           | this climate, that would emulate that decision.
        
           | emmelaich wrote:
           | I suspect this is part of the interview process and why it
           | takes so long and so many people.
           | 
           | Character and trustworthiness is extremely important.
        
             | kccqzy wrote:
             | The Google interview process is overly focused on algorithm
             | skills and absolutely does not select for character and
             | trustworthiness. In fact the leaks from Google to the news
             | started circa 2017 and in response the leadership basically
             | neutered internal forums like TGIF and memegen. Remember
             | the Damore incident? While Damore was wrong, it wouldn't be
             | as big of a deal if the incident wasn't leaked to the
             | press. It's clear that Google would be a much better
             | company if its interview process actually accounted for
             | character and trustworthiness.
             | 
             | The old article, _Three Years of Misery Inside Google, the
             | Happiest Company in Tech_ is still the best description of
             | what went wrong inside Google:
             | https://www.wired.com/story/inside-google-three-years-
             | misery...
        
         | Ocerge wrote:
         | I recently left Google and knew it was going to be a step down
         | from Google's build ecosystem, but I wasn't prepared for how
         | far a step down it would be. It's the only thing I miss about
         | the place, it' so awesome.
        
           | pmb wrote:
           | People who have never had it have no concept of how much they
           | are missing. It's so frustrating.
        
         | Thaxll wrote:
         | Most of those arguments are not about code quality though.
        
       | ramses0 wrote:
       | It's kindof a dumb answer, but `abcde` (A Better CD Encoder)
       | continues to be my go-to reference for "everything shell".
       | 
       | https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=summ...
       | 
       | https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
       | 
       | ...it's just so damned... like... it _is_ 5000 lines of bash, but
       | given the constraints and the problem domain, it's an incredible
       | (and well-organized) feat.
       | 
       | If I ever question "How would I do something like XXX in shell?",
       | I can usually find some clues, inspirations, or answers in the
       | `abcde` codebase.
        
         | e12e wrote:
         | > https://git.einval.com/cgi-
         | bin/gitweb.cgi?p=abcde.git;a=blob...
         | 
         | Interesting. I was a bit surprised by the use of echo in the
         | usage() function (and not cat and a heredoc) -- but that
         | certainly looks like a very structured shell script.
         | 
         | Thanks for sharing.
        
       | paxys wrote:
       | One that was written from scratch, by me. All others are
       | terrible.
        
         | ChrisMarshallNY wrote:
         | Heh. Can relate.
         | 
         | Even my own codebases/config. management systems are difficult,
         | if I haven't dealt with them in a while.
         | 
         | However, I am pretty good at understanding old stuff I did,
         | mainly because I leave a legacy for myself.
         | 
         | https://littlegreenviper.com/leaving-a-legacy/
        
         | wvenable wrote:
         | ...unless I haven't looked at it in a few years.
        
         | tiffanyh wrote:
         | The problem I have is, I'll revisit code I wrote 1-2 months
         | prior and swear someone else wrote it.
        
           | pavel_lishin wrote:
           | Same.
           | 
           | I _know_ I wrote perfect, maintainable code that can easily
           | live in my brain.
           | 
           | But when I go to work on it again a few months since I wrote
           | it, it seems like some _asshole_ stepped in and overwrote my
           | beautiful code with a bunch of jumbly mish-mash and had the
           | gall to overwrite the git commit history with it, using the
           | exact same commit messages, and my own name!
        
             | lagrange77 wrote:
             | Right. I even quietly mumble 'asshole' while reading it.
        
           | takeda wrote:
           | This is the weird thing. I heard it a lot and I don't know if
           | people just over-exaggerate or maybe I'm just forgetting
           | things quicker, but I don't have that. I come to the code
           | after month or two and it is the same as if I came a week or
           | two. I also don't feel ashamed of it.
           | 
           | I also got compliments of my code in the past.
           | 
           | I do try to make my code readable and avoid being clever.
           | Basically my rule of thumb is that if I need to write a
           | comment about block of code what it does, then that code is
           | too complex.
        
             | tazjin wrote:
             | Same. I think it's a meme that people keep repeating, and
             | for some fraction of them it's true.
        
             | doctor_eval wrote:
             | A month or two is nothing. Go back to it after 5 years.
        
           | grotorea wrote:
           | Past me is someone else.
           | https://en.wikipedia.org/wiki/Ship_of_Theseus
        
         | intelVISA wrote:
         | We were all thinking it, thank you for having The Courage.
        
         | stephenhandley wrote:
         | This entire website in a single comment
        
         | ohyes wrote:
         | Mine is always terrible and the others are terrible as well.
        
         | mikeqq2024 wrote:
         | One day I got an unexpected email from someone asking
         | permission to use a piece of code on github I forgot I wrote
         | almost 10 years ago without a license clause. I was very
         | surprised and confused because I forgot I wrote that code, and
         | I was not involving coding for quite some years and my github
         | profile was blank without any contact information. I asked. It
         | happened the guy simply made a search and found a post on a
         | forum with my nickname, and email address - newly created
         | recently. At that moment, it clicked and I felt some unusual
         | arrangment. I quickly added an open source license, and the
         | guy, CTO of a decent company, offered to make a donate. That
         | made me a day.
        
       | ryancnelson wrote:
       | Illumos, and specifically the SmartOS distribution.
       | 
       | Being ringside for the fork of Sun's OpenSolaris, and watching
       | absolute geniuses steward a thing like that into a useful product
       | and a build process was foundational for my understanding of how
       | and why process is important, as well as how to get things fixed
       | in the real world, while not muddying the idea of the pure design
       | principles. A balance can exist!
        
       | jowdones wrote:
       | >> Ask HN: What is the best code base you ever worked on?
       | 
       | LOL. Obviously. Mine! :)
        
       | lanstin wrote:
       | AOL's server code was excellent. It was written by some people
       | that had done similar things for BITNET and had a deep
       | understanding of Unix programming and how to use event loops for
       | scalability.
       | 
       | It was in C, but the process was expected to run for months
       | without crashing or running out of memory; if it had an abnormal
       | exit, you'd (the owner as set in some code thing) get an email
       | with the core backtrace. If it had a memory leak, ops would be on
       | you quickly. Ops was considered a primary driver of requirements
       | to dev, and absolutely everything could be reloaded into a
       | running server without restarting it. There was a TCP control
       | port where a TCL interpreter inside the service was exposed and
       | generally you wrote the (very simple, CS 101 style) TCL commands
       | to manage the server. It was a "No Threads Kernel", scaled to the
       | dozens or hundreds of physical machines communicating over a very
       | well managed network, and most 1 process per core, and 1 core for
       | the OS. The 200 or so unix developers (as we were called) had a
       | common understanding of how the framework worked and if you were
       | just writing app code it was basically impossible to write slow
       | services. We had technical writers that would interview the
       | developers and write books that could be handed to outside
       | developers and lead to a successful integration with no developer
       | time spent.
       | 
       | The NTK was primarily for sending msgs over the network - we had
       | a principle to never write to disk (which were pretty slow in the
       | 1990s), so everything was just a server to get network messages
       | and then send out other messages in reponse and then assemble the
       | replies/timeouts and send back a msg to the caller. All done over
       | persistent connections established by the infrastructure, the
       | applications just registered callbacks for msg type 'X' which
       | would present one with the caller information, the msg as a
       | buffer, and potentially a "user word" which would be a way to
       | keep server state around between calls.
       | 
       | The layering, from the main select loop thru different layers of
       | TCP connection, SSL handling (if used, not 100% was SSL in the
       | 90s), thru persistent link handling, application msg handling,
       | timers, memory allocation, etc. was done with such art that I
       | felt it to be a thing of beauty.
        
         | mrpippy wrote:
         | Fascinating, what kind of hardware/OS was used in production?
        
           | lanstin wrote:
           | Started with Tandem and Stratus pre Unix then migrated to
           | HPUX then Solaris then Linux on stock hardware. Also Digital
           | for big machines and early usage of 64 bit. I ported to Mac
           | OSX but only for local dev.
           | 
           | Funny thing about HPUX to Solaris migration is we found a ton
           | of null pointer dereferences which didn't cause SEGV on HPUX
           | but did on Solaris. And Linux migration was to a different
           | endianness so had to fix any copying to/from ints to a
           | buffer.
        
             | tibbydudeza wrote:
             | I worked at a bank where we used Tandem NonStop on Guardian
             | OS - loved the programming environment using TAL language
             | and the built-in SQL database.
        
         | patrick451 wrote:
         | Sadly, it seems like the moral of this account is that
         | engineering quality is orthogonal to business success.
        
           | winrid wrote:
           | I wouldn't say it was a failure. They had a good run in a
           | very quickly evolving time. That they got a slice of business
           | for years during that period isn't bad!
        
             | CPLX wrote:
             | When did they fail? They acquired Time Warner, creating a
             | company that after continued consolidation is now ranked
             | #13 on the Fortune 500 list.
        
           | foooorsyth wrote:
           | AOL was an incredibly successful business. Nothing lasts
           | forever.
        
             | __loam wrote:
             | The creative destruction of the business cycle grinds ever
             | onward.
        
           | huhtenberg wrote:
           | Orthogonal implies independence, not inverse correlation
           | (which is what I'm guessing you meant).
        
             | justin66 wrote:
             | I bet they meant what they wrote, since it makes more sense
             | than what you're suggesting they meant.
        
           | dmd wrote:
           | Er, what? AOL was one of the most successful companies...
           | ever. Things moved on, but this is like saying the roman
           | empire was never successful.
        
           | deanishe wrote:
           | AOL was extremely successful in its day.
           | 
           | They had 40 million customers paying $20/month 30 years ago.
        
           | mtoddsmith wrote:
           | AOL's primary business was dialup modems. Those died off
           | after internet became ubiquitous.
           | 
           | They pre-dated Yahoo and Google as an index of the internet.
        
         | vmsp wrote:
         | Is this it?
         | 
         | https://aolserver.github.io/
        
           | edgyquant wrote:
           | Can't be the same thing GP is talking about as the repo says
           | it's multithreaded and the poster said what they worked on
           | wasn't
        
             | mtoddsmith wrote:
             | Might have been updated later on.
        
           | mtoddsmith wrote:
           | Wild seeing "19 years ago" as the last commit time.
           | 
           | https://github.com/aolserver/aolserver
        
       | jsrcout wrote:
       | It's refreshing to read the replies here as I am currently
       | working in the worst codebase I've ever seen in my life. It's
       | just unimaginably bad.
        
         | mythrwy wrote:
         | You must be the guy who replaced me at my last job.
        
       | tonymet wrote:
       | Facebook but for different reasons than most. We called this
       | swapping out the airplane parts mid-flight
       | 
       | A lot of effort goes into language design and tooling to enable
       | continuous migration of code. Rather than re-writing entire
       | repos, existing code is continuously upgraded through semi- &
       | fully-automated code-mods. Every day thousands of patches are
       | landed to upgrade APIs with new type safety, security
       | restrictions, deprecations and other code maintenance.
       | 
       | Most other company repos I worked on had major re-writes outside
       | of the mainline until one day there was a sudden and often
       | troublesome switch-over to the new code.
       | 
       | Code is constantly changing, and unless you have a continuous
       | process for landing and testing those changes, you are going to
       | suffer when you try to apply the changes needed to address the
       | accumulated tech-debt.
        
         | lanstin wrote:
         | I really wish more people believed in continuous improvement
         | and the malleability of software. Giant heroic refactors or
         | redos are so annoying, but so are ten year old compiler or
         | kernels.
        
         | ceejayoz wrote:
         | > We called this swapping out the airplane parts mid-flight...
         | 
         | I work fairly heavily with Facebook's Graph API, and I think
         | they may have employed too many Boeing engineers for the part
         | swaps lately.
        
           | tonymet wrote:
           | Most of the compatibility issues over the past few years at
           | least have been due to added privacy restrictions and reduced
           | permissions scoping.
        
             | ceejayoz wrote:
             | The same few years have seen the developer bug tool closed
             | down in favor of the mostly abandoned "community forums" no
             | one uses.
             | 
             | We had big multi-year bugs in "assigned" state before its
             | closure, too.
        
               | tonymet wrote:
               | That's true too. The developer api is no longer a major
               | growth generator as it once was. What remains is the bare
               | minimum to keep partners happy, typically ads vendors
        
         | fragmede wrote:
         | > We called this swapping out the airplane parts mid-flight...
         | 
         | At ksplice we swapped the code out from under the kernel
         | without rebooting, so I know what you mean.
        
         | 01HNNWZ0MV43FF wrote:
         | For curious readers, the "building a plane as it flies"
         | metaphor dates back to at least 2000, to these commercials for
         | EDS: https://www.youtube.com/watch?v=L2zqTYgcpfg
         | 
         | I thought it was IBM but I was quite mistaken, so good thing I
         | looked it up lol
        
       | mrazomor wrote:
       | In my case, it was the integration testing framework built for a
       | large Python service.
       | 
       | This was ~10y ago, so my memory might not serve me well. A bit of
       | context:
       | 
       | - proprietary service, written in Python, maaany KLOC,
       | 
       | - hundreds of engineers worked on it,
       | 
       | - before this framework, writing the integration tests was
       | difficult -- you had a base framework, but the tests had no
       | structure, everyone rolled out their own complicated way of
       | wiring things -- very convoluted and flaky.
       | 
       | The new integration tests framework was build by a recently
       | joined senior engineer. TBF, it's wrong to say that it's was a
       | framework, if you think in the xUnit sense. This guy built a set
       | of business components that you could connect & combine in a
       | sound way to build your integration test. Doesn't sound like
       | much, but it significantly simplified writing integration tests
       | (it still had rough edges, but it was 10x improvement). It's rare
       | to see the chaos being tamed in such elegant way.
       | 
       | What this guy did:
       | 
       | - built on top of the existing integration tests framework
       | (didn't rollout something from zero),
       | 
       | - defined a _clear semantic_ for the test components,
       | 
       | - built the initial set of the test components,
       | 
       | - held a _strong ownership_ over the code -- through the code
       | review he ensured that the new components follow semantics, and
       | that each test component is covered by its own test (yep, tests
       | for the test doubles, you don 't see that very often).
       | 
       | Did it work well longterm? Unfortunately, no. He stayed
       | relatively short (<2y). His framework deteriorated under the new
       | ownership.
       | 
       | Travis, if you are reading this and you recognized yourself,
       | thank you for your work!
        
         | MathMonkeyMan wrote:
         | I wonder if this is a tragedy of "the org wasn't ready for his
         | solution" or more of a "if I were to go back now, I'd notice it
         | wasn't so good."
        
           | bornfreddy wrote:
           | Why only those two options? It could be just "the company had
           | other priorities and developers, while they appreciated
           | Travis's work, didn't find it worthwhile to carry the torch
           | themselves". Having a great testing framework is just one of
           | the things that need devs' attention.
        
             | MathMonkeyMan wrote:
             | Fair point, it's a false dichotomy; though your example is
             | close to "the org wasn't ready for it," which is itself
             | vague.
             | 
             | Maybe the issue is that it easily comes apart without a
             | dedicated censor, so to speak, and nobody wanted to have
             | that role.
        
           | f1shy wrote:
           | Without knowing anything, and before reading this comment, I
           | had the feeling of nr. 2. Strong feeling. I think the reason
           | is ,,many klocs of Python". I have developed alergy to big
           | Python programs. I like python for small things, probably
           | wrapping some C code.
        
             | bboygravity wrote:
             | Why would KLOC in Python be worse than in any other
             | language?
        
               | munch117 wrote:
               | Because you're doing more work per line.
               | 
               | A 20 KLOC Python program is not more complicated than the
               | 100 KLOC C program performing the same function. Quite
               | the contrary. But if you're comparing 100 KLOC of C to
               | 100 KLOC of Python, the Python program may seem unwieldy.
        
               | yosefk wrote:
               | So 500 KLOC of C doing the same thing would be better?..
        
               | munch117 wrote:
               | Not at all.
               | 
               | But when you come across 500 KLOC of C, then you say "wow
               | this is big" and you're forewarned that this is going to
               | be unwieldy. You may underestimate the 100 KLOC of
               | Python. That's all I'm saying.
        
               | mrazomor wrote:
               | It's about the different ways the language allows you to
               | shoot yourself in the foot.
               | 
               | I worked on large Python, C++, Java & Go services. I have
               | 10y+ of experience with the first 3. C++ allows you to
               | write incomprehensible code (even to the experienced C++
               | devs) and justify its existence (because of the
               | performance gains). But you need to be a top expert to
               | write a compileable code of that type. I'm comfortable
               | with diving in any C++ codebase except for the libraries
               | like std, boost, abseil, folly, etc. Most of the code
               | there is absurdly difficult to comprehend.
               | 
               | On the other hand Python leads in the ways a junior dev
               | can introduce hell in the code. Especially if the team
               | doesn't rely on the strict type check. I have seen
               | horrors.
               | 
               | I was bewildered when I realized that working with
               | JavaScript with type checks (Closure compiler) was
               | insanely more productive and smooth than working with
               | Python (before the type checks).
               | 
               | That's why Java won the enterprise world. It takes an
               | effort to make a mess in Java (but people still manage).
               | Go is in a similar place.
        
         | PartiallyTyped wrote:
         | > Did it work well longterm? Unfortunately, no. He stayed
         | relatively short (<2y). His framework deteriorated under the
         | new ownership.
         | 
         | I think the issue is that integ tests are not really a place
         | that sees "development". You write the integ tests, and move
         | on, you are not actively introducing more integ tests.
         | 
         | I think that's a shame that this is the common view of the
         | people that need to fund this. Bad integ tests mean bad dev
         | exp, which then results in increased attrition and
         | dissatisfaction.
        
       | cwbriscoe wrote:
       | One that I built and maintained 100% on my own. Nothing like
       | doing things the way you like to without someone else messing
       | everything up.
        
       | antishatter wrote:
       | Had to work through a weirdness in kubernetes many years ago when
       | using it on prem for a project (not at google). It stands out as
       | the best code base I have had to go through to figure stuff out.
        
       | cvwright wrote:
       | libjpeg 6 is pretty good. It's oldschool C so it's pretty
       | spartan, but it's easy enough to read it and follow what it's
       | doing if you have a basic understanding of the concepts.
       | (Discrete cosine transform, Huffman coding, etc)
       | 
       | And it's only had a couple of vulnerabilities, in nearly 30 years
       | of being on the Internet. That's not quite like DJB code, but
       | it's darn close.
       | 
       | My understanding is that it was written by a very small number of
       | people who were experts in the field, as part of the standards
       | process way back in the early 90s.
        
       | brynet wrote:
       | OpenBSD. Good people, consistent style, correctness.
        
       | JensRantil wrote:
       | The first few years at a previous startup I worked on. Java.
       | CI/CD in place when merging PRs. But mostly, the thing that made
       | the code base great was the fact that the code wrapped external
       | libraries in small interfaces that were _owned by the company_,
       | and dependency was used to not depend on implementation
       | interfaces. In short, adhering to the basics on the SOLID
       | principles.
        
         | meowtimemania wrote:
         | Kinda sounds like abstraction hell. What did you gain by
         | wrapping all external libraries in interfaces?
        
           | oasisaimlessly wrote:
           | In my experience, any third-party dependency will inevitably
           | eventually be in-sourced. Third-party dependencies are
           | technical debt - they let you make fast progress initially,
           | but have recurring maintenance costs.
           | 
           | Wrapping the API lets you see, all in one place, the surface
           | area of an external API that is in use, and minimizes the
           | changes required here when reimplementing it.
           | 
           | ---
           | 
           | Also, not that Clean Code is any great authority (but it is
           | popular!):
           | 
           | > [Wrapping] third-party APIs is a best practice. When you
           | wrap a third-party API, you minimize your dependencies upon
           | it: You can choose to move to a different library in the
           | future without much penalty. Wrapping also makes it easier to
           | mock out third-party calls when you are testing your own
           | code.
           | 
           | > One final advantage of wrapping is that you aren't tied to
           | a particular vendor's API design choices. You can define an
           | API that you feel comfortable with.
        
       | kristopolous wrote:
       | NetBSD
        
       | OnlyMortal wrote:
       | Used: the Boost C++ libraries. In particular ASIO. Variant has
       | also been very useful.
       | 
       | The Mac GUSI BSD socket library is worth a mention too. I built a
       | multithreaded Quark XPress XTension with it that used PowerPlant
       | for UI. Quark even wanted to interview me.
       | 
       | Added to: a pre-Boost cross platform library for Mac (Classic)
       | and Windows NT. It did much of what Boost did way back when.
        
       | elwell wrote:
       | ClojureScript + Re-frame + GraphQL subscriptions over WebSockets:
       | https://github.com/Vetd-Inc/vetd-app
        
       | fooker wrote:
       | The LLVM infrastructure.
       | 
       | Other compilers don't even come close.
        
       | sunir wrote:
       | I worked on an incredible C++ codebase back in the dot.com. We
       | built a mobile SVG renderer and packed into 128kb on a Palm 3 and
       | similar code sizes on Blackberries, Symbian (the worst OS ever),
       | Fujitsu phones, and whatever you threw at us.
       | 
       | I built a few systems. The DOM, a build system that could cross
       | compile to a completely new target phone while the phone
       | executives were at lunch with our sales team (that's great
       | demo!), and an optimizing build step in Perl that rewrote the C++
       | to minimize bytes, cycles and joules of energy.
       | 
       | The rendering engine was built by a genius. It was the most
       | efficient and accurate vector graphics renderer I have ever seen.
       | 
       | But of course no one could understand how this system worked.
       | 
       | I mean, Perl that transpiles C++ into more optimized C++ just is
       | abusive. I was not popular with the senior devs.
       | 
       | The main coders including me were coop students who worked
       | furiously at all hours for the fun of it.
       | 
       | But of course the product had no revenue. The wiser devs probably
       | knew this and didn't care.
       | 
       | Nevertheless, we eventually were let go during the dot.com bust
       | and the remaining main engineers had to throw the code out and
       | try to rewrite it in Java. That didn't work because in that era
       | Java was too slow to fit on a phone. Or maybe the market was
       | dead.
       | 
       | I learnt a solid lesson.
       | 
       | What's amazing technically is not an amazing code base.
       | 
       | What is an amazing code base isn't an amazing product.
       | 
       | What is an amazing product isn't an amazing business.
       | 
       | Literally every product I have ever seen that is profitable is
       | not a good code base. But if it pays me to code, I love it.
        
       | jeffrallen wrote:
       | The Go standard library. The level of detail, the politeness of
       | the reviews, the fact that the features of the language allowed
       | me to put more attention into correct code than "at least it's
       | working" code, the pragmatism and personalism of the core team.
        
       | malkosta wrote:
       | The best codebases I worked on were from startups that failed due
       | to lack of product market fit. Oh man, their code was so
       | beautiful and optimized...
        
         | BobbyJo wrote:
         | Is this true or a bit of humor? Of the three startups I worked
         | for, the one that had the best exit also had the best code. Ime
         | shitty code hamstrings startups into slow execution during the
         | growth phase, and kills them despite pmf, because competition
         | winds up moving much faster.
        
       | yobert wrote:
       | I tried to add a new SQL function to CockroachDB. I had never
       | seen the source before, but since it was such nice Go code, it
       | only took me a few hours to have a working build, with tests.
        
       | jchrisa wrote:
       | CouchDB's HTTP API used to fit in just one file (less than 1k
       | lines of Erlang), and it was a great jumping-off point for
       | learning databases and web programming:
       | https://github.com/apache/couchdb/blob/0.8.1/src/couchdb/cou...
       | 
       | It was written by one engineer, and then later refactored by a
       | team to look like this (and many other files):
       | https://github.com/apache/couchdb/blob/main/src/chttpd/src/c...
       | 
       | It's an interesting exercise to see how something grows and
       | changes as it transitions from inspiration to real-world usage.
        
         | phito wrote:
         | So little blank lines, I need to breathe
        
       | CSMastermind wrote:
       | I've seen enough code bases that I feel like I have a good
       | spectrum from truly exceptional to horrific.
       | 
       | The best code bases had these things in common:
       | 
       | 1. _Consistency_ : they had clear patterns that were followed. It
       | doesn't matter what the pattern was; some were strictly object-
       | oriented, some were purely functional, some used RESTful API, and
       | others leveraged gRPC or GraphQL. The important thing was that it
       | was all consistent across the codebase so even if you were
       | looking at a part you'd never seen before you could still orient
       | yourself and reason about it quickly.
       | 
       | 2. _Ownership_ : there was always a single individual person who
       | was ultimately responsible for each section of code. That doesn't
       | mean they wrote all the code themselves; they were the final
       | arbiter on the rare occasion a truly contentious conflict arose.
       | They also had the unilateral power to make changes within their
       | section of the code if they felt it was needed. This was always a
       | rarely exercised power but they could, if they had to, push a
       | change through. There could be many such people spread out across
       | the codebase but for each discrete part the number was always
       | one.
       | 
       | 3. _Clear Boundaries_ : it was clear what each part of the
       | codebase's purpose was, and the boundaries were rigidly enforced.
       | Things were never tightly coupled across these boundaries.
       | Business logic was always isolated from things like
       | serialization/deserialization, each system was forced to maintain
       | its own models of the world that it cared about.
       | 
       | The worst code bases had these things in common:
       | 
       | 1. _Lack of Eng Representation_ : Product controlling development
       | schedules and insisting everything is high priority and needs
       | done right now! Project managers who always wonder, "yes but what
       | if 9 women could make a baby in a month. Why don't we try adding
       | more resources? Can we just try?" Business types who see software
       | as a cost center not a profit driver. This can also happen if
       | your engineer leadership didn't come up through software
       | engineering but rather QA or IT or started out in academia or is
       | just plain and MBA type with no eng background.
       | 
       | 2. _" Time saving" Tech_: We don't need a database schema, we can
       | just go NoSQL and have JSON blobs, it will save so much time! We
       | can share models between the front end and back end, it will save
       | so much time! Don't write SQL, use this ORM, it will save so much
       | time! Don't think about DevOps, just use this all in one hosting
       | solution, it will save so much time! What about this low code/no
       | code solution? It will save so much time!
       | 
       | 3. _Misaligned Incentives_ : This could be because they were
       | contractors with no stake in the company or this could be because
       | it was a large company and there was no realistic way they'd ever
       | be fired. Either way there were no consequences for writing bad
       | code.
        
       | jokoon wrote:
       | * medical imaging software to drive a X ray machine
       | 
       | * software was relatively new, and well made. age is probably the
       | main reason why it was good "at the time".
       | 
       | * there were good practices in place, yes, led by an opinionated
       | team lead, which felt like it was the main cause for such a
       | project to work well under medical standards.
       | 
       | I feel like I will never find another job like this one again.
        
       | cbanek wrote:
       | The Xbox codebase was pretty good. We were doing enforced code
       | reviews back in 2011-ish, which was a big thing at the time.
       | There was also a good layer cake going on and a few good
       | architects.
       | 
       | The worst codebase by far was Outlook for Mac. It had code going
       | back to Entourage, and was never properly cleaned up.
        
         | ruff wrote:
         | Ha, if you think that was bad, you should have been there when
         | it was just Entourage ;).
        
         | christophilus wrote:
         | Office in general is a mess. I think it's a good example of the
         | fact that the sales team / marketing power of a business is the
         | real driver of success, much as that pains me as a developer.
        
           | cbanek wrote:
           | I feel that Office's trouble is always backward
           | compatibility. Which makes sense.
        
       | raggi wrote:
       | Code wise, probably Fuchsia. I worked up and down the Fuchsia
       | stack regularly, from bootloaders to GUI programs and all over
       | in-between. For a sizeable code base with so many layers and
       | concepts, and regularly moving between these without much memory
       | for what's there (as it also moved around quite quickly in my
       | time), it was the lowest average cost of entry to new areas of
       | most other millions of lines of code repositories I've dealt with
       | in my career. Still plenty of bugs came up, lots of polish and
       | more design work to do, it's not at all to over-sell it, but
       | almost everything was approachable with low impediment.
       | 
       | There were also lots of Ruby libraries back in the hayday of the
       | language that were very clean for what they are. I've rarely seen
       | programs structured so well since - some of this is essentially a
       | forced practice by the language - if you don't do it, you'll
       | suffer as there's no types, lsps and whatnot to otherwise save
       | you, but still, it's worth calling out, as I think back
       | occasionally with fond memories of being unimpeded anywhere I
       | needed to go.
       | 
       | Tooling wise it's got to be noted as others have that the Google
       | monorepo was well integrated, but it wasn't what I'd describe as
       | "best" at all. I had a fairly broad time at Google having worked
       | in most of the sub orgs over the years and in doing so saw a lot
       | of the variance even with the chosen integrations. I saw teams
       | who regularly built from no where near head cherry picks out of
       | the repo, I ended up owning stuff that always built off of the
       | main branches (most Googlers don't even know exist) setup before
       | my time to avoid running into otherwise hardline constraints
       | around single version policy that couldn't be adhered to for
       | specific product reasons (hi cloud). Exactly how much and which
       | parts of buganizer teams used varied hugely, which service
       | management tools they used varied, which database systems, etc.
       | The single patch system while also fairly well integrated
       | combined with code review practices that often stymied medium
       | scale refactors (large refactors within a single product area),
       | though making large scale changes that touch every single top
       | level directory, while a bit rough on the edges of the tooling,
       | the tooling existed and worked really well - pushes out hundreds
       | or even thousands of patches and automates pinging owners and
       | informing you of failures, etc. In many ways a lot of smaller
       | systems with far less tooling, and far far less integration were
       | more productive - I've never seen such a bad way of _organizing
       | work to be done_, or such a poor volume management of bugs as I
       | have at Google in various teams/areas, despite all the tooling.
       | Any delivery that involved client programs (desktop programs,
       | mobile programs, etc) was always pretty remarkably awful, and
       | only very very small teams were on the hook for making any of
       | this better. They'd always plaster their new thing with "don't
       | you dare make another or use the deprecated thing" meanwhile they
       | never had enough time to make the new thing ready - a common
       | trope in the company, but really highlights there are edges
       | outside of the "pure saas" side of the business which were pretty
       | bad. The best part of the monorepo and monotooling culture for
       | the saas side is the security properties of it all, which were
       | outstanding, and are among the reasons why I still trust Google
       | as the holder of some main accounts. I hope they never start to
       | lose those properties given the scale and impact they could have.
        
       | didip wrote:
       | Many of the large Go codebases. Including Kubernetes.
       | 
       | It is super easy to find what I am looking for.
        
       | rrdharan wrote:
       | I work at Google on database systems and would still pick the
       | VMware codebase (specifically the virtual machine monitor
       | component).
       | 
       | It was by far the most impressive piece of software engineering
       | I've ever had privilege of perusing.
        
       | bitwize wrote:
       | I haven't contributed to it yet, but one of my personal projects
       | involves hacking NetBSD. Absolutely, legendarily good code base,
       | so thoroughly documented you can actually begin kernel hacking
       | just by _reading the man pages_. The code itself is really
       | consistent and easy to read; it 's pretty clear the NetBSD team
       | enforces a consistent style as well as high standards for
       | thorough documentation. NetBSD is how C code _should_ be done.
       | 
       | The other code base I really liked was QRes, ITA Software's
       | airline reservation system written in Common Lisp. I contracted
       | at Google a few years ago doing maintenance on this code base
       | before it was shuttered for good. Aside from being just high-
       | quality Lisp code, it had its own testing language that allowed
       | new tests to be written in just a few lines or so -- tests that
       | sent XML queries with generated data to the running system and
       | checked for specific results in specific XPath paths. Because the
       | system was Lisp, implementing and extending this testing language
       | was relatively easy. Using it even more so. Truly the only system
       | I was happy to write tests for, rather than seeing it as a
       | necessary chore.
        
       | fnordpiglet wrote:
       | SecDB at Goldman. There were runtime issues aplenty but the SDLC
       | was top notch and I've never heard of or seen better - and it's
       | 30 years old. Instant deployments and rollbacks globally, you
       | show up to work and commit to prod in the first half day. Within
       | a week you can go on rota. It's why Goldman was so nimble in the
       | 90's, and 00's. It's still a remarkable system but
       | underinvestment has taken its toll after margins evaporated in
       | 2010.
        
       | arriu wrote:
       | Blender is incredible, harfbuzz equally so, pretty much all
       | modern browsers do the impossible so they need to be included as
       | well.
        
       | bogwog wrote:
       | Open 3D Engine's codebase is pretty great: https://o3de.org/
        
       | zombiwoof wrote:
       | Java SDK 1.0
        
       | codezero wrote:
       | I'm not a software engineer, but the best codebase I worked on
       | was at Quora. I attribute this to them having several years of
       | time to build a base that was solid. That said, I found plenty of
       | bugs, but it was just so nice to work with. Almost everything was
       | Python. DOM, CSS, JS embedding, etc... keeping in mind this was
       | way before Angular/React, but the system worked very similar,
       | doing DOM diffing via a C++ service on the backend and pushing
       | updates out over a long polling JS endpoint.
       | 
       | On top of that the data which was read heavy went through several
       | layers of cache that were very easy to understand (process,
       | system, global - called L1, L2, L3 like a system as a CPU). The
       | front-end had a "Konami code" that let you see the rendering
       | hierarchy from the backend while viewing it from the frontend
       | that let you also understand the caching state of each level of
       | hierarchy that was rendered as well as all the CSS/JS handlers
       | affecting the element, not unlike current "inspect" view of most
       | browsers.
       | 
       | What made this all so good? A very strong foundation that was
       | opinionated, but which was consistent. This was ~ 2012-2014 -
       | they had Flask as an underlying framework, and much was built out
       | on top of Python decorators.
       | 
       | Besides the day-to-day code - they had a bespoke packaging
       | system, auto-scaler, and a number of other services that "just
       | worked" - it was very different from any other startup I've
       | worked for.
        
       | ryukoposting wrote:
       | This is going to sound insane, but I worked on a firmware
       | codebase at an industrial automation company that was remarkable.
       | 
       | Everything was working against it. No RTOS. Subversion. GCC
       | 5-point-something (I think?).
       | 
       | It was an incredible mass of preprocessor crimes. I'm talking
       | about #including the same file 10 times and redefining some
       | macros each time to make the file do something different.
       | 
       | It used a stackless coroutine library called Protothread, which
       | itself is a preprocessor felony.
       | 
       | And yet? It was brilliant. Compilation was lightning quick. F5,
       | lean back in your chair, and boom, you're running your code. I
       | understand that this kind of thing is normal for web/backend/etc
       | folks, but I yearn for the days of sub-15 second firmware compile
       | times.
       | 
       | It was easy to flip a couple flags and compile it to a
       | Win32-based simulator. Preprocessor felonies are felonies, but
       | when you stick to a small handful of agreed-upon felonies, you
       | can actually reap the benefits of some very sophisticated
       | metaprogramming, while staying close enough to the hardware that
       | it's easy to look at a datasheet and understand exactly what your
       | code is doing.
        
         | jhayward wrote:
         | > Preprocessor felonies are felonies, but when you stick to a
         | small handful of agreed-upon felonies, you can actually reap
         | the benefits
         | 
         | There's a saying, "Only commit one crime at a time". Knowing
         | when to cheat is a fairly high-risk but pro move.
        
       | agentultra wrote:
       | I never got to submitting my own patches but the Linux kernel is
       | nice.
       | 
       | Where I work now I get to work in Haskell. And that's been a
       | pleasure. Recently rewrote over 1k lines. Deployed to production
       | with one reviewer. No issues.
       | 
       | Just good test coverage and a great type system with no-nonsense
       | code. Refactoring in Haskell is a heady drug.
       | 
       | If this was C++ or even Python there would have been much more
       | intense reviews and probably some fail-forward patches to fix
       | things we may have missed.
       | 
       | But I dunno. Maybe not. Maybe I'm just that good. /s
        
       | sgtnoodle wrote:
       | A former coworker texted me out of the blue yesterday, saying he
       | missed working on a "<my name> codebase". He specifically
       | appreciated that "doing something simple things were kept
       | simple." Made my day!
        
       | Joel_Mckay wrote:
       | It was a 63kB assembly project that compiled down to 384 bytes.
       | However, that project handled phase-locked control of a power
       | dimmer, logged up to 2 days of power-factor stats information
       | into an eeprom, integrated clock timers, and several user inputs.
       | 
       | It was the smallest, and most bizarre interleaved task-structure
       | I'd seen fit into a $0.17 micro-controller. I kind of admired
       | that the code itself formed the delays for other areas of the
       | multiple tasks, and essentially achieved near perfect resource
       | utilization for actual work.
       | 
       | Never saw anything that efficient again for several decades. =3
        
       | theusus wrote:
       | Worked on a domain driven design code. It was simple and
       | intelligible. Now, I want to copy that methodology everywhere.
        
       | Aeolun wrote:
       | I'm firmly convinced that any codebase written by any 4 people in
       | the same room is better than any written by any higher number or
       | distributed.
        
         | tinco wrote:
         | You'll be surprised, I think the best codebase I ever
         | contributed to is the Rust compiler, which is being built by a
         | huge distributed team.
         | 
         | Despite it being complex and me being in there for the first
         | time, I could quickly track down a bug in the parser and fix
         | it. As a cherry on top the communication around the pull
         | request was top notch too.
        
           | Aeolun wrote:
           | I'm not trying to say it can't be good despite that factor,
           | but if the rust compiler were written by the top 4
           | contributors sitting in the same room, it'd be even better
           | (and possibly not finished yet ;) )
        
         | whoknowsidont wrote:
         | You could at least contribute to the thread instead of
         | grandstanding about an opinion you have.
         | 
         | Name the codebases.
        
         | f1shy wrote:
         | I've seen plenty of examples of exactly the opposite.
        
         | bboygravity wrote:
         | So NVIDIA must write terrible code?
        
           | Aeolun wrote:
           | That would not surprise me. There's a reason they need all
           | that hardware.
        
         | sensanaty wrote:
         | It took my company a team of 8 fully-remote devs to fix the
         | clusterfuck of a codebase that the "4 guys in a room" (the CTO
         | and early devs) built up with spit and tree bark,.
        
         | JSDevOps wrote:
         | The entire world disagrees, Windows, The Linux Kernel, Anything
         | at Netflix, Google, Facebook etc. Try getting them all in a
         | room.
        
       | unkulunkulu wrote:
       | The one I'm working at right now!
       | 
       | What made it great: the company is 7 years old, I work basically
       | since inception, half year late.
       | 
       | The codebase had some serious problems due to migrating build
       | systems, rapid team growth and lack of proper communication while
       | developing important infrastructure tools, lack of foresight at
       | the start of a crucial requirement for the whole system that
       | became clear 3 years in.
       | 
       | Aaand... Due to unique sequence of events half a year ago we had
       | to start everything from scratch! We could not use a line of code
       | from the old repo !
       | 
       | This brought miracles out! We started designing the right things
       | right. People started talking to each other on the global scale,
       | without the code limiting them!
       | 
       | The old code not only had problems on the engineering level, but
       | it held people and team responsibilities along suboptimal
       | interface boundaries! This is the biggest magic! After the
       | rewrite started, all the people in the team landed exactly where
       | they wanted and their areas of responsibility became clear and
       | highly efficient. Some people started shining like I could not
       | imagine. Because their organizational and communication skills
       | were limited by a huge suboptimal codebase.
       | 
       | Also, it is very empowering to design a system when your whole
       | team has real experience 5 years into the future. Also, everyone
       | turned out to be caring about code simplicity and quality, I
       | finally feel at home.
       | 
       | I could not wish for a better developer experience. The sexiest
       | C++ coding time in my life :)
        
         | pcatach wrote:
         | Interesting! Seems like you were uniquely well-positioned to
         | tackle the infamous system rewrite in a way that worked out
         | great.
         | 
         | Fixing the interface boundaries is a strong case for it despite
         | all of the downsides (mostly opportunity cost, I'm thinking).
        
       | ephaeton wrote:
       | NetBSD.
       | 
       | The docco, the culture, the clarity and simplicity of design from
       | first principles. The coherency of code across kernel, user-
       | space, accompagnying material. The vibe.
       | 
       | You may know NetBSD (if, at all) as the BSD that wants to be
       | "portable" and may dismiss it against FreeBSD's focus-claim of
       | "performance" and OpenBSD's focus-claim of "security".
       | Interestingly, trying to remain wildly portable requires a
       | cleanliness inside that is very soothing.
        
         | ajay_san wrote:
         | Question, I want to get into more lower level programs like
         | compositors or docker daemon or kernels (I do realize they all
         | are different things) but I find tech behind them so
         | intimidating and don't know where to get started. Do you have
         | any advice where I can learn them? I'm about to grad next year
        
           | spai2 wrote:
           | https://stevens.netmeister.org/631/
        
       | v3ss0n wrote:
       | Litertar.dev and Tornado web . Litestar code base, architecture
       | and core design is what modern python code should be. Very clean,
       | fully type annotated, DTO, and many cool stuff happening there.
        
       | m00x wrote:
       | redis's C code is incredible
        
       | purple-leafy wrote:
       | My own, lol. Which just means I need to work on more codebases.
       | 
       | I'm making a tool to convert data schemas to SQL via a UI for
       | lay-users. Just like https://react-querybuilder.js.org/ which is
       | basically a UI based SQL generator. For work.
       | 
       | Except my version extends the idea much further, blending Excel
       | like functionality with functions that can act on Fields and
       | Rules.
       | 
       | What makes it good?
       | 
       | For one, it's a from scratch project with almost no third party
       | libraries.
       | 
       | For two, I fortunately chose a recursive data structure to
       | represent the data schema, and that has really worked out well.
       | Early on I tried 4 other approaches to represent the data, but
       | went back to recursive feeling it was the best choice.
       | 
       | Furthermore I'm using React, but it's heavily leveraging
       | reducers. Specifically Groups have a reduced, Rules have a
       | reducer, and Fields have a reducer. The reducers are linked in a
       | chain top to bottom, where changes on a Field change a Rule, and
       | changes on a Rule change a Group. It's been extremely clean to
       | work with.
       | 
       | Because the base data schema is recursive (Groups contain groups
       | contain groups), most of the functions that manipulate the schema
       | are recursive. There is a real elegance to the code, each
       | recursive function has a very obvious base case, and a very
       | obvious recursive path.
       | 
       | And for the final outcome, walking the query data structure and
       | spitting out the equivalent SQL is also recursive, and feels
       | elegant, coming in at under 40 lines.
       | 
       | Literally as I've been writing this codebase, everything somehow
       | perfectly fell into place. I was marvelling near the end that it
       | felt like I chose all the best possible logic paths to build this
       | somehow.
       | 
       | I'm hoping to get the okay from work to open source it (fully
       | open source). The only cruft of the project is the types I'm
       | using, the interface of the code could be improved with generics
        
       | w10-1 wrote:
       | Java, because: Mark Reinhold
       | 
       | He learned from the best.
        
       | lolive wrote:
       | Not wanting to start a flame war, but the appearance of jquery
       | has been an eye-opener for me. The official API of the Web was
       | instantly outdated thanks to the incredible cleverness of a bunch
       | of developpers. Wow ... that was, for me, exactly what open
       | source was all about.
        
       | somenameforme wrote:
       | Unreal Engine, by far.
       | 
       | It's not only just extremely well written in general (with your
       | only chance of 'really' learning the engine to go through it,
       | read comments, and the like) but it also defies the wisdom
       | 'everybody knows.' That wisdom being that premature optimization
       | is the root of all evil - like we're supposed to just benchmark
       | things, see where the bottleneck is and then try to work on
       | optimizing spots like that.
       | 
       | Unreal doesn't do this. There are countless thousands of micro-
       | optimizations everywhere. For one striking example there's even a
       | fairly substantial system in place that cache the conversion
       | between a quaternion and a euler rotation. This would never, in a
       | zillion years, be even close to a bottleneck. But with thousands
       | of these little micro-optimizations everywhere you get a final
       | system that just runs dramatically better than any comparable
       | engine.
       | 
       | In more general terms, they've also taken advantage of the
       | 'idiomatic flexibility' that C++ offers to create a sort of
       | Unreal C++ language that is also just lovely to use and feels
       | much closer to something like C# in terms of luxuries like the
       | lack of manual memory management, garbage collection, reflection
       | (!!), and so on. The downsides are that compile times are
       | horrible (even though a cached compile might only take 10 or 15
       | seconds, it feels like _forever_ when trying to work out one
       | specific issue) and C++ intellisense, especially in a
       | preprocessor heavy environment, is pretty mehhhh.
        
         | 01HNNWZ0MV43FF wrote:
         | > there's even a fairly substantial system in place that cache
         | the conversion between a quaternion and a euler rotation
         | 
         | I hope that does show up on a benchmark though, cause that
         | sounds complex
        
       | ecuaflo wrote:
       | The one I made from scratch after iterating over 3 years. You can
       | see the technologies here [0]
       | 
       | [0] https://gitsell.dev/u/bitofbreeze/r/bitofbreeze/git-sell
        
       | hightechnical wrote:
       | For me best code base I've read so far is Redis code. Feels like
       | a poem to me.
        
       | liampulles wrote:
       | The best codebases have been my own, because I can monopolize the
       | style and design to my own exacting vision of how things should
       | be.
       | 
       | The best work codebases are ones where I may disagree with a lot
       | of the style, but the lack of enforcement at the level of classes
       | and functions gives us creative freedom, builds mutual respect,
       | and enables ego-less, pragmatic, tradeoff-driven discussions
       | about things which are actually important. If these discussions
       | end with a majority but not universal agreement, then its fine,
       | the minority is not offended, and we happily continue with our
       | work.
       | 
       | That might also be because we're all pragmatists and somewhat
       | cynical of shiny things, or the latest fad.
        
       | jhon-crypt wrote:
       | The best codebase I've ever worked on was when I started software
       | development officially, I had zero knowledge of the stack for
       | real, the codebase was run by just 5 mid level engineers, but
       | based on how well structured and understandable it was, i was
       | able to pick up the stack without reading a tutorial or a
       | documentation, i would later read up on it when i started doing
       | complicating stuff tho, but it's important for engineers to write
       | good code to the extent that another person reading it does not
       | have to look up something just to understand why this does that,
       | and how stuff works in the code
        
       ___________________________________________________________________
       (page generated 2024-06-29 23:02 UTC)