[HN Gopher] "We ran out of columns"
___________________________________________________________________
"We ran out of columns"
Author : poidos
Score : 777 points
Date : 2024-08-03 12:26 UTC (10 hours ago)
(HTM) web link (jimmyhmiller.github.io)
(TXT) w3m dump (jimmyhmiller.github.io)
| Waterluvian wrote:
| The first line really hits me hard. There's something so
| incredibly freeing about being a kid and doing stuff like coding.
| There's simply no expectations. Even the smallest project felt
| like such an achievement. But now I code professionally and I
| don't know how to turn off engineering brain. I don't know how to
| be okay doing something poorly, but on my terms.
| actionfromafar wrote:
| What ever small success I had outside work has been whenever I
| managed to turn off Enterprise brain.
| captn3m0 wrote:
| Writers often say that your first N books will be crap, and the
| answer to that problem is to just write more than N books. I
| feel that's true of software as well - the first years of the
| magical programming I did without any expectations meant those
| projects were utter crap yet functional.
| datavirtue wrote:
| It's because you don't get to write apps from scratch enough.
| Everything is a POC. You get it working. Optimization is just
| dumb until the project is so big it doesn't fit in the
| developers brain anymore. Then you write tests for the core
| part so you can keep interating fast. You will see parts that
| clearly need refactorwd before you can move on.
|
| Also, be aware of, but don't get hung up on, engineering best
| practices. Chances are that someone is going to see your POC
| and put it in front of a client or promise it to someone. It
| will enter service being less than perfect, if you are lucky.
|
| When you pull this off people will tell stories about you and
| brag about you when you are not around. None of them will know
| anything about the code engineering.
| zbentley wrote:
| > When you pull this off people will tell stories about you
| and brag about you when you are not around.
|
| True!
|
| > None of them will know anything about the code engineering.
|
| Not true. In environments where it's very easy to fuck up
| (hyperconnected codebases, lots of legacy tech debt, fragile
| systems generally), programmers who create tools/libraries
| that let other engineers get work done without blowing their
| feet off or spending huge amounts of time on silly things do
| get praised. Maybe the praise comes from different people
| than in your example, but it definitely is rewarded.
|
| Two caveats: this only happens in certain environments; some
| gigs are just culturally unable to care about quality tools
| (death march feature factories and the like). Also, you
| _actually_ have to save time and reduce defect rates with
| your engineering work; starting an architecture-astronaut
| rewrite that isn 't actually useful for other programmers'
| day-to-day work doesn't count. Making "one tool to rule them
| all" that requires a Ph. D or pairing session with the author
| doesn't count either.
| neilv wrote:
| I think you can add a mode to engineer brain, in which it's
| aware when it's kludging something, but kludging is the
| appropriate thing to do.
|
| Might help to have enough experience that you have (well-
| placed) confidence in your gut feel for how much engineering a
| particular thing needs.
|
| (More common is to not have an engineer mode, or to not have
| enough experience to do it well.)
|
| If you don't have the kludge mode, try some things where
| kludges are outright necessary. One time I recall this was when
| writing Emacs extensions. For example, it used to be that you'd
| hit a point where normal programmatic navigation/parsing of the
| buffer was just too slow, and a well-placed regexp operation
| that worked 99.99% of the time (and the other 0.01% of the time
| it's OK enough, and the user understands) made the rest of the
| code viable.
|
| Another example is personal open source projects, when you
| really want to experiment with an usual implementation
| approach, and you give yourself permission.
|
| (Maybe don't do this when the open source is done only for
| resume purposes, where you're trying to demonstrate you follow
| all the current fashionable conventions. You're almost
| guaranteed that someday a new-grad on a public forum or not-
| very-technical hiring manager will stumble across that bit of
| code, and fixate on the one thing they think they recognize as
| a bad practice. Documenting the kludge and rationale, in, say,
| a code comment, is great practice, but, again, it will also
| draw the attention of the least-skilled. Much like, if you're
| relating an anecdote in a job interview, and implied are 20
| things you did right and 1 thing you did brilliantly, and you
| note as an aside one mistake you made, most of that will whoosh
| over the head of the dimmest FAANG interviewer, but they'll
| latch onto that mistake you spelled out, as a wise insight they
| have, and that's what's going in their report. :)
|
| Then, armed with an experienced multi-skilled brain, when your
| startup's imminent MVP launch is facing crazy constraints, you
| triage what bits need to be rock-solid so they'll absolutely
| work and not fail, what bits need creative kludging so you can
| hit your launch window for other key requirements, and what
| bits you need to mitigate any compromises. Ideally, you have
| the wisdom to know which is which, and can activate different
| skills for each kind of work.
|
| That Turbo Encabulator needs to be well-machined, but the new
| sensor you just decided it needs for a test doesn't need a week
| to be drilled into the engine block, nor a mount to be designed
| and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or
| maybe wad of chewing gum will do.
| Waterluvian wrote:
| All such great advice. The one time I've found success is
| when I figure out the entire scope of the little project up
| front. Then I find myself not even worrying about maintenance
| or scalability. I know I won't need that extra class
| abstraction or modularity or whatever. I just don't like
| planning at the start, but it's probably, counterintuitively,
| part of the answer.
|
| P.S. you opened a parenthesis and forgot to close it so now
| everything I read all day is part of your anecdote)
| jimmyhmiller wrote:
| This is something I've had to train myself to overcome. The
| first step for me was having a place where I clearly signaled
| that this wasn't my best work. Where the rules were allowed to
| be broken. That place is my junk drawer of code[1].
|
| I have a zsh alias `alias changes='git add . && git commit -am
| "Changes" && git push'` that I use for my commits in the repo.
|
| This all may feel silly, but it's what I needed to get back to
| that time of playing. Where I never worried about the code. But
| where I also didn't feel I was wasting my time working on code
| no one could see. I'd definitely recommend trying something
| like that if you are struggling with it.
|
| [1]: https://github.com/jimmyhmiller/PlayGround
| PreInternet01 wrote:
| > I miss that direct connection. The fast feedback. The lack of
| making grand plans.
|
| There's no date on this article, but it feels "prior to the
| MongoDB-is-webscale memes" and thus slightly outdated?
|
| But, hey, I get where they're coming from. Personally, I used to
| be very much schema-first, make sure the data makes sense before
| even _thinking_ about coding. Carefully deciding whether to use
| an INT data type where a BYTE would do.
|
| Then, it turned out that large swathes of my beautiful, _perfect_
| schemas remained unoccupied, while some clusters were heavily
| abused to store completely unrelated stuff.
|
| These days, my go-to solution is SQLite with two fields (well,
| three, if you count the implicit ROWID, which is invaluable for
| paging!): ID and Data, the latter being a JSONB blob.
|
| Then, some indexes specified by `json_extract` expressions, some
| clever NULL coalescing in the consuming code, resulting in a
| generally-better experience than before...
| jrochkind1 wrote:
| Oh good question on date essay was written -- put dates on your
| things on the internet people!
|
| Internet Archive has a crawl from today but no earlier; which
| doesn't mean it can't be earlier of course. My guess is it was
| written recently though.
| lexicality wrote:
| created 14 hours ago https://github.com/jimmyhmiller/jimmyhmi
| ller.github.io/commi...
| __MatrixMan__ wrote:
| But clearly in retrospect. It sounds like some of the
| things I was doing in 2009.
| chatmasta wrote:
| It includes a reference to Backbone and Knockout JS,
| which were released in 2010, so presumably it was around
| that era. The database, though, was probably much
| older...
| stavros wrote:
| At least we know it wasn't written after today!
| shermantanktop wrote:
| Falsehoods Programmers Believe...?
| klysm wrote:
| I'm still in the think hard about the schema camp. I like to
| rely on the database to enforce constraints.
| ibejoeb wrote:
| Yeah, a good database is pretty damn handy.
|
| Have you had the pleasure of blowing young minds by revealing
| that production-grade databases come with fully fledged
| authnz systems that you can just...use right out of the box?
| scythmic_waves wrote:
| Can you say more? I'm interested.
| stavros wrote:
| I guess they mean something like Postgres' row-level
| security:
|
| https://www.postgresql.org/docs/current/ddl-
| rowsecurity.html
| Merad wrote:
| Databases have pretty robust access controls to limit (a
| sql user's) access to tables, schemas, etc. Basic
| controls like being able to read but not write, and more
| advanced situations like being able to access data
| through a view or stored procedure without having direct
| access to the underlying tables.
|
| Those features aren't used often in modern app
| development where one app owns the database and any
| external access is routed through an API. They were much
| more commonly used in old school apps enterprise apps
| where many different teams and apps would all directly
| access a single db.
| jaggederest wrote:
| Here's a vendor, but the underlying functionality is
| shipped with Postgresql server
|
| https://supabase.com/docs/guides/database/postgres/row-
| level...
| hobs wrote:
| This is perfectly fine when you are driving some app that has a
| per-user experience that allows you to wrap up most of their
| experience in some blobs.
|
| However I would still advise people to use a third normal form
| - they help you, constraints help you, and often other sets of
| tooling have poor support for constraints on JSON. Scanning and
| updating every value because you need to update some subset
| sucks.
|
| You first point is super valid though - understanding the
| domain is very useful and you can get easily 10x the
| performance by designing with proper types involved, but
| importantly don't just build out the model before devs and
| customers have a use for anything, this is a classic mistake in
| my eyes (and then skipping cleanup when that is basically
| unused.)
|
| If you want to figure out your data model in depth beforehand
| there's nothing wrong with that... but you will still make tons
| of mistakes mistakes, lack of planning will require last minute
| fixes, and the evolution of the product will have your original
| planning gather dust.
| jsonis wrote:
| > Scanning and updating every value because you need to
| update some subset sucks.
|
| Mirrors my experience exactly. Querying json can get complex
| to get info from the db. SQLite is kind of forgiving because
| sequences of queries (I mean query, modify in appliation code
| that fully supports json ie js, then query again) are less
| painful meaning it's less moprtant to do everytning in the
| database for performance reasons. But if you're trying to do
| everything in 1 query, I think you pay for it at application-
| writing time over and over.
| jsonis wrote:
| > These days, my go-to solution is SQLite with two fields
| (well, three, if you count the implicit ROWID, which is
| invaluable for paging!): ID and Data, the latter being a JSONB
| blob.
|
| Really!? Are you building applications by chance or something
| else? Are you doing raw sql mostly or an ORM/ORM-like library?
| This surprises me because my experience dabbling in json fields
| for CRUD apps has been mostly trouble stemming from the lack of
| typechecks. SQLite's fluid type system haa been a nice middle
| ground for me personally. For reference my application layer is
| kysely/typescript.
| PreInternet01 wrote:
| > my experience dabbling in json fields for CRUD apps has
| been mostly trouble stemming from the lack of typechecks
|
| Well, you move the type checks from the database to the app,
| effectively, which is not a new idea by any means (and a
| _bad_ idea in many cases), but with JSON, it _can_ actually
| work out nicely-ish, as long as there are no significant
| relationships between tables.
|
| Practical example: I recently wrote my own SMTP server (bad
| idea!), mostly to be able to control spam (even worse idea!
| don't listen to me!). Initially, I thought I would be
| _really_ interested in remote IPs, reverse DNS domains, and
| whatever was claimed in the (E)HLO.
|
| So, I designed my initial database around those concepts.
| Turns out, after like half a million session records: I'm
| _much_ more interested in things like the Azure tenant ID,
| the Google 'groups' ID, the HTML body tag fingerprint, and
| other data points.
|
| Fortunately, my session database is just 'JSON(B) in a single
| table', so I was able to add those additional fields without
| the need for any migrations. And SQLite's `json_extract`
| makes adding indexes after-the-fact super-easy.
|
| Of course, these additional fields need to be explicitly
| nullable, and I need to skip processing based on them if
| they're absent, but fortunately modern C# makes that easy as
| well.
|
| And, no, no need for an ORM, except
| `JsonSerializer.Deserialize<T>`... (And yeah, all of this is
| just a _horrible_ hack, but one that seems surprisingly
| resilient so far, but YMMV)
| throwup238 wrote:
| _> And, no, no need for an ORM, except
| `JsonSerializer.Deserialize <T>`... (And yeah, all of this
| is just a horrible hack, but one that seems surprisingly
| resilient so far, but YMMV)_
|
| I do the same thing with serde_json in Rust for a desktop
| app sqlitedb and it works great so +1 on that technique.
|
| In Rust you can also tell serde to ignore unknown fields
| and use individual view structs to deserialize part of the
| JSON instead of the whole thing and use string references
| to make it zero copy.
| jvans wrote:
| why is a migration such a burden in that scenario
| Vampiero wrote:
| > Well, you move the type checks from the database to the
| app, effectively, which is not a new idea by any means (and
| a bad idea in many cases), but with JSON, it can actually
| work out nicely-ish, as long as there are no significant
| relationships between tables.
|
| That way you're throwing away 50% of the reason you use a
| relational database in the first place. Has it occurred to
| you that MongoDB exists?
|
| Also I don't understand why you're afraid of migrations,
| especially since you're the only developer on your own SMTP
| server.
| throwup238 wrote:
| _> That way you 're throwing away 50% of the reason you
| use a relational database in the first place. Has it
| occurred to you that MongoDB exists?_
|
| Did you miss that he's using sqlite? The dev experience
| with a sqlitedb is way better than running yet another
| service, especially for personal projects.
|
| Sqlite is used just as much as an application file format
| as it is a relational database.
| PreInternet01 wrote:
| > Has it occurred to you that MongoDB exists?
|
| My original comment started with "but it feels "prior to
| the MongoDB-is-webscale memes""
|
| So, care to take another guess? And, while we're here,
| does MongoDB run fully in-process these days? And/or
| allow easy pagination by ROWID?
| throwaway173738 wrote:
| How does MongoDB handle someone pulling the power cord
| out of the server? Because that's another reason to use
| something like SQLite, and it often gets used in embedded
| systems.
| fooster wrote:
| Same way as SQLite.
| randomdata wrote:
| _> Has it occurred to you that MongoDB exists?_
|
| What gain would MongoDB offer here?
|
| You certainly would lose a lot of things, like a well
| supported path to linking with to the database engine,
| and a straightforward way to start to introduce
| relational tables as the project matures. Nothing
| completely insurmountable, of course, but carry a lot of
| extra effort for what benefit?
| Izkata wrote:
| > Fortunately, my session database is just 'JSON(B) in a
| single table', so I was able to add those additional fields
| without the need for any migrations. And SQLite's
| `json_extract` makes adding indexes after-the-fact super-
| easy.
|
| Our solution for a similar situation involving semi-
| structured data (in postgres) was to double it up: put all
| the json we send/receive with a vendor into a json field,
| then anything we actually need to work on gets extracted
| into regular table/columns. We get all the
| safety/performance guarantees the database would normally
| give us, plus historical data for debugging or to extract
| into a new column if we now need it. The one thing we had
| to monitor in code reviews was to never use the json field
| directly for functionality.
| collinmanderson wrote:
| > I recently wrote my own SMTP server (bad idea!)
|
| Recently I've been tempted to make an SMTP server that
| translates emails into a web-hook. Please tell me more
| horror stories so that I might be convinced not do it.
| cyberax wrote:
| In my current company, we're using a similar approach: just
| shove everything into a JSON blob. If you need a constraint
| or an index, you can create a computed column (in PostgreSQL)
| that pulls out a field from JSON.
|
| For the data schema, we're using Protobufs with buf validate.
| This works surprisingly well, you can use the same types in
| the backend API and on the frontend. We even have a cron job
| that reads all the data periodically and verifies that the
| JSON blobs conform to the schema. Our next intern is going to
| write a PostgreSQL extension to do that on commit/update :)
|
| One real advantage of this approach is that it's easy to do
| stuff like "search everywhere".
| kubectl_h wrote:
| Do you go to the trouble of updating individual values in
| objects using some kind of deep/partial updating function
| or do you just accept race conditions that come with
| updating full objects?
| cyberax wrote:
| We use optimistic versioning, with a dedicated "version"
| field (that is actually always pulled out of the blob in
| all tables).
|
| Classic fine-grained schemas are not that much different.
| A lot of high-level ORM frameworks simply save all the
| objects' fields on update, without doing fine-grained
| diffs.
|
| In addition, our frontend apps also support offline mode.
| They can get all the relevant objects, and then operate
| on them locally. So our API was designed from the start
| to deal with conflicts.
| duskwuff wrote:
| A lot of databases include functions for manipulating
| JSON which can be used for atomic updates. SQLite has
| JSON_PATCH(), for instance.
| icedchai wrote:
| I've worked on a system were a guy "migrated" a database from
| NoSQL to SQL as a proof-of-concept. Except he really didn't
| migrate anything. He basically created tables with "ID"
| (UUID) and "data" (json) columns everywhere. No indexes.
|
| The actual "data" is a mess: different data types for the
| same JSON field. Imagine storing "price" as both a float and
| a string, depending on whatever buggy code happened to do the
| inserts.
|
| It worked enough for a prototype and that was enough for
| management to believe the project was a success. I can't wait
| until we actually try and deploy it.
| teaearlgraycold wrote:
| Byte vs. Int is premature optimization. But indexing, primary
| keys, join tables, normalization vs. denormalization, etc. are
| all important.
| RaftPeople wrote:
| > _Byte vs. Int is premature optimization_
|
| I think you can only judge that by knowing the context, like
| the domain and the experience of the designer/dev within that
| domain.
|
| I looked at a DB once and thought "why are you spending
| effort to create these datatypes that use less storage, I'm
| used to just using an int and moving on."
|
| Then I looked at the volumes of transactions they were
| dealing with and I understood why.
| teaearlgraycold wrote:
| Absolutely
| xp84 wrote:
| Deep down, the optimizer in me wants this to be true, but
| I'm having trouble seeing how this difference manifests in
| these days of super powerful devices and high bandwidth.
|
| I guess I just answered my own question though. Supposing
| there's a system which is slow and connected with very slow
| connectivity and still sending lots of data around, I guess
| there's your answer. An embedded system on the Mars Rover
| or something.
| chrisldgk wrote:
| I actually love your approach and haven't thought of that
| before. My problem with relational databases often stems from
| the fact that remodeling data types and schemas (which you
| often do as you build an application, whether or not you
| thought of a great schema beforehand) often comes with a lot of
| migration effort.
|
| Pairing your approach with a ,,version" field where you can
| check which version of a schema this rows data is saved with
| would actually allow you to be incredibly flexible with saving
| your data while also being able to be (somewhat) sure that your
| fields schema matches what you're expecting.
| dvdkon wrote:
| Having to write and perform migrations for every small schema
| change is a bore, but it means your software doesn't have to
| worry about handling different versions of data. Going
| "schemaless" with version numbers means moving code from
| "write-and-forget" migrations to the main codebase, where it
| will live forever.
|
| I think not doing database migrations only makes sense when
| you can make do _without_ version numbers (or if you can 't
| do atomic migrations due to performance constraints, but
| that's only a problem for a very small number of projects).
| JadeNB wrote:
| > Not having to write and perform migrations for every
| small schema change is a bore, but it means your software
| doesn't have to worry about handling different versions of
| data.
|
| Is that "not" at the front supposed to be there?
| dvdkon wrote:
| Thanks, edited.
| chrisldgk wrote:
| You're correct there. I mostly work on CMSes with page
| builder functionality, which often bake the content schema
| into the database columns, which makes changing that schema
| (for new frontend features or reworking old ones) difficult
| and often prone to losing content, especially in dev
| environments. Best case is obviously that you never have to
| version your changes, but I'd prefer making a new schema
| and writing an adapter function in the codebase depending
| on the schemas version to spending a lot of time migrating
| old content. That might just be due to me not being too
| comfortable with SQL and databases generally.
| layer8 wrote:
| > remodeling data types and schemas (which you often do as
| you build an application, whether or not you thought of a
| great schema beforehand)
|
| This is not my experience, it only happens rarely. I'd like
| to see an analysis of what causes schema changes that require
| nontrivial migrations.
| zo1 wrote:
| Same here. If your entities are modelled mostly correctly
| you really don't have to worry about migrations that much.
| It's a bit of a red herring and convenient "problem" pushed
| by the NoSQL camp.
|
| On a relatively neat and well modelled DB, large migrations
| are usually when relationships change. E.g. One to many
| becomes a many to many.
|
| Really the biggest hurdle is managing the change control to
| ensure it aligns with you application. But that's a big
| problem with NoSQL DB deployments too.
|
| At this point I don't even want to hear what kind of crazy
| magic and "weird default and fallback" behavior the schema
| less NoSQL crowd employs. My pessimistic take is they just
| expose the DB onto GraphQL and make it front ends problem.
| v-erne wrote:
| >> If your entities are modelled mostly correctly you
| really don't have to worry about migrations that much
|
| I'm gonna take a wild guess here that you have never
| worked in unfamiliar domains (like lets say deep cargo
| shiping or subpremium loans) where your so called subject
| matter experts provided by client werent the sharpest
| people you could hope for and actually did not understand
| what they where doing for most of the time?
|
| Because I on the other hand am very familiar with such
| projects and doing schema overhaul third time in a row
| for production system is bread and butter for me.
|
| Schemaless systems is the only reason I'm still developer
| and not lumberjack.
| xp84 wrote:
| I agree. Migrations have never been a problem at my
| company and this codebase is 9 years old. Just regular
| old postgres with a few JSONB columns of things that
| don't need to be relational.
|
| Same for the database that was about 8-9 years old at my
| last company. Migrations are fine. It's the relationship-
| changing that is painful.
| gonzo41 wrote:
| This is essentially just a data warehousing style schema. I
| love me a narrow db table.
|
| But I do try and get a schema that fits the business if I can.
| someuser2345 wrote:
| So, you're basically running DynamoDB on top of a sql server?
| arnorhs wrote:
| I think they are referring to the fact that software
| development as a field has matured a lot and there are
| established practices and experienced developers all over who
| have been in those situations, so generally, these days, you
| don't see such code bases anymore.
|
| That is how I read it.
|
| Another possible reason you don't see those code bases anymore
| is the fact that such teams/companies don't have a competitive
| comp, so there are mostly junior devs or people who can't get a
| job at a more competent team that get hired in those places
| nextaccountic wrote:
| why a separate id and rowid? why not just rowid and data?
| cowsandmilk wrote:
| He acts like sequence key is odd, but that's quite normal in
| database world.
|
| https://www.postgresql.org/docs/current/sql-createsequence.h...
| williamdclt wrote:
| yeah, I've seen tables with a single row and single columns a
| few times, for alright reasons. Sometimes you do just want to
| store a single global value!
| MarceColl wrote:
| Yes, but both these have very different properties. He said (I
| don't know if its the case) that the db didnt have an
| autoincremental type. Postgres uses these sequence objects to
| implement autoincremental ids as he was referring to, they are
| implemented in-engine and are very fast and have already solved
| data races.
|
| In the article, what he complains about is not about what a
| sequence is, but about implementing it manually with a table
| that is read, incremented and then saved. This us more
| expensive, and depending on how it was implemented you need to
| take care of the whole data flow so you are unable to allocate
| the same id twice. That's what he considers odd
| pbronez wrote:
| The scary thing to me about that setup is how the global
| value is updated. Every individual script must successfully
| increment the value to avoid duplicate keys.
|
| Really hope they had a "get key" stored procedure to handle
| that.
| SoftTalker wrote:
| It's possible they had some requirement for gapless ID
| numbers. You can't do that with a sequence.
| _elf wrote:
| Databases have built-in features for this now. What the author
| is talking about is a regular table.
|
| In reality, that wasn't too unusual to see because frameworks
| would use that technique because it's a lowest common
| denominator across RDMS.
| ssdspoimdsjvv wrote:
| Does SQLite have sequences yet?
| mnahkies wrote:
| I think the intriguing part was purposefully using the same
| sequence value for rows in multiple tables.
|
| I've worked with globally unique (to our application) integer
| keys, and per table integer sequences (which obviously aren't
| globally unique), but I don't recall seeing anyone use a global
| sequence but purposefully reuse elements of the sequence
| before.
| collinvandyck76 wrote:
| I've seen this fairly recently and I was surprised to also
| see it in this article because I think that makes it twice
| that I've seen it in ~25 years.
| airstrike wrote:
| They may even be the same codebase!
| linux2647 wrote:
| It's kind of like an idempotency key assigned either by the
| request or generated at the beginning of processing that
| request
| masklinn wrote:
| I think you missed the crucial part: every related record
| across all tables would have the same sequence item (the same
| id). That's really not normal in the database world. It sounds
| a lot like ECS though.
| codetrotter wrote:
| > went by the name Munch
|
| How do you pronounce that?
|
| Was it like the word munch in "munching on some snacks"?
|
| Or like the name of the painter Edward Munch?
| https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 (note:
| this link is in Norwegian)
| jimmyhmiller wrote:
| As in munching on snacks
| thrwaway1985882 wrote:
| Hey former colleague - just had to say hello on a comment
| where you might see. I started reading this article and
| everything started feeling so familiar... as soon as you told
| me Munch was the resident shaman, everything clicked.
|
| My favorite factoid for others was that when I was there, we
| used split-horizon DNS to squat on an in-use domain name for
| tons of internal services, including Github. I kept wondering
| what would happen if the owner realized & set up his own
| services to catch people who weren't on the VPN.
| rendall wrote:
| _::Edvard Munch scream emoji::_
| trte9343r4 wrote:
| Slicing columns into multiple tables is fairly common type of
| sharding. Sort of SQL way to do columnary store.
| pelagicAustral wrote:
| This is such an horrendous practice, I am yet to find a
| database were this makes sense. Maybe my brain is not wired for
| it
| zo1 wrote:
| The problem is maybe not so much the splitting and putting
| extra columns in a separate table. It's that you even have a
| table that large that it necessitates such a thing. Worst
| case you have a main table and a detail table that has a one
| to one correlation to the main entity table.
| magicalhippo wrote:
| Why is that worse than a couple of dozen joins?
| zo1 wrote:
| Because that means your data is highly denormalized and
| has plenty of duplicates. But in all likelihood it means
| no one knows wtf this table actually represents and you
| should be firing people.
|
| I've seen this play out. Usually the many columns is
| because everyone misuses the table and eventually their
| special little business scenario or "filter" needs to be
| a column. Bonus points is whoever has to reference this
| table, they have to copy over whatever the hell your PK
| seems to be, and the cycle repeats, this time a bit
| worse.
|
| Last place I did a brief project in had this. Queue 1000
| tables spread across 25 schemas, each table having wide
| PKs, 20 redundant indexes on each table, and despite all
| this the database performs poorly. No one can tell you
| what each table represents, the table names are
| meaningless and the same data is everywhere. In order to
| get anything done you have to ask a small cabal of
| priests that knows the processes that write between these
| tables. After about 10 years, a partial rewrite happens
| and you now have 1/3rd of the data on each side with
| plenty of duplicate and overlap because hey.
|
| I feel torn, I really wanna name&shame this company as a
| warning to all developers thinking about working there.
| pelagicAustral wrote:
| I feel you... I mean "...you should be firing people",
| this is my day to day way of thinking.
|
| My thoughts are that, hyper-specialization, and grind
| breed this type of data structure. But so many companies
| are forced to choose, and generally tend to sacrifice on
| the database side of things. Then you end up with this
| type of unruly structure.
|
| Database theory and practice should be a MUST on all
| software development courseware.
| magicalhippo wrote:
| > But in all likelihood it means no one knows wtf this
| table actually represents and you should be firing
| people.
|
| That's almost exactly the opposite of my experience, but
| then I've worked on smaller teams with long-term team
| members, so perhaps that's why.
|
| Our tables are wide because law require we store the data
| for 10 years, and it's easier and faster to store things
| in a single row, rather than spread out over several
| dozen detail tables.
|
| We don't denormalize in the sense of storing only invoice
| items, repeating the head data on each item. We do
| denormalize in the sense of storing full name and address
| for each party rather than storing that in a detail
| table. So minimal, if any, duplication of data. If we
| duplicate it's almost always due to the law, ie data
| might change in one table but shouldn't be changed in the
| other to preserve history.
|
| For child tables we always have a unique ID, sequence or
| autoinc, and the FK to the parent, but we include the
| root-level ID as well. This way we can select all the
| child rows in a single select regardless of level, again
| without tons of joins.
|
| This way our core data is just a handful of tables, and
| there's not much doubt about what goes where.
| mort96 wrote:
| But splitting into multiple tables because you hit the 1024
| column limit is probably not a common type of sharding...
| dakiol wrote:
| Honestly, I like to work on such systems because:
|
| - there is so much stuff to improve. There's nothing better than
| the feeling of improving things with code (or by removing it)
|
| - it's back to school again and everything goes. You can
| implement features in any way you want because the constraints
| the system imposes. Granted, sometimes it's painful to add
| functionality
|
| - there's usually no room to subjective topics like clean code
| and architecture. The most important thing with these systems is
| correctness (and this is usually an objective topic)
|
| - nobody can blame you for something that doesn't work. It's
| always the fault of the legacy system
|
| I wouldn't recommend working on such systems to junior engineers,
| though.
|
| I don't really like to work on "perfect" codebases where everyone
| follows the same pattern, with linters, where if something breaks
| is because of your shitty code (because the codebase is "clean").
| It's very frustrating and limiting.
| korhojoa wrote:
| I mean, it is kind of nice to notice that something isn't going
| to work because you have the linters and tests. When your
| developer count goes up, chances that erroneous behavior is
| included also goes up.
|
| I've created proof-of-concepts that worked perfectly but would
| make you cry if you looked at how they worked. Eventually they
| became that mess. Everything is a self-contained unit so it
| doesn't mess anything else up. Of course, there is always time
| to keep adding new stuff but never to refactor it into what it
| should be.
|
| I prefer the way with linters and tests, it at least lessens
| the chances of whatever is put in being broken (or breaking
| something else). (Then again, somebody putting "return true" in
| a test _will_ surprise you sooner or later)
| poikroequ wrote:
| > there is so much stuff to improve. There's nothing better
| than the feeling of improving things with code (or by removing
| it)
|
| And there is nothing worse than a crappy codebase the company
| won't let you improve.
| SoftTalker wrote:
| You also get to see some genuinely creative stuff that works
| well, written by people who aren't indoctrinated in a
| particular approach.
| gamepsys wrote:
| > All that remained were ragtag interns and junior developers.
|
| For many people, their first job in software engineering is the
| worst codebase they will deal with professionally for this
| reason. The first job hires lot of people with little/no
| experience. As soon as someone gains some experience than can
| move on to better paying jobs, where there are better developers
| with better standards.
| prudentpomelo wrote:
| This is the exact boat I am in. I always say the best thing
| about our codebase is the worst thing: junior developers can do
| whatever they want.
| mleo wrote:
| Worst code bases are often ones taken over from IT
| consultancies. They drive young, inexperienced developers
| working many hours to deliver functionality. While the project
| may start out "clean" using whatever is the current hotness in
| technology, at some point getting stuff developed and throwing
| over the wall to QA is the important part.
| surfingdino wrote:
| I had dubious pleasure of working with similar codebases and
| devs. I'll remember one of those guys forever, because whenever
| he wanted to work on a new branch he would clone the repo, make
| changes to the master branch, and push code to a new repo
| numbered repo0001, repo002, ... He refused to change his ways,
| because "I have a PhD so you are wrong".
|
| Another WTF moment was realisation that MS SQL Server does not
| support BOOLEAN type. That made porting code fun.
| marcosdumay wrote:
| > Another WTF moment was realisation that MS SQL Server does
| not support BOOLEAN type.
|
| The standard does not have a boolean type. It's a postgres
| extension that the other open source databases adopted
| (because, yeah, it's obvious). But the proprietary ones insist
| on not having.
|
| The official recommendation is using byte on MS SQL and char(1)
| on Oracle. Both are ridiculous.
| nolist_policy wrote:
| _C programmer joins the chat._
| mort96 wrote:
| Hey we've had _Bool and <stdbool.h> since 1999!
| dvfjsdhgfv wrote:
| Well, everything is a boolean in ANSI C. If I use `if (a)`
| I know the code will be executed only if `a` is nonzero.
| bdcravens wrote:
| Pedantic, but the type is bit in MSSQL.
| Nexialist wrote:
| My worst codebase story:
|
| In my first real job, I worked for a company that maintained a
| large legacy product programmed in a combination of COBOL and
| Java.
|
| In order to work on the Java side of the product, you checked out
| individual files from source control to work on, which 'locked'
| the files and prevented other developers from checking out the
| same files. This functionality was not part of our actual source
| control system, but was instead accomplished with a series of csh
| shell scripts you could run after ssh'ing into our development
| server.
|
| Each of our customers had a 'master' jar file that represented
| the actual final compiled product (a jar file is really a zip
| file archive, which bundles together the resulting compiled java
| class files).
|
| Once you had finished implementing your code changes, you ran
| another set of scripts which found the master jar file for each
| customer, unzips it, copies the compiled files from your local
| machine into it, and zips it back up again. Finally the source
| control lock is released.
|
| This means, effectively, that the codebase was never compiled as
| a whole at any point in the process, instead, we just manually
| patched the jar file over time with individually compiled class
| files.
|
| Over the years, small errors in the process allowed a huge amount
| of inconsistencies to creep into the codebase. Race conditions
| would allow two developers to lock the same file at once, or a
| developer would change a class that was a dependency of some
| other code that somebody else was changing. Sometimes code
| changes would make it into some of the customer jar files, but
| not others. Nobody knew why.
|
| It took a small team two years to migrate the entire codebase to
| git with proper CI, and a huge chunk of that time was reproducing
| a version of the codebase that actually compiled properly as a
| whole. After the project was finished, I resigned.
| sir-dingleberry wrote:
| This is insane. Thanks for the post.
| pelagicAustral wrote:
| This sounds so much like dealing with MS Access databases...
| Unfortunately, part of my responsibility in my current role is
| to manage a handful of Access applications... they are ancient,
| I am talking early 2000's. They are the most unruly thing I
| have ever had to work with, and I will not go into details but
| your story reminds me so much of having to work on these
| apps...
| stavros wrote:
| We all take too much for granted how much git "just works",
| even when it doesn't.
| Izkata wrote:
| Even those before git - what they're describing sounds a bit
| like RCS, the precursor to CVS, which came before SVN.
|
| I've never used RCS or CVS myself, but I remember the file
| locking thing in descriptions of it, and that it was why CVS
| was named "concurrent" - it fixed that limitation.
| whatever1 wrote:
| I bet now they deal with broken pipelines and dependency hell.
|
| Tools do not fix bad design.
| mikeocool wrote:
| I recall something similar from my first job, except the shared
| file locking was a full on feature in Macromedia dreamweaver.
|
| CSS was just starting to get adopted and every project we
| worked on just had one "gobal.css" file. When someone else had
| global.css locked, you'd call dibs if you needed it next.
| Inevitably, everyday someone would leave the office and forget
| to unlock global.css and no one else could get anything done.
| q7xvh97o2pDhNrh wrote:
| ...and whoever did that "accidentally" when they had to leave
| around lunchtime on Friday was, presumably, celebrated as a
| local legend.
| mnahkies wrote:
| Macromedia flash / .fla files weren't particularly ideal for
| collaboration either, though I still feel a bit nostalgic for
| working with flash in general
| evan_ wrote:
| It just did this by creating a sentinel file- so if you
| needed to, you could just delete the file manually.
| masklinn wrote:
| SVN provided the ability to steal locks, and locks were
| opt-in so e.g. you wouldn't have made CSS need locks
| because it's a text file and merges fine. Mandatory locking
| was mostly for binary work files e.g. PSD and the like.
| mikeocool wrote:
| Good to know, though I'm optimistic this is knowledge I'll
| never need again!
| cjbgkagh wrote:
| So they had a problem, got 2 years of approved development
| effort of a small team to solve it property which they did
| successfully, and then you resigned? After they fixed the
| problem?
|
| Of course where they started was just awful but a place that
| recognized it's problems, commits to fixing it, and has
| sufficient competency to actually fix it sounds rather nice to
| me. Many orgs get stuck at step 1.
|
| I presume there were other reasons for resigning, or you just
| like massive refactoring projects.
| Nexialist wrote:
| It was a little tongue in cheek, but yes. I had large
| grievances with the software culture there, but after I got
| sign off on the project to modernise our build process, I
| couldn't bring myself to abandon ship in the middle of trying
| to fix it.
|
| After everything was finished up, I was feeling burnt out and
| realised that I'd held on for too long at a company with a
| fundamentally bad culture that wasn't going to change just
| because the tech did, so I moved on.
| gnat wrote:
| Thank you for the clarification. Because you said "it took
| a small team ... and then I resigned", it was unclear that
| you were part of that small team and instead made it sound
| like you left because the problem was fixed.
| dfee wrote:
| For what it's worth, it wasn't unclear when I read it.
| werdnapk wrote:
| Visual SourceSafe would show that a file was checked out
| hinting to maybe stay away. Good times.
| heywire wrote:
| We still use VSS for a couple codebases. We're a small enough
| team that conflicts are rare, but the occasional Teams
| message "hey, let me know when you're done with module.cpp"
| is not unheard of.
| cebert wrote:
| I'm impressed VSS still works. I used it as part of my
| first professional software engineering job in 2008 and it
| felt old then.
| jsrcout wrote:
| I'm so sorry. We were using it in the late 90s, and on a
| monthly basis it would be down for a day or two while the
| admin plunged out the database. Good times.
| heywire wrote:
| It surprisingly works well. We do use a front end to it
| called VssConnect, but to be honest, I really don't have
| any complaints.
| varjag wrote:
| Fascinating. An anachronism worthy of steampunk novels!
| telgareith wrote:
| Did you know, theres a GitHub repo called [you-poor-
| bastard](https://github.com/SirkleZero/you-poor-bastard)? It
| converts a vss repo to git (not very well, but well enough),
| ignoring the VSS "passwords."
| SoftTalker wrote:
| IIRC SourceSafe could be configured with strict locking or
| advisory locking? I might be wrong about that.
|
| The admin user could override or unlock locked files. We had
| to do this if a developer left a file checked out after they
| left, or were on vacation. Nobody knew the admin password for
| the SourceSafe repository. That was OK though, all you had to
| do was make a local account on your PC named the same as a
| the source safe admin acccount, and you'd have admin access
| in SourceSafe.
| masklinn wrote:
| > IIRC SourceSafe could be configured with strict locking
| or advisory locking? I might be wrong about that.
|
| IIRC SourceSafe could be configured with either strict
| locking (you had to lock a file in order to edit it) or no
| locking a la SVN (the tool would check if your file was up
| to date when trying to commit it).
|
| I recall that I spent a while suffering under strict
| locking at my first internship before a colleague
| discovered this mode existed and we were able to work more
| reasonably (there were no editable binary files so locking
| was never really needed).
| wredue wrote:
| I've worked on "check out" code bases very similar to this. I
| mean. Nothing so insane as scripts patching class files in to
| jars Willy nilly, but it seems the "just check out a file so
| nobody else can work on it" thing is a "common" "solution" to
| this.
|
| I am interested in how you managed it when someone had to hold
| code for years (as I've seen)? Basically, we also had a way to
| share control with one other person, and the person who was
| taking the second source were responsible for updating both
| versions at the same time manually (which never happened, so
| implementing a large project that touched hundreds of source
| files had to dedicate a couple weeks to manually hand comparing
| files, manually implementing the changes, and then manually
| retesting)
| brightball wrote:
| Had a similar thing on a small team at a bigger company. We
| didn't have any type of version control so the team did all
| development on a shared dev server that we SSH'd into. We'd use
| vim to edit files one at a time and rely on vim's file locking
| to make sure that we weren't editing anything at the same time
| as anybody else.
|
| Oddly, the project was one of the best I've ever worked on.
| ufmace wrote:
| I think a lot of people who've only learned programming in the
| last 10 years or so don't realize that Git, and even more so
| the large-scale popularity of Git, is actually pretty new. A
| lot of programming happened before Git was created, and before
| it became mature and popular enough to be the default for
| everyone.
|
| Many of these earlier systems sound terrible and hacky to us,
| but they were the best that was available at the time. Systems
| to "lock" files you were working on were pretty common because
| basically nobody did merges well. Most of them were based on
| having a server to manage the whole thing too, so they were
| only really common in corporations and larger and more mature
| hobbyist teams - this was also before you could spin up a cloud
| server for most things with a few keystrokes and $5 a month.
| It's low-effort to spin up a local Git repo to track your work
| on a tiny personal project, but nobody's going to set up a CVS
| server for that.
|
| Anybody remember Joel On Software's 12 Steps [0], written in
| the year 2000, 5 years before the Git project was started?
| Where "use source control" is Step 1? There's a reason why
| that's there - at the time it was written, source control was
| clunky and a pain in the ass to set up, so a lot of smaller
| companies or ones with less mature development teams never got
| around to it.
|
| I may be getting a little "old man yells at clouds" here, but
| be really thankful that Git is FOSS, ubiquitous, works great
| for 98% of projects, and is superlatively awesome compared to
| everything that came before it.
|
| [0] https://www.joelonsoftware.com/2000/08/09/the-joel-
| test-12-s...
| fiddlerwoaroof wrote:
| I use CVS, RCS and Subversion on my personal projects for
| several years before I learned git. I don't remember any of
| them being a pain to setup for small projects and Sourceforge
| provided free CVS hosting for small projects.
| brabel wrote:
| I started with Rational Clearcase, which I interacted with
| via Eclipse, and I remember that you did have to lock a
| file to make changes to it... pretty sure there was no
| merge capability (but maybe I was just too junior to know
| stuff like that, it was definitely not something you would
| do as a matter of routine, like we do now with git).
| Someone leaving for vacation and forgetting to unlock a
| file could be a huge pain in the ass.
|
| After that I used Subversion via TortoiseSVN for Windows,
| which was quite nice to use, though I don't really remember
| much of the details... I do remember asking myself why
| everyone was moving to git when "perfectly good" source
| control already existed, but after 10+ years on Git, I
| think I was wrong and we did need it, we just didn't know
| it yet.
| redserk wrote:
| Rational Clearcase is a disaster but you could do merges.
|
| The fun/frustrating part was getting branching and
| "views" correct.
|
| I worked at an organization on this side of the century
| where it was one person's full-time job to handle merging
| code from different teams. The same organization didn't
| actually use raw Clearcase either -- it was wrapped in
| hundreds-of-thousands-of-lines-of Perl scripts.
| ufmace wrote:
| I've never actually set up CVS myself, so I suppose I can't
| accurately say how easy or not easy it is. The "nobody"
| might deserve an asterisk - I'm sure it's not literally not
| a single person, but it seems to be pretty rare in my
| impression. Nevertheless, AFAIK, the non-distributed nature
| seems pretty limiting.
|
| In Git, I can just create a local repo for anything, and I
| don't have to decide where I want it. I could never push it
| anywhere, or push it to a private hosted repo to
| collaborate with a person or team, or push it to a public
| repo to open-source it, and change between any of those at
| any time. I don't think you can really do that with these
| hosted systems. You could spin up a local CVS server if you
| really wanted to, but if you later decided you wanted to
| open-source it on Sourceforge, don't you have to abandon
| the history and publish it there as-is?
| wiether wrote:
| > In my first real job, I worked for a company that maintained
| a large legacy product programmed in a combination of COBOL and
| Java.
|
| > In order to work on the Java side of the product, you checked
| out individual files from source control to work on, which
| 'locked' the files and prevented other developers from checking
| out the same files.
|
| COBOL + Java and file lock... I thought we worked on the same
| project and it brought back many nightmares!
|
| But we were using Subversion's lock system and had a proper
| CI/CD with Jenkins.
|
| So you won, your project was worse!
| matheusmoreira wrote:
| > It took a small team two years to migrate the entire codebase
| to git with proper CI
|
| I'm amazed they even allowed people to spend company time and
| money doing this. Corporations tend to not see the value in
| improving such precarious situations as long as the software
| keeps working. They just want employees to cope and accept the
| messiness. People usually cope by quitting.
| tryauuum wrote:
| Would be great to work in such a company as a Linux guru.
|
| There are so many entangled services and machines that you feel
| like an Indiana Jones. You ssh into a machine and feel century-
| old dust beneath your footsteps. And you never know what will you
| find. Maybe a service which holds the company together. Maybe a
| CPU eating hog which didn't do anything useful last 3 years.
|
| I don't enjoy writing new code much. But in such an environment
| even with my limited skills I can do decent improvements,
| especially from security point of view. Feels great
| noisy_boy wrote:
| Basically the entire application is loaded with low hanging
| fruits and things that look like fruits but are bombs that
| explode upon contact.
| arnorhs wrote:
| Well it sounded like there are exactly 0 Linux machines running
| there.. it's all windows .net / c# and a bunch of native
| windows apps, as I understood the article.
|
| But maybe you can replace your statement with "windows guru"
| and SSH with "remote desktop" and perhaps that would be fun
| tryauuum wrote:
| well, with linux at least I get get the source code of the
| kernel and of the MySQL when given a database server which
| hasn't been restarted for 7 years
| doublepg23 wrote:
| When I started my recent job the team kept referring to a box
| running "weird linux". After getting on-boarded and accessing
| the box it turned out to be running a very old version of
| OpenBSD. To this day I'm curious who had the wherewithall to
| install OpenBSD in prod but was seemingly ignorant of the 1yr
| support cycle.
| xp84 wrote:
| I started reading and was hoping someone had installed one of
| the Linux distros that were popular at the turn of the
| millennium, like Mandrake, Slackware, etc. and it was still
| trucking along 25 years later.
| airstrike wrote:
| This is glorious. I can imagine people could write versions of
| this for every industry out there and they would all be equally
| fun to read.
| Twirrim wrote:
| Back about 15 years ago, I worked for a web hosting company that
| provided some sysadmin consultation services. Customer paid us,
| and I would take a look.
|
| I had one customer who came back with the same request, slightly
| differently worded, every single month, and every single month
| I'd say the same thing. They had this site they were running that
| was essentially a Yellow Pages type site. They had a large set of
| companies with contact details, each with multiple business
| categories associated with it. You'd choose a category, and
| they'd return a list of matching companies.
|
| The problem was the site was really slow. I took a quick look
| around, and saw that all the time was lost querying the database.
| Taking a quick look at the schema I discovered that their
| approach to categorisation was to have a TEXT column, with
| semicolon separated 4 character strings in it. Each 4 character
| string mapped to a business category.
|
| So when someone wanted to load up, say, all pest control
| companies, it would check the category mapping table, get the 4
| character string, and then go to the companies table and do:
| SELECT * FROM companies WHERE categories LIKE "%PEST%"
|
| So on each page load of the main page type the site was there to
| provide, it did a full text search over the category field for
| every single record in the company table.
|
| I guess that's probably okay for the developer without real world
| scale data, and real world traffic counts to worry about. But
| they had lots of data in the database, and that category field
| could have dozens of categories against a company. As soon as
| they had more than about 4-5 simultaneous customers performance
| started tanking.
|
| I could never get them to accept that they needed to rethink the
| database schema. One month they were bleating about how is it
| possible that Google can manage to do such a search across a much
| larger amount of data, much faster. They really didn't like my
| answer that amounted to "By having a sane database schema". All
| they were willing to do was pay over the odds for our most
| powerful server at the time, which had enough capacity to hold
| the entire database in memory.
| throwup238 wrote:
| In case anyone is looking for a performant way to implement
| categories like that in Postgres:
| https://news.ycombinator.com/item?id=33251745
|
| I stumbled across that comment a few years back and it changed
| the way I handle tags and categories so just sharing it here.
| If anyone has an equivalent for Sqlite, I'd love to hear it!
| wizzwizz4 wrote:
| It's a relational database: why not just use a PageCategories
| table with two foreign keys?
| Twirrim wrote:
| That's what my suggestion came down to. A two column table
| with company id and category id. They had ids already. They
| could index off category and get the results in split
| seconds
| tehlike wrote:
| postgres also supports gin indexes, so could do int arrays &
| contains queries, for example.
| dmd wrote:
| Probably some of the worst code I ever worked on was a 12k+ line
| single file Perl script for dealing with Human Genome Project
| data, at Bristol-Myers Squibb, in the late 1990s.
|
| The primary author of it didn't know about arrays. I'm not sure
| if he didn't know about them being something that had already
| been invented, or whether he just didn't know Perl supported
| them, but either way, he reimplemented them himself on top of
| scalars (strings), using $foo and $foo_offsets. For example, $foo
| might be "romemcintoshgranny smithdelicious" and $foo_offsets =
| "000004012024", where he assumes the offsets are 3 digits each.
| And then he loops through slices (how does he know about slices,
| but not arrays?) of $foo_offsets to get the locations for $foo.
|
| By the time I was done refactoring that 12k+ was down to about
| 200 ... and it still passed all the tests and ran analyses
| identically.
| rokkamokka wrote:
| Goes to show that solutions don't need to be good to be
| impressive. Because damn, I'm impressed he did that
| dezsiszabi wrote:
| That's a very impressive achievement, reducing that to around
| 200 lines. Congrats :)
| itsameputin wrote:
| Well the lines could very well be 1000s of bytes long. Perl
| oneliners used to be a thing in a company that I used to work
| for.
| sandebert wrote:
| Nice. But I have to ask, considering it was Perl: Could an
| outsider understand it after you reduced it to 200 lines?
| dmd wrote:
| Certainly better than they could when it was 12000.
| randomdata wrote:
| 199 of the lines were comments.
| hinkley wrote:
| We should use the Antarctic highlands as a prison colony for
| people found guilty of writing Stringly Typed Code. Siberia
| isn't awful enough for them. I thought people who stuffed
| multiple independent values into a single database column were
| the worst and then I saw what people can accomplish without
| even touching a database.
| dmd wrote:
| Ha - we did that too at BMS. We were paying Oracle by the
| column or something like that, so people would shove entire
| CSV rows into a single value (because corporate said
| everything HAD to be in Oracle) and then parse them
| application-side.
| nyokodo wrote:
| > people would shove entire CSV rows into a single value
|
| So you invented a "no-sql" document database in Oracle.
| hinkley wrote:
| We got rid of most of the DBAs because they got overfond
| of saying 'no' and they became a major source of friction
| for doing reasonable architecture. People started
| figuring out ways to work around the DBAs like stuffing
| more shit into existing columns or adding secondary
| tables that established new relationships between the
| sacrosanct tables.
|
| A lot of the worst sins happened on Oracle databases,
| mostly because they were the most popular at the time,
| and they were the only real DB cult, so the cottage
| industry of specializations that told other developers
| not to worry their pretty little heads was especially
| bad.
| harry_ord wrote:
| Admittedly I've somehow only worked in perl but the worst code
| I tried for fix felt similar. They know about arrays but every
| map and grep used perks default $_ and there was enough nesting
| the function was near 1k lines if I remember right.
| apwell23 wrote:
| > and it still passed all the tests
|
| All his sins can be forgiven because he made it so easy to
| refactor by writing comprehensive tests.
| layer8 wrote:
| You were very lucky there were tests.
| arghnoname wrote:
| If I were being charitable (sometimes I try), I'd guess he was
| concerned that an array of scalars is going to have a high per
| scalar size overhead and perhaps worse cache locality, whereas
| the non-unicode perl string implementation at the time probably
| just stored the string as a large char* with a length prefix.
|
| In a similar vein, an alternate explanation may just be that a
| lot of bioinformatics algorithms are described as string
| algorithms. It would be a pity if that was just taken too
| literally, but I've seen a lot of bioinformatics perl that
| works on strings in this way.
|
| While trying to be charitable though, it's almost impossible to
| come up with a rationalization for the three character offsets
| string. Sometimes I think when people really go wrong is when
| they hear of some problem or are trying to be too clever and
| end up going beyond their abilities and stray into farce.
| lemme_tell_ya wrote:
| My first day at my first full-time programming gig, I was asked
| to look at some reporting job that had been failing to run for
| months. I logged in, found the error logs, found that it needed a
| bit more memory assigned to the script (just a tweak in the
| php.ini) and let the team lead know it should run fine that
| night. He was shocked, "Dude, if you just fixed that report you
| probably just got a promotion, no one has been able to figure
| that out for months." He was joking about the promotion, but my
| boss was just as shocked. I'd realize later that most the other
| people on the dev team didn't like linux and wanted to rewrite
| everything in .NET and move everything to Windows so no one even
| tried with anything related to any of the linux machines.
| mleo wrote:
| I know things have gotten somewhat better, but the amount of
| wasted time and latency of using RDP and Windows UI for
| development, testing and production maintenance is insane.
| Throw in some security requirements of RDP into host 1 to the
| RDP jump to host 2 and companies are just wasting money on
| latency. There is, often, not an appreciation of the
| administrative costs of the delivery. Not necessarily system
| admin costs, but developer and QA time associated with
| delivering and ongoing maintenance.
| lemme_tell_ya wrote:
| Oh yeah, been there. I personally find Windows to be a
| terrible server OS. I'd rather be stuck with a fleet of AIX
| machines or something equally uncommon (but still nix-ish)
| than any Windows system.
| jiggawatts wrote:
| The reverse also applies, where people that like Linux just
| totally ignore Windows maintenance. They'll look down their
| noses at it and say that it's insecure when they spend their
| time patching Linux and leaving Windows to fend for itself
| against attackers.
| interactivecode wrote:
| sure it might be a mess, but at least it's purpose built. I love
| that kind of performance gains. Honestly most companies die
| before purpose built code like that becomes a problem.
| bloaf wrote:
| Truly, the Tao was alive in that company.
|
| https://www.mit.edu/~xela/tao.html
| qxmat wrote:
| Jira's now discontinued server version had a sequence table to
| stop you sharding it. It also made disaster recovery from a hard
| shutdown awful. I have nothing good to say about Atlassian.
| Atreiden wrote:
| Atlassian, and JIRA specifically, are responsible for so much
| wasted time and capital expenditure. If I could get ahold of
| metrics like "hours spent building, using, and maintaining
| JIRA" versus "value obtained from JIRA" for each of the
| companies I've worked at, I'm pretty sure I could generate a
| report so scathing that nobody would ever use it again.
|
| Gatekeeping your work organization system from the people
| working on, and often MANAGING it is such a huge friction point
| I'm amazed they ever got clientele. I'm an Admin on a project
| right now and I can't change our status types without going
| through our assigned Atlassian rep.
|
| And don't even get me started on the dumpster fire that is
| BitBucket. Ever tried to use the API? it's somehow even more
| worthless than the UI.
| mrighele wrote:
| > Now the story I heard at the time was that once upon a time SQL
| Server didn't support auto-incrementing ids. This was the
| accepted, correct answer.
|
| At a company that I used to work, they heard the same rumor, so
| instead of using identity columns or sequences, they kept a table
| with a number of ids "available" (one row per id). Whenever
| unique id was needed, the table would be locked, an id selected
| and marked as used. If there were no ids available, more ids
| would be added and then one used. A scheduled job would remove
| ids marked as used from time to time. Note that there was a
| single "sequence table", that was shared among all of the
| entities.
|
| That was not even the weirdest part. That id was unique, but NOT
| the primary key of the entity, only part of it.
|
| The structure of the database was fairly hierarchical, so you had
| for example a table CUSTOMER in 1-to-many relation with a USER
| table, with a 1-to-many relation with an ADDRESS table.
|
| while the primary key of the CUSTOMER table was a single
| CUSTOMER_ID column, the primary key of the USER table was
| (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table
| was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or
| 6 columns as a primary key.
| bdcravens wrote:
| A couple of weeks ago I had a flat on the way to the airport. It
| was a total blowout, and my car doesn't include a spare. We were
| already over budget on our trip, so I had the car towed to the
| closest tire shop and had them put on the cheapest tire that
| could get me to the airport. I know I'll need to replace other
| tires, as it's an AWD, and I know it's not a tire I really want.
| I made a calculated choice to make that a problem for future me
| due to the time crunch I was under.
|
| Programming is a lot like this.
| alt227 wrote:
| Aside - the fact that cars dont include spares these days is
| infuriating.
|
| IMO its a classic example of real world enshittification
| bdcravens wrote:
| Supposedly there are reasons (for example, efficiency by
| avoiding the extra weight) and they give you a kit that's
| essentially a "fix a flat" product, but in the 2 times I've
| had a flat in a car outfitted like this, what they provided
| wasn't useful, usually due to the type of damage to the tire.
| fragmede wrote:
| that's not enshittification!
|
| it sucks, yes, but when coined, the term referred to a
| platform extracting value from its users by degrading their
| experience, forcing them to upgrade to a premium plan. your
| car not having a spare isn't an example of that!
|
| https://en.wikipedia.org/wiki/Enshittification
| rendall wrote:
| I'm glad OP was able to maintain a good sense of humor about it.
| Such discoveries as the nested classes of empty methods have sent
| me into teeth-gnashing fury followed by darkness and despair. One
| such experience is why I would rather change careers if my only
| option were to build on the Salesforce platform, for instance.
| jkestner wrote:
| Early on in my career, at a large company, I encountered someone
| who took "codebase" a little too literally. At the time every
| department had their own developers, sometimes just employees who
| had an aptitude for computers.
|
| This one guy established himself by making an Access database for
| their core business, and when the web became a thing, built a
| customer site. But not on it--in it. He simply served ASP pages
| directly from the database, inserting dynamic content in queries.
| When I was asked to help improve their terrible design, I was
| forced to untangle that unholy mess of queries, ASP (new to me)
| and HTML. It was easiest to write all the HTML and insert their
| ASP right before I sent the files back (because I wasn't given
| access to their DB/web server). Thinking "I could do better than
| this" got me into programming.
|
| He was a Microsoft-everything head. Finally went too far when he
| presented a new web interface starring a Clippy-like parrot using
| Microsoft's DirectX avatar API. The executives were unimpressed
| and then I noted that 20% of our customers couldn't use his site.
| (I probably still had a "best viewed with IE" badge on the main
| site, lol)
| omoikane wrote:
| This codebase sounds like a haunted graveyard[1], where everyone
| just fixes their local corner of things and avoid the risk of
| untangling the existing mess.
|
| Not needing to conform to some company-wide standard is probably
| really pleasant while it lasted, but every such effort adds to
| the haunted graveyard, and the lack of consistency will
| eventually come back to bite whoever is still around.
|
| [1]
| https://www.usenix.org/sites/default/files/conference/protec...
| collinmanderson wrote:
| It could maybe work if every team is responsible for their own
| area of the code base, where the code base starts to match the
| company tree
| brunoarueira wrote:
| Once not long enough, I'd worked on 4 projects which was
| literally copied from the first made and changed parts of the
| customers and internal users according to each use of it. So, the
| main problem is bugs found in one project was found on the other
| 3 and I'd to fix the same bug!
|
| Codebases like this or from the OP is cool to learn how to not do
| certain things.
| mg wrote:
| That is why people these days tend to use a single JSON blob
| instead of multiple columns. And because it is so popular, SQLITE
| and other DBs are building better and better JSON support into
| the DB.
|
| I wonder if better support of EAV tables would solve this issue
| better.
|
| If one could do "SELECT price,color,year FROM cars! WHERE
| status='sold'" and the "!" would indicate that cars is an EAV
| table ... entity attribute value 1
| price 20750 1 color red 1 year
| 2010 1 status sold
|
| ... and the result of the query would be ...
| 20750 red 2010
|
| That would solve most of the use cases where I have seen people
| use JSON instead.
| telgareith wrote:
| First real job, stayed for 10yrs, 5yrs too long, put VB6 COM
| objects into the asp.net codebase.
| arnorhs wrote:
| I really love these kinds of stories. Does anybody know if
| there's a collection of similar stories/code bases anywhere?
|
| I guess there is dailywtf but that's mostly bugs. Probably good
| enough though
| jancsika wrote:
| Oh man, this article reminds me of an article that was a parody
| of some horrid business logic.
|
| Something like this: a "genius" programmer was somehow, for some
| reason using svn commits as a method dispatcher. Commit ids were
| sprinkled throughout the codebase. A new hire broke the entire
| system by adding comments, and comments weren't compatible the
| bespoke svn method dispatcher.
|
| Does anybody remember this article? I really want to read it
| again.
| carderne wrote:
| The Inner JSON Effect. Most recent HN discussion of it:
|
| https://news.ycombinator.com/item?id=40923258
| dang wrote:
| Thanks! Also:
|
| _The Inner JSON Effect (2016)_ -
| https://news.ycombinator.com/item?id=35964931 - May 2023 (11
| comments)
|
| _The Inner Json Effect_ -
| https://news.ycombinator.com/item?id=12185727 - July 2016
| (142 comments)
| ivanjermakov wrote:
| Tom is a genius!
|
| https://thedailywtf.com/articles/the-inner-json-effect
| jancsika wrote:
| Yes!
|
| Has anyone tried implementing it? If not I'm going to give it
| a shot. :)
| masklinn wrote:
| It'd be way easier (and actually not completely insane,
| though still far from sane) in git: you'd store the
| functions in blobs instead of misusing svn commits. It'd
| probably be a lot faster too as Git is fairly good at
| giving you object content.
|
| Admittedly the GC would be an issue, but I think that would
| be fixable: instead of a `functions` key in a json, link
| all the functions in a tree object, the link that tree
| object from a commit that way you have the author for free.
| And then the class name can be a ref' in a bespoke
| namespace.
|
| Well that's the issue that it's not necessarily clear what
| commits go together to compose the system, so might
| actually be better to replace the ref' per class by a tree
| for the entire thing, with each class being a commit entry,
| and an other commit capping the entire thing. Of course git
| will not understand what's happening as commit entries in
| trees are normally for submodules but that's fine, git is
| just used as a data store here.
| hleszek wrote:
| When I started at my first company, they had a very complex VB
| application running on dozens of customers around the country,
| each having some particular needs of course. There was a LOT of
| global variables (seemingly random 4 uppercase letters)
| controlling everything.
|
| At some point, the application had some bugs which were not
| appearing when the application was run in debug mode in Visual
| Studio. The solution was obvious: installing Visual Studio for
| each customer on site and teaching the users to run the app in
| debug mode from Visual Studio. I don't know how they convinced
| the users to do this and how they managed with the license but it
| was done.
|
| What happened next was even worse.
|
| There was no version control of course, the code being available
| on a shared disk on the local network of the company with the
| code copied over in multiple folders each having its own version,
| with no particular logic to it either, V1, V2, V2.3, V2a,
| V2_customer_name, V2_customer_name_fix, ...
|
| After that, when there was a problem for a customer, the
| programmer went there to debug and modified the code on site. If
| the bug/problem was impacting other customers, we had to dispatch
| some guys for each customer to go copy/edit the code for all of
| them. But if the problem was minor, it was just modified there,
| and probably saved on the shared folder in some new folder.
|
| What happened next was to be expected: there was no consensus on
| what was the final version, each customer having slightly
| different versions, with some still having bugs fixed years
| before for others.
| fouronnes3 wrote:
| This is amazing. I can so well imagine a bright young hire
| joining that team, helpfully offering to "setup this thing
| called git" only to be laughed out of the meeting by all the
| "senior" staff.
| mleo wrote:
| Anecdote seems long before git creation, so Visual SourceSafe
| maybe. Which did not work well over a WAN. Needed other tools
| to replicate and synchronize VSS.
| masklinn wrote:
| You can remove the "over a WAN" part: VSS had been designed
| as a local VCS, so until the addition of a proper server in
| the mid aught using it over a network share was the only
| way to actually use it. And it really wasn't good.
|
| I don't know if that made it better, I assume not much, VSS
| was really trash.
| breput wrote:
| I was working at a game development company in the mid 90s
| that used Visual Source Safe and to avoid file corruption
| due to concurrent commits, we had a shiny silver baseball
| cap which you had to find and physically possess to commit.
|
| After we had off-site devs, the physical cap wouldn't work.
| So the project had a "silvercap.txt" file and you had to
| exclusively check out . And of course people forgot to
| release that and work ground to a halt.
| kolanos wrote:
| I'm sure I'm not alone in actually having lived such an
| experience.
|
| I joined a dynamic DNS provider once that had been around
| since 1999. Their tech, sadly, had not progressed much beyond
| that point. Showing the higher ups version control was like
| showing cavemen fire. Of course once the higher ups arranged
| to have training sessions led by the new hire for the entire
| dev team the VP of Engineering couldn't handle it and had me
| fired. Fun times.
| The_Colonel wrote:
| At this level of dysfunction, installing git won't do
| anything. You need a holistic change in thinking which starts
| with convincing people there's a problem.
| parpfish wrote:
| Yeah, this level of dysfunction takes years to cultivate.
|
| You need the "Dead Sea effect" to be in effect long enough
| that not only have the good people left, but for them to
| have been gone long enough that people rising into
| management have never even worked with somebody competent
| so they don't know there's a better way
| lloydatkinson wrote:
| I've yet to see people actually change their mind at this
| level. Firing is often the only way.
| djbusby wrote:
| I was one of those once. Tried to get CVS in a project.
|
| Then some other dev committed 9MB of tabs 0x09 at the end of
| a file. Then the site was "slow" (cause the homepage was
| 10MB). And the blame went to...CVS somehow.
|
| I left.
| qup wrote:
| Then they probably fired the guy who deleted 2MM lines of
| tabs for not meeting LoC metrics.
| djbusby wrote:
| Many years later there was a scene in Silicon Valley
| where their VC set a tequila bottle on the delete key and
| caused havoc.
|
| That's when I figured out what happened in 1998.
| dtech wrote:
| And what's the less opaque reference to 1998?
| djbusby wrote:
| Me, up thread, I was the junior in 1998, trying to
| introduce CVS.
| collinmanderson wrote:
| I have had a website taken down by (pretty sure) a person
| setting something on their F5 key.
| cyberax wrote:
| > And the blame went to...CVS somehow.
|
| CVS was notorious for doing "text/binary" conversions
| (CR/LF line endings to CR and vice versa), sometimes
| inappropriately. More than once, it resulted in files where
| every other line was empty. I can very well see this
| happening several times, resulting in exponential growth of
| whitespaces.
| djbusby wrote:
| Yep, that bit me/my teams more than once. But this case
| we had the commit from CVS showing it was just a fat-
| finger.
|
| But the seniors on the team pointed to CVS. Not the
| individual, not the "test" process, not the deploy
| checks.
|
| Nobody noticed it slow cause everything ran on LAN. Only
| noticed the issue on 56k
| paulddraper wrote:
| That's a really fat finger
| hleszek wrote:
| I started in 2008. This is what I did eventually. Over the
| years I introduced the small company to Linux, git, defensive
| programming, linting, continuous integration, Scrum..., but
| only for the new projects and stayed 13 years there.
|
| That old project though was never fixed though, probably
| still running that way now.
| refactor_master wrote:
| Same story here. Introduced static Python to our team's
| entire codebase. Best place I've ever worked.
| FpUser wrote:
| I think of Scrum as a crime against humanity
| llmblockchain wrote:
| I've been that person a few times.
|
| 1. The only developer on the team with Github and put forward
| the idea of the company not hosting their own source code
| with TFS.
|
| 2. The only developer using branches with git when the co-
| founder asked (demanded) everyone to only use master.
|
| The list goes on!
| malux85 wrote:
| Here's a few of my horror stories where I was a consultant
| at various companies:
|
| 1. Previous consultants left no documentation or anything,
| and a running Hadoop cluster handling (live!) 300 credit
| card transactions a second. Management hired 8 junior
| sysadmins - who were all windows sysadmins, had never used
| Linux before, and were expected to take over running this
| Linux cluster immediately. They all looked at me white as
| ghosts when I brought up SSH prompt, that's the point where
| I learned they were all windows sysadmins.
|
| 2. Another company: all Java and MySQL developers who were
| trying to use Spark on Hadoop, refusing to learn anything
| new they ended up coding a Java app that sat on a single
| node, with a mysql database on the same node, that "shelled
| out" to a single trivial hello-world type function running
| in spark, then did the rest of the computation in Java on
| the single node, management celebrated a huge success of
| their team now using "modern cluster computing" even though
| the 20 node cluster did basically nothing and was 99.99%
| idle. (And burning huge $ a month)
|
| 3. Another company: setup a cluster then was so desperate
| to use the cluster for everything installed monitoring on
| the cluster, so when the cluster went down, monitoring and
| all observability went down too
|
| 4. A Cassandra cluster run by junior sys-admins and queried
| by junior data scientists had this funny arms race where
| the data scientists did what was effectively "select * from
| *" for every query and the sysadmins noticing the cluster
| was slow, kept adding more nodes, rather than talk to each
| other things just oscillated back and forwards with costs
| spiralling out of control as more and more nodes were
| deployed
|
| Any many more!
|
| This might sound like I'm ragging on juniors a bit but
| that's definitely not the case - most of these problems
| were caused by bad management being cheap and throwing
| these poor kids into the deep end with no guidance. I did
| my best to upskill them rapidly and I'm still friends with
| many of them today, even though it's nearly 10 years later
| now,
|
| Good times!
| llmblockchain wrote:
| "Senior" holds no weight with me. I've had plenty dumb
| founding conversations with "seniors".
|
| My favorite was at the company that was self hosting
| their code. The senior team lead wanted me to help him
| find a memory leak that plagued the product for months.
| Customers were told to restart the application every few
| weeks (this was a C++ application).
|
| I sat down with the senior and looked at the code. I
| spotted the error.
|
| I was like, "You know when you do new[] you need to use
| delete[]?" as all of his deletions were without [].
|
| The look on his face was the best.
| malux85 wrote:
| Yeah it reminds me of that old saying "Some developers
| have 20 years experience and some have 1 year experience
| 20 times" haha
| addaon wrote:
| > I was like, "You know when you do new[] you need to use
| delete[]?" as all of his deletions were without [].
|
| This seems like a pretty major lack of a specific piece
| of knowledge on the senior developers part, yes, but it
| seems like a much more unforgivable miss on the part of
| the code reviewers. Was the team stuck in a rut where
| only a single person (with coincidentally the same blind
| spot) was reviewing his code, or did multiple reviewers
| somehow miss this?
| llmblockchain wrote:
| Code review? There was no code review. We did get our own
| offices though!
| Waterluvian wrote:
| No rank means much. Not even degrees like a PhD. In fact
| I swear there's negative correlation.
| capitainenemo wrote:
| Given the various accidental leaks dues to people not
| realising deletion still has a history with git when
| publishing (not to mention git has no equivalent to
| mercurial's "censor" operation), or people confusing public
| and private repos on their github accounts, or even the
| story just last week here on HN with counterintuitive
| behaviour of forking in terms of privacy
| (https://news.ycombinator.com/item?id=41060102), I can
| totally understand a company being opposed to being on
| _github_.
|
| Perhaps you should have proposed anything-but-TFS
| (git/mercurial/even svn - no need for distributed in a
| small company...
| https://www.bitquabit.com/post/unorthodocs-abandon-your-
| dvcs...) but internal.
|
| Might be a relatively easy sell if you just setup a local
| migration of their data to repository with a decent web gui
| for viewing the commits and associating with tickets, given
| my experiences with TFS in terms of slowness, and
| crashes/data corruption.
| rblatz wrote:
| There are advantages to trunk based development, not
| exactly only using master, but close enough that you could
| probably spin it to comply.
| MBCook wrote:
| A friend's company was still doing it (in SVN) as of late
| last year. Quite possibly still are. It's "what they've
| always done".
| sergiotapia wrote:
| I did this at my first, learned quick oldheads would get
| flustered and feel challenged if not eased into things a
| certain way.
|
| Ultimately by the time I left I tried to introduce redbeanphp
| (orm), git for source control, and CakePHP for some
| structure. Nothing stuck. When I left it was still raw sql
| string queries, .zip files when they remembered for backups,
| and 400,000 line php files with everything caked on there.
| kolanos wrote:
| Yes, there is a good lesson here. If you walk onto a dev
| team still using stone tools, there is likely a culture
| that distrusts of new technology, so tread lightly or you
| may be perceived as a heretic.
| fragmede wrote:
| I'm assuming this story predates git and the Internet, but
| I'm not OP. Things were different back in those days.
| donall wrote:
| I had a visceral reaction to this comment! I once joined a
| company doing ETL with Apache camel and a half dozen
| underpowered pet machines. Ingesting their entire dataset and
| running a suite of NLP models took 3-6 months (estimated; it
| was so slow nobody ever reprocessed the data to fix bugs or
| release improvements). I drew up a simple architecture using
| Kafka, hbase, and MapReduce to implement a lambda
| architecture. The CTO very patronizingly told me that just
| because something is shiny and new it doesn't mean we need to
| implement it. This was 2017 :laugh-cry:.
| blacklight wrote:
| Been there. There was this old fashioned developer in one of
| the companies I worked for a decade ago who never understood
| nor embraced version control (we were talking of SVN at the
| time, not even git). Luckily that wasn't the case for all the
| others developers in the company. But when it came to the
| projects he owned, I witnessed several scenes along the lines
| of "hey, customer X has an issue with your component Y, what
| version do they have?"
|
| He had a spreadsheet where he kept track of the versions used
| by every customer. Once identified the version, he would open
| (no joke) a drawer in his desk and pick the right USB stick
| with that version on it.
|
| I've always wondered whether this overhead was a worth price
| to pay for not wanting to learn a couple of SVN commands.
| icedchai wrote:
| My first job out of college, way back in the 90's, I had to
| "train" an entire department on how to use CVS. This wasn't
| a startup. This was a billion dollar, established company
| with 1000's of employees.
| Aloha wrote:
| He was in essence doing manual version and release control
| - just.. like you would in 1990 with floppy disks.
| kwhitefoot wrote:
| Some of these stories sound a bit far fetched, especially
| those that involve Unix systems. RCS was released in 1982 and
| CVS in 1990 so Unix systems have had version control
| available for over forty years.
| bdbdhxhdh wrote:
| There is a lag between availability and mindshare to apply
| a technology.
|
| I myself introduced SVN as versioning solution to a company
| in 2007, years after git was available I and I was aware of
| it
|
| git simply didn't have the mind share jet, thus I went with
| what I knew
| cruffle_duffle wrote:
| I can assure you they are true. Version control was still
| "controversial" in a lot of shops for quite some time.
| Plenty of places had the classic "v1, v2.3,
| v2_next_debug_cruffle_duffle" way of managing versions for
| a very, very long time.
|
| Well into the subversion era...
| ics wrote:
| I was in this position before but would point out that there
| is a tactical approach when you know that others will not
| follow. I set up a cron job (on Windows, not really cron) to
| check scan the network location for updated source files. The
| git repo was on my drive and on the corporate GitHub account,
| safe from those who should have been using it. Whenever files
| changed it would just auto commit on the main branch with the
| username included in the message. I could do whatever I
| wanted on my own branches, keep track of what others were
| doing, and essentially wield git. You don't have to try to be
| a hero inflicting proper source control upon your teams
| (their perspective) to still occasionally appear like a
| wizard to save them from inevitable, oft-occurring peril.
| gumby wrote:
| Astonishingly, It took a long time for revision control to
| become widespread.
|
| Around 1991 when Cygnus had 6-7 employees and was based in
| the apartment complex where I lived, none of the GNU codebase
| was hosted in any sort of revision control. Everything was
| FTPed around as obscurely named tarballs. We had gathered
| something like 27 different forks of gdb floating around the
| net, for example. This was back when forking was generally
| considered a tragedy, something I managed to change five or
| six years later).
|
| Rich Pixley came and said "all of our code should be in
| revision control, and I want to use this newfangled thing
| called CVS." Michael was OK with it but John and I were
| steadfastly opposed. We agreed to an experiment, grudgingly,
| subject to a whole whiteboard of absurd conditions ("must be
| transparently integrated into emacs so we don't have to know
| it's there).
|
| Pixley agreed to all of that _and then ignored all of it
| completely_. It was immediately such a win that everybody
| adopted it without complaint, including us two obstreperous
| a-holes.
|
| A few years later a preventable crisis was how revision
| control first became client-server.
| asveikau wrote:
| Linux was managed with tarballs and patch emails until a
| little over 20 years ago when they started using bitkeeper,
| then Linus wrote git.
| NetOpWibby wrote:
| What a legend
| jimbokun wrote:
| > Pixley agreed to all of that and then ignored all of it
| completely.
|
| Hahaha that's brilliant, and an important lesson for junior
| developers. Sometimes this is the best way forward. High
| risk of course. But the reward can be great.
| mewpmewp2 wrote:
| Why is it astonishingly?
|
| Git is not a natural development at all. Obviously, it is a
| standard right now.
|
| But I as a hobby coder at my teens I started out with
| FileZilla and copying over index2.php, index3.php,
| index_final.php, index_final_2.php and all of it worked
| well enough when at that point.
|
| I took a little break from that hobby, and it still took me
| a lot of time to build intuition around Git when I started
| out professionally.
|
| Obviously now that I have over a decade of professional
| experience I think it's a no brainer, but I don't think
| it's natural to understand it as it's suggested to you. I
| at least kind of had to go along with it and trust that
| it's good. It felt frustrating at first as many other
| things are. Git as a thing was FAR more difficult for me to
| understand in my 20s than PHP and how to do anything with
| it was for me at teens. Coding is raw logic, Git is about
| imagining teamwork at scale, which is a whole other thing.
|
| The funny thing is that I feel like I was overall still
| more productive when I was at my teens building stuff solo
| with FileZilla and PHP, compared to a corporate environment
| now with all the processes and 1000s of engineers.
| bsder wrote:
| > Git is not a natural development at all. Obviously, it
| is a standard right now.
|
| Git is actually an _unnatural_ development. Its UI is
| atrocious. And it worked like crap on Windows for
| forever.
|
| Over time, I taught non-computer people who used Windows
| all of CVS, Subversion, and Mercurial. They got each one
| and why things were better. The first time they had to
| recover something, they _really_ got it. Source control
| got out of their way and was generally fine.
|
| Then Github won due to VC cash dumping and foisted Git on
| all of us. Source control was no longer fine.
|
| Thankfully, there is now Jujutsu(jj) which works on top
| of the Git storage layer so you can forget about the
| craptastic UI of Git. Source control is now fine again.
| acchow wrote:
| > Git is not a natural development at all. Obviously, it
| is a standard right now.
|
| Most standard tools we use today are not obvious to
| beginners nor would they be the first thing that
| beginners reach for.
|
| But senior developers can understand the problems that
| they claim to address, and why they are important and
| common problems.
| ragebol wrote:
| Have been that person before. As an intern, and they even
| listened! In the days of SVN, just before git, so I ran a
| server in my laptop and my manager somehow decided we needed
| a big Red hat server or something, IIRC. In a 20 ppl company.
| MBCook wrote:
| I never had to deal with "we don't use source control",
| luckily.
|
| One company I joined was cleaning up the last vestiges of
| "customize it for every customer by letting people edit their
| copy on server," which predictably turned into a mess. They
| were all very small customizations to styles or images but
| enough to make upgrades a total mess.
|
| I did work at a company where despite having full source
| control they didn't actually know of they could ever deploy
| the server component again. Edits got made to the live server
| but then made again in source control, or vice versa. There
| was one more senior person who couldn't be talked out of
| their old workflow.
|
| In theory everything matched.Eventually they even checked and
| got it all under control where they were positive it was the
| same and kept it that way.
|
| But it had only ever been deployed from scratch... once. And
| for like 15 years it lived there and kept getting upgraded.
| It would all be moved when new hardware was brought in.
|
| But it wasn't installed from scratch. We truly did not know
| if we were capable of doing that. It is possible if that
| server was destroyed and we couldn't restore from a back up
| it would take us an unknown amount of time. Even though in
| theory deploying should be as simple copying the files and
| starting the web server.
|
| Were there odd configurations that had been made eight years
| ago that kept it running? Some strange permission changed
| somewhere?
|
| I wasn't on that team. But it always made me nervous. That
| was absolutely a core application of the business.
|
| I really like small shops sometimes. You get a lot of freedom
| and get to have your hands in a lot of disciplines. You learn
| a lot of things, including things that's should never be
| duplicated.
| culi wrote:
| Can I ask roughly what year this was? I can't even imagine
| something like that today
| hleszek wrote:
| 2008
| nine_k wrote:
| Visual Basic was a smash hit released in 1991.
|
| For reference, Perforce was released in 1995, Subversion in
| 2000, git in 2005. RCS existed, but only in the Unix world.
| icedchai wrote:
| Don't forget CVS, released in 1990. It was popular for
| about a decade before Subversion took over.
| nine_k wrote:
| Did it run under DOS? I'm not even asking about a win16
| native GUI client. (You can still enjoy CVS if you want
| to contribute to OpenBSD, for instance.)
| icedchai wrote:
| I'm not sure. In the mid-90's, I worked at a Solaris and
| NT shop, which was where I first used it.
| mike_hearn wrote:
| Yes, and for Win95 users there was actually a very nice
| GUI plugin to explorer called TortoiseCVS.
|
| The funny thing about CVS and Subversion is that they
| were last version control systems that non-programmers
| could actually use. TortoiseCVS/TortoiseSVN were easy
| enough that with a _lot_ of training you could get
| technical writers or artists to use it too. Valuable for
| game projects. With git, forget about it. It can 't even
| handle binary files well.
| bsder wrote:
| Mercurial and TortoiseHg were fine, too.
|
| Only Git had the brain damage to take the source control
| idea of "there are two areas: working or committed" and
| add the dumbass index/staging/stash areas that confuse
| the hell out of everything.
| dafelst wrote:
| Don't forget Visual Sourcesafe, that came out around 1995
| as well and was the standard source control package for
| most of the Windows shops in the 90s and even up to the mid
| 2000s (at least in my experience)
| bartread wrote:
| The thing is, I encountered something very similar with a
| product that had maybe 20 customers... in 2017. All of them
| had slightly different versions of the codebase. Version
| control was used, but haphazardly.
|
| You'd think this sort of thing would be a problem of the 90s
| or early 2000s, but I'd bet you there are any number of
| companies with similar situations today.
| thayne wrote:
| > There was a LOT of global variables (seemingly random 4
| uppercase letters) controlling everything.
|
| I once ran across a (c) program that had 26 variables, each one
| letter long, one for each letter of the alphabet. They were all
| global variables. And many of them were re-used for completely
| unrelated things.
| bdbdhxhdh wrote:
| wow this is just evil
|
| I want an obfuscator doing such a transformation:D
| leni536 wrote:
| Ah, the "user-defined registers" paradigm.
| sghiassy wrote:
| This is nightmare-fuel. How does this happen?
|
| As a glib answer, can I suggest, that without proper training,
| there were a lot of developers who had never trained under
| anyone or any company with proper practices??
|
| Honest question. How does our profession root out intrinsically
| obvious bad practices?
| lazide wrote:
| It happens because it's easier - until it's impossible,
| anyway.
|
| The training and best practices you're talking about is
| learned experience about how to avoid it getting impossible.
| But that almost always involves expense that the business
| side considers 'stupid'.
| insane_dreamer wrote:
| How long ago was this?
| lowdownbutter wrote:
| My first role was with a company who had hit a limit for VB6
| variable names iirc. So they'd all been renamed to shorter
| names. This may be the same issue. They were in the process of
| rewriting in VB.net.
| matheusmoreira wrote:
| > with some still having bugs fixed years before for others
|
| Hopefully they won't discover at some point that customers were
| in fact _depending_ on those bugs. That 's poison for the
| programmer's soul.
| osigurdson wrote:
| There is "do things that don't scale" and the there is bending
| over backward to do things in the dumbest possible way. Not
| sure where this lands.
| TacticalCoder wrote:
| > At some point, the application had some bugs which were not
| appearing when the application was run in debug mode in Visual
| Studio. The solution was obvious: installing Visual Studio for
| each customer on site and teaching the users to run the app in
| debug mode from Visual Studio.
|
| Holy smoke! That's actually the most creative solution
| (horrible, but creative) I've ever heard to fix a Heisenbug:
|
| https://en.wikipedia.org/wiki/Heisenbug
| tbm57 wrote:
| Working on something like that would drive me absolutely batty. I
| am happy you were able to find your zen in the middle of that
| chaos. This post truly speaks to the human condition
| jbkkd wrote:
| Sounds awfully like my first job, with the addition of not having
| _any_ sort of test - functional, integration or unit. Nothing.
|
| A few months in, when I approached the CTO and asked if I could
| start writing a test framework, he deemed it a waste of time and
| said "by the time you'd commit the test, it would go out of date
| and you'd need to rewrite it".
|
| Naturally, the build would break about 5 times a week.
|
| Boeing was a major customer of this system, so when shit hit the
| fan at Boeing a while ago, I wasn't surprised.
| nikodunk wrote:
| What a beautiful, humorously written, bitter-sweet piece of
| writing!
| agentultra wrote:
| The ending is pure gold. Some of the best times in my career were
| working on a codebase for an application serving folks I knew on
| a first name basis and had had lunch with.
|
| I could talk through pain points they were having, we'd come up
| with a solution together, I'd hack up a quick prototype and
| launch it just to them to try out. We'd tweak it over a couple of
| weeks and when it was good I'd launch it to all customers.
|
| Messy, ugly code base. But it worked well because it wasn't over-
| managed. Just developers doing development. Amazing what happens
| when you get out of the way of smart, talented people and let
| them do their work.
| rochak wrote:
| [Deleted]
| failbuffer wrote:
| ... _and_ get them talking directly to the user. I feel that 's
| where the real magic happens.
| deathanatos wrote:
| Two early databases I worked on.
|
| The first contained monetary values. These were split over two
| columns, a decimal column holding the magnitude of the value, and
| a string column, containing an ISO currency code. Sounds good so
| far, right? Well, I learned _much_ later (after, of course,
| having relied on the data) that the currency code column had only
| been added after expanding into Europe ... but _not_ before
| expanding into Canada. So when it had been added, there had been
| mixed USD /CAD values, but no currency code column to distinguish
| them. But when the column was added, they just defaulted it all
| to USD. So and USD value _could_ be CAD -- you "just" needed to
| parse the address column to find out.
|
| Another one was a pair of Postgres DBs. To provide "redundancy"
| in case of an outage, there were _two_ such databases. But no
| sort of Postgres replication strategy was used between them,
| rather, IIRC, the client did the replication. There was no formal
| specification of the consensus logic -- if it could even be said
| to have such logic; I think it was just "try both, hope for the
| best". Effectively, this is a rather poorly described multi-
| master setup. They'd noticed some of the values hadn't replicated
| properly, and wanted to know how bad it was; could I find places
| where the databases disagreed?
|
| I didn't know the term "split brain" at the time (that would have
| helped!), but that's what this setup was in. What made pairing
| data worse is that, while any column containing text was a
| varchar, IIRC the character set of the database was just
| "latin1". The client ran on Windows, and it was just shipping the
| values from the Windows API "A" functions directly to the
| database. So Windows has two sets of APIs for like ... everything
| with a string, an "A" version, and a "W" version. "W" is supposed
| to be Unicode1, but "A" is "the computer's locale", which is
| nearly never latin1. Worse, the company had some usage on
| machines that were set to like, the Russian locale is, or the
| Greek locale. So every string value in the database was,
| effectively, _in a different character set_ , and nowhere was it
| specified _which_. The assumption is the same bytes would always
| get shipped back to the same client, or something? It wasn 't
| always the case, and if you opened a client and poked around
| enough, you'd find mojibake easily enough. Now remember we're
| trying to find mismatched/unreplicated rows? Some rows were
| mismatched _in character encoding only_ : the values on the two
| DBs were technically the same, just encoded differently. (Their
| machines' Python setup was also broken, because Python was
| ridiculously out of date. I'm talking 2.x where the x was too
| old, this was before the problems of Python 3 were relevant.
| Everything in the company was C++, so this didn't matter much to
| the older hands there, but ... god a working Python would have
| made working with character set issues so much easier.)
|
| 1IIRC, it's best described as "nearly UTF-16"
| vertis wrote:
| Reminds me of a date column where half the dates were
| MM/DD/YYYY and the rest DD/MM/YYYY. I did manage to eventually
| find the point in time when it changed and normalise the
| database. For those wondering about DD/MM, Australia.
| tsujamin wrote:
| And most countries :)
|
| https://en.wikipedia.org/wiki/List_of_date_formats_by_countr.
| ..
| jacobyoder wrote:
| > But when the column was added, they just defaulted it all to
| USD. So and USD value could be CAD -- you "just" needed to
| parse the address column to find out.
|
| I bet no one even considered the idea that someone in Canada
| might pay in USD dollars.
|
| I had to... discuss... with someone the possibility that if we
| were recording money received from a bank, we might do well to
| record the currency type as well, because... banks can deal
| with multiple currencies. "No, banks don't do that. That's not
| possible". This US company was getting ready to expand their
| services over to Europe, and I couldn't understand why no one
| could understand that this might be a necessary item to record.
|
| Someone using this system in France to help finance a project
| in New York, for example, might want to know whether Euros or
| USD were the subject of the project. This was sort of a PM tool
| to note the cost of a project and how much was left in a budget
| - it wasn't moving money directly. We had a system to choose
| specific banks and note that 'person X moved Y currency units
| to the project' but... no ability to select the currency units.
| If it was from a UK bank, it would be GBP. A US bank, USD, etc.
| I was voted down as not understanding how banks worked.
|
| Months later, a demo was presented right before launch to the
| people who were financing the project, and they asked how
| someone would know the currency being used for each project.
| Back came a work ticket "high priority" because we were already
| late for launch and multiple screens now had to accommodate
| this "new" requirement.
|
| I know at least some of this is how I present info, but...
| being 'nice', being 'inquisitive', raising issues up the chain
| of command, etc.. rarely work. These sorts of extremely obvious
| things get ignored until the actual end users bring up the same
| concern. Somehow I wasn't supposed to know this sort of info
| because "you're a developer, you're not some international
| banker". Insane...
| throwaway93982 wrote:
| When it comes to building things, the outcome is dictated by the
| constraints. But not the way most people realize.
|
| Constraints affect outcomes in (at least) three ways:
| - by absolute limitation (you cannot violate these) - by
| path of least resistance (the constraint makes X easier) -
| by strategy (creative use of knowledge & skills leads to novel
| solutions)
|
| ---
|
| There is an injured person on top of a mountain, and they need
| medical attention. You need to go up the mountain, get them, and
| bring them down.
|
| You only have so much strength/endurance, so nothing you do will
| ever be faster than what your body is capable of. You need to get
| up and down in less than two hours. You need to carry an injured
| person. The mountain has two slopes: a long gradual one, and a
| short steep one.
|
| Most people would climb the long gradual slope, because it's the
| path of least resistance. But it will take 4x as long to climb up
| and down it. Climbing straight up the steep slope would be
| incredibly tiring, and unsafe to bring someone down.
|
| You can, however, zig-zag up and down the steep hill. It will
| take more time than going straight up, but faster than the long
| way, you will be less tired, and it's safer to bring someone down
| hill.
|
| ---
|
| Constraints can be good and bad. Good use of constraints can
| allow you to get something done effectively. Bad use of
| constraints leads to failure. So it's important to have someone
| who can utilize those constraints effectively.
|
| Vision, leadership, and wisdom is more important than the people,
| skills, materials, or time involved. The former determines the
| quality of the outcome more than the latter.
| crngefest wrote:
| So to summarise: ,,It depends"
| RadiozRadioz wrote:
| I built a system as horrible as this. One of my many terrible
| inventions was as follows:
|
| Originally our company only did business in one marketplace (the
| UK). When we started doing business in multiple marketplaces, it
| came time to modify the system to cope with more than one. Our
| system assumed, _everywhere_, that there was only ever one
| marketplace. I had a plan: I would copy-paste the system, make an
| "international" version that supported many marketplaces, then
| transition over our original marketplace to the international
| version and shut down the old system. This way, everyone could
| keep working on the original marketplace like normal, they'd get
| the new marketplaces on the new system, and we'd do a clean
| cutover once ready.
|
| It started out quite well. I got the international version
| working and onboarded our first new marketplace. The business was
| very happy, there was lots of opportunity in the new
| marketplaces. They asked that I delay the cutover from the old
| system and focus on developing things for these new marketplaces.
| After all, the old system still works for our original
| marketplace, we can move it over once our work is done on the new
| marketplaces. I said yes, of course!
|
| It's now 5 years later, it turns out there's a lot to do in those
| other marketplaces. To say that the system is split-brained is an
| understatement. When training new hires, we teach them the
| difference between a "product" and an "international product".
| When either system does something, it double checks and cross-
| references with the other system via a colourful assortment of
| HTTP APIs. There are a variety of database tables that we "froze"
| in the old system and continued in the new system, so you could
| tell that an ID referred to something in the new system or the
| old system if it was above or below the magic number we froze it
| at. We have overarching BI dashboards that query both systems to
| produce results, and they have to heavily manipulate the old
| system's data to fit with the new multi-marketplace model, and
| any other changes we made. Both systems are extremely tightly
| coupled, the old one is a ball and chain to the new one, but the
| new one is so tightly integrated into it that there's no hope of
| separating them now.
|
| I've learned to say no since then.
| collinmanderson wrote:
| I'm seeing a lot of comments about terrible code bases, but I do
| think there's something really beautiful here, something like
| "worse is better":
|
| > This may sound like a mess to you. But it was remarkably
| enjoyable to work in. Gone were the concerns of code duplication.
| Gone were the concerns of consistency. Gone were the concerns of
| extensibility. Code was written to serve a use, to touch as
| little of the area around it as possible, and to be easily
| replaceable. Our code was decoupled, because coupling it was
| simply harder.
| MBCook wrote:
| This reminds me of my first job at a very small shop. Here's two
| stories:
|
| The calendar table struck a chord. We had one for holidays. One
| system needed to know when they were to calculate pay dates.
| Except once a year it would "run out" and someone would have to
| go add in the next year's worth after a bad calculation was
| spotted.
|
| The second time I was told to do it, I put in 10 years worth. The
| company didn't survive long enough to need more.
|
| My first "big" project was actually that pay date code. Every
| once in a while the server would hang, and people had deduced the
| date calculation was the problem. But no one knew why. And it
| wasn't frequent enough to be worth taking time from the other two
| programmers. But I was new and thus "spare".
|
| After not being able to find any errors, I brute forced it. I ran
| that calculation for every day of the year for every pay type
| (weekly, monthly, bi-monthly, etc) and it quickly got locked in
| an infinite loop. Armed with the "bad" data that triggered it, it
| was easy to solve and provide tests to prove it would never
| happen again. I don't remember what it was, exactly.
| thayne wrote:
| I've seen something very similar to the Sequence key table. But
| there wasn't just one of them, it was a common pattern. And it
| had 2 rows and 2 columns.
|
| The reason we had it is we had master-master replication, but
| without requiring the peer to acknowledge befor committing a
| transaction. To avoid inconsistencies, we preferred one server
| for even ids and the other for odd ids. But when writing a new
| record, an autogenerating sequence would just give the next id
| without regard to what "side" the request was on. So we had a
| table to keep track of the next id to use for each side, where we
| incremented the next id by 2 each time a new id was allocated.
|
| It was a little weird, but it worked fairly well. Although we
| eventually changed the architecture and removed the need for
| those tables.
| codeptualize wrote:
| What an amazing read, funny, nostalgic, the duality of the
| perfect mess but still so much opportunity and ability to make
| progress, and somehow it all just chugging along.
|
| I feel a lot of this is the difference between theory and
| practice. Sure each of these things are bad, but probably a lot
| might have been the right choice at the time, and in a way, most
| companies, even most projects running for many years, end with
| similar quirky processes, messes, and hacks.
|
| It's the war stories of what used to be, often only told by the
| code and the database as the creators have long left.
|
| When I look at our couple year old startup, we have some of these
| things. Prototypes that just keep functioning and only get more
| attention when they break, manual processed that sort of make
| sense but also don't, integrations with systems that were build
| in the early 2000's (it works, but it ain't pretty). We fix many,
| but I'm sure some will survive way too long.
|
| As software engineers, especially online, we like to discuss the
| ideal way to do things, best practices, testing strategies,
| redundancies, the whole nine yards. When time is of the essence
| and stakes are high, it all goes out of the window and you gotta
| just make it work in whatever way is possible to fight another
| day.
| compiler-guy wrote:
| It's hard for those who came into the discipline in the past
| twenty years to realize just how much things have changed around
| version control and building.
|
| Joel Spolsky released the "Joel Test" for determining if the
| software team you were interviewing had good practices in _2000_.
| One of the requirements was that they used version control. Not
| all that many teams actually did. Especially during the dotcom
| craze.
|
| Today, passing the Joel Test is table stakes, and it's the rare
| shop that doesn't. But it took years for that sort of thing to
| become ubiquitous.
|
| https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
| khalilravanna wrote:
| This reminds me of my first software job, an internship in
| college in 2012 building an application from scratch to
| facilitate evaluating teachers. It was basically an app to
| allow people to create a form for their school's evaluation
| criteria and then submit those forms. Sounds super
| straightforward, right? It was. The catch was our team was 2 CS
| undergrad, a masters CS student, and a high school student. All
| with no professional experience. We knew _nothing_. Well kind
| of but more on that in a second. Our manager was absolutely non
| technical. In fact they were the second highest person in the
| company (fairly small company) and were managing our project
| and a bunch of other stuff at the company. And somehow with
| almost 0 oversight we built a functional Django application
| that the business was able to sell and make money from. My
| favorite highlights were 1) the codebase was initially shared
| over FTP ("Hey you're not editing file X, right? Oh you are? Ah
| woops I just overwrote all your changes.") till someone
| intelligently suggested "Uhhh Git?" 2) the actual best
| programmer amongst us was the high schooler. They suggested
| Django, picked the DB, they suggested using Celery to speed up
| async work, Redis for caching, and yes, "Uhh Git?" In
| retrospect the only reason we succeeded was because of them.
| They were like top 5 on the stack overflow Code Golf site IIRC.
| 3) My interview was basically showing my aforementioned manager
| who had never coded in his life a project I worked on at school
| and him being like "Yeah looks good. You're hired."
|
| With 10 years of hindsight, I cringe thinking back to all the
| bad decisions I pushed for and the no-doubt terrible code I
| wrote. But I also marvel and look back fondly at being given a
| shot and being in an environment where I could just build
| something from the ground up and learn everything soup to nuts
| on the job. God bless whoever inherited that codebase.
| grishka wrote:
| The worst codebase I've ever worked on was the Telegram client
| for Android. I mean just look at this file, it's so huge GitHub
| gives up rendering it:
| https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/...
|
| Or this one, it implements EVERYTHING for rendering and
| interacting with a message in a single class, all non-service
| message types, all drawn manually on a canvas "for performance",
| and all input processed manually too:
| https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/...
|
| I even had the permission from the main developer to refactor
| some of it, but I never got around to actually doing it.
|
| It's a miracle this app works at all.
| bartekpacia wrote:
| It's even more crazy because Telegram is by far the smoothest
| messenger app on Android.
|
| BTW, would you mind sharing how you got that job?
| Klonoar wrote:
| They didn't specify it was their job, the project is open
| source after all.
| grishka wrote:
| > BTW, would you mind sharing how you got that job?
|
| I started working with Pavel Durov in 2011 on the VKontakte
| Android app, after winning several of his developer contests,
| and it all just went from there. I was the last one of the
| "old" team to transfer to Telegram. My main job was actually
| the libtgvoip[1] library and its integration into the Android
| app (the other platforms were done by their corresponding
| devs).
|
| I got fired in 2019 because the calls weren't working
| satisfactorily for Pavel. He got furious when he was unable
| to call some journalist, cursed at me in front of the entire
| team and sent me a termination notice. In hindsight, I
| should've definitely not taken building a VoIP library from
| scratch alone upon myself. It's a massive undertaking. But at
| least I learned a lot, and many people say that my
| implementation performed noticeably better than the current
| WebRTC-based one -\\_(tsu)_/-
|
| I'm now working on two fediverse projects at the same time --
| Smithereen[2], a VK-like ActivityPub server, and the Mastodon
| Android app[3].
|
| [1] https://github.com/grishka/libtgvoip
|
| [2] https://github.com/grishka/Smithereen
|
| [3] https://github.com/mastodon/mastodon-android
| hyperbolablabla wrote:
| Telegram happens to be one of the smoothest app experiences on
| Android, both in terms of perf and UX, so they must be doing
| something right.
| xpressvideoz wrote:
| Elasticsearch and OpenSearch have a similar issue. They have a
| soft limit on the number of "searchable" fields (called
| "mapping") a table can have (by the way, a table is called an
| "index" in their terminology. How confusing!), which is 1000. We
| ran into this problem because we tried to jam all the logs in
| every single microservice into a single table. Each microservice
| had a different log format, so after combining them all the
| number of different fields surged. We did it this way because the
| operations team wanted to maintain a single table, by the name of
| simplicity, which to this day is a reason I can't completely
| fathom. We were surprised because we thought Elasticsearch and
| OpenSearch were some kind of magic box that can somehow ingest
| all the data we put yet still are perfectly performant. After
| that incident, we introduced a common log format that applies to
| every microservice.
| ddgflorida wrote:
| In one of my early career jobs, we had applications that were so
| buggy we had 1 or 2 developers dedicated to logging into
| customers databases and fixing data.
| peteforde wrote:
| Man, this post is gold. I think we all have our own version of
| the best worst codebase.
|
| My take on this story was a backend "deal capture" system for an
| emissions trading desk about 25 years ago. The application was
| written in classic ASP, T-SQL and VB6 COM components. Oh, and
| VBScript. It only ran in Internet Explorer 6.
|
| The heart of the machine was a massive, unholy stored procedure.
| It was about 35kb, and what it did was generate a very long
| string which was - you guessed it - a gigantic SQL statement with
| countless JOINs across many temporary tables.
|
| The business logic was deemed sophisticated enough that it was
| decided we needed a workflow engine and some genius decided that
| we should use an expensive proprietary enterprise-y tool which
| used Microsoft Exchange Server as its datastore and operating
| environment. This was as terrible an idea as it sounds, because
| the "API" to talk to this engine was single-threaded and
| massively bottlenecked by whatever made Exchange Server suck.
| Most significantly, it also could only be properly accessed by a
| very specific version of a DLL, requiring that a server setup
| follow an agonizingly specific software installation procedure
| that was scientifically proven to be impossible for IT to execute
| because they would always jump to the end and install the latest
| version of the service packs and suddenly the DLL we needed would
| be impossible to access without wiping the machine and starting
| over. This is what people meant when you hear the term "DLL
| Hell".
|
| The most interesting aspect of this system was the client UX.
| This was 1999, and Ajax was literally not a thing yet. However,
| the precursor to XmlHttpRequest was an IE-only thing called
| ActiveXObject. You could use VBScript to wire up form elements to
| be updated by an ActiveXObject in the way we'd consider familiar
| today, except that instead of FORM GET/POST, your ActiveXObject
| would talk to a VB6 COM+ DLL running inside of Microsoft
| Transaction Server. Looking back on it now, the whole thing was
| simultaneously an abomination and years ahead of its time. The
| security profile would likely give a modern developer night
| terrors; I'm pretty sure that if you were on a page that was
| being served by an IIS instance that had COM+ registered, your
| ActiveXObject could call methods on that object so long as the
| VBScript had the UUID that identified the COM+ object. End of
| story. Just wow.
|
| Final detail that my memory retains is that the room full of
| chain smoking women who operated this system did not use mice.
| That is, they were entering data almost exclusively through the
| number pad and furiously slamming the tab key to move to the next
| field. They would develop intense muscle memory for the 200-300
| fields on this form. They needed repeatability or it would
| literally break them. Thing is, there were many instances where
| they didn't want to fill fields in the order they are defined in
| the HTML, so over time about 90% of those form elements gained
| in-line event handlers, again, written in VBScript, which
| replaced the built-in tab key handling functionality of the
| element with whatever was deemed the correct next step in their
| insanely specific process. If we accidentally broke their
| expectation, there would be hell to pay.
|
| That is, we would login to the live server with VNC and live edit
| the changes into the codebase while they were on the phone over
| the remote screen share connection.
|
| There was no source control, and if there were backups, I'm
| dubious that they would have been restorable.
| prudentpomelo wrote:
| It's honestly relieving to read all these stories because I feel
| like I am in the middle of this right now. Our legacy product is
| a mishmash of perl and php that was probably started in the early
| 2000s. Which I wouldn't feel bad about supporting if it wasn't
| for the new system that has been bolted on top of it. The
| previous team that started the migration is all gone and now we
| are halfway between two two systems that we have to support. On
| top of that my manager is afraid to say no to our eccentric CEO
| so we are flooded with new feature requests but can never get
| anything done because we spend so much time chasing down problems
| in the old system. Is there any sane way out of this? I just feel
| like I am white-knuckling it through every day.
| dave7 wrote:
| For online Poker, one of the current database tools runs in to
| this issue - PokerTracker 4 (PT4).
|
| These tracker databases are usually used to generate a HUD -
| numerical readouts surrounding each player shown on the table. PT
| allows for custom HUDs to be saved/exported/shared, and there is
| a cottage industry building and selling such HUDS. These HUDs can
| often bundle hundreds of custom stats - basically a subset of SQL
| queries, a simple example would be "times_raised /
| opportunities_to_raise WHERE position= 'button'", that sort of
| thing.
|
| For performance reasons these custom stats are cached, which
| obviously makes some sense. However, each cached stat creates a
| new column in the custom_cache table of PT4's current database, a
| PostgreSQL 9.0 backend. If you play mostly Heads-Up SNG or the
| 3-handed Spins, it's actually quite easy to go over the (IIRC)
| 4096 column limit there by purchasing and importing one too many
| fancy HU HUD packages!
|
| This completely borks PT4, it can no longer open - and players
| can no longer make money! In my previous work, I've supported
| many a player and fixed this "too many columns" error numerous
| times (by deleting enough custom HUDs from their account until
| the PT4 can start up correctly once more). Funny to see this pop
| up elsewhere!
| sanktanglia wrote:
| Im currently deep in rewriting a c# monolith thats 10+ years old
| that has thousands of lines of extra code that i was able to
| throw away because most of it was written before there were
| optional arguments so they made overloads for every permutation
| of arguments for every framework function
| gouggoug wrote:
| I worked many years with an open source eCommerce platform called
| Magento[0] which, at the time, used something called the "Entity
| Attribute Value" (or EAV) system[1].
|
| One particularity of the EAV system is that you end up having
| tables with hundreds (and growing) of columns. It made Magento
| itself extremely hard to work with and optimize. I hope they
| moved away from this model since.
|
| To be fair, this was before nosql databases were a thing.
|
| [0]: https://en.wikipedia.org/wiki/Magento
|
| [1]:
| https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
| jabart wrote:
| This codebase sold for $4.3 billion. That table was cursed.
| alexpotato wrote:
| And weird code can lead to crazy outages.
|
| I put together a thread of all of the wild incidents I've seen
| over my many years of working as a FinTech SRE:
|
| https://x.com/alexpotato/status/1215876962809339904
___________________________________________________________________
(page generated 2024-08-03 23:00 UTC)