[HN Gopher] Get me out of data hell
___________________________________________________________________
Get me out of data hell
Author : pavel_lishin
Score : 583 points
Date : 2024-10-31 19:01 UTC (3 days ago)
(HTM) web link (ludic.mataroa.blog)
(TXT) w3m dump (ludic.mataroa.blog)
| jaygreco wrote:
| I really like the author's writing style here. The quips about
| the tea especially.
| remram wrote:
| It reminds me of the classic series on /r/talesfromtechsupport
| by airz23. It's uncanny.
|
| They would almost always feature a "I sip my coffee. It tastes
| like <feelings at the time>" Example:
| https://www.reddit.com/r/talesfromtechsupport/comments/203qx...
| alex_suzuki wrote:
| Same. For me the association was the Bastard Operator From
| Hell, with a darker, more cynical view of life. Just very
| enjoyable reading.
| ludicity wrote:
| Very much appreciate the kind words! From you and the other
| commenters too.
| ctippett wrote:
| > of course, we're serverless, because how can you hurt yourself
| without a cutting-edge?
|
| A beautiful epigram.
| storafrid wrote:
| Yes, but I'm surprised that they attribute "cutting-edge" to
| Lambda. It's about as old as Docker.
| zdragnar wrote:
| I was part of a company that went all in on using lambdas for
| the majority of their web facing APIs. That was 7 years ago.
|
| The cutting edge bit is a nice quip, though I agree not
| exactly accurate anymore.
| stackskipton wrote:
| I worked for a company that went all in on Lambda as well.
| The knots they had to twist themselves into so that
| everything ran nice and smooth in Lambda environment was
| mindboggling. We have certain actions like orders that
| would pass through 8 Lambdas before completion because of
| execution time or just the big code base would result in 7
| seconds start up time (node) so it would get broken down.
| If any of them failed, and it felt like failed a ton due to
| Amazon backend stuff, it was a nightmare to resolve.
|
| All of it could probably been handled by larger node
| application in docker container somewhere but AUTO SCALING,
| FAILOVER, SERVERLESS!
|
| Once I started as SRE for a new team, we built a larger
| monolith using Node and docker on EC2. We would get massive
| complements for our uptime and reliability but there were
| some architects extremely unhappy when I revealed in
| division presentation that it was just Docker + m4.xlarge
| running Ubuntu 18.04. When I left, more and more Lambdas
| were being broken down into docker running on EC2. They are
| probably on some container managed solution now.
| sph wrote:
| It sounds like you like to deal with much sharper edges than
| I am comfortable with.
|
| Or maybe I am too old with this shit. Still haven't found a
| use for "serverless" knives.
| hinkley wrote:
| For a bank, Lambda is brand new.
|
| You haven't worked in conservative industries I take it? Late
| adopters, every one.
|
| OP is still trying to replace Cobol. I know an insurance
| company that started that process 15 years ago. Fifteen
| years.
| Aeolun wrote:
| It's what everyone runs to when server based stuff would save
| them so much pain.
| the_af wrote:
| On one hand, yes. Beautiful.
|
| But on the other, the same sentence could be written about
| software deployed to traditional servers. "Because of course,
| how can you hurt yourself without the joys of badly configured
| servers?".
| skydhash wrote:
| You can hurt yourself with a badly held butter knife, and you
| can hurt yourself juggling katanas. Which situation would get
| people saying you're crazy?
| zdragnar wrote:
| Well, if you narrow down the metaphor to just knives, a
| dull knife is more dangerous to a chef than a sharp knife,
| because you need to apply more pressure and you get less
| control over the cutting action.
| downut wrote:
| The people juggling aren't chefs.
| zdragnar wrote:
| Clearly, you haven't seen a good chef at work.
| hinkley wrote:
| Yes, but which one will people say you're _crazy_?
| hackable_sand wrote:
| It's a pun.
| layoric wrote:
| Came here to praise the same sentence, well done author, gave
| me a good laugh!
| salt-thrower wrote:
| Beautifully written and fun to read. Blog posts like this give me
| a boost of mental strength to keep going during my worst episodes
| of burnout.
| reverius42 wrote:
| > I've even degraded team morale because I've convinced some of
| the engineers that things should be better, but not management,
| so now some of the engineers are upset.
|
| Oof, that hits a little close to home.
| ludicity wrote:
| The good news is that, since the others are also looking for
| work elsewhere, there will be more engineers out in gen pop
| that actually thinks tests are useful, hah.
| hinkley wrote:
| I have this illusion in my head that I stayed so long at my
| last company that almost all of my favorite people left, but
| one of my coworkers had my number.
|
| After a person I liaised with on another team left, I asked his
| superior if there was someone else I should build bridges with.
| We started talking about one of the team members and he said,
| "I don't want you to talk to him. We like him, and if you talk
| to him he'll leave."
|
| This was on Slack so I don't know if this was a jest or he was
| serious/mad. But it's entirely true. I've convinced at least
| half a dozen people that we should expect better from a team
| environment and ourselves, and that this org (not the whole
| company, just this division) is a cult of stupidity.
|
| I was trying to recruit collaborators to fix the bullshit but
| apparently they decided it would be much easier to just start
| over.
| refulgentis wrote:
| There's some finer points to parse in your comment that I'm
| not 100% on (ex. if "this org" is your division or the
| partner division), so I'm out on a ledge a little bit here,
| might not relate to what you meant.
|
| I was lucky enough to get ~6 years running my own tech
| company after 6 years as a waiter. Then I sold it, yadda
| yadda, went to Google 6 months later, got ~7 years in there.
|
| It really, really, really, disturbed me how approximately
| every situation, in every division, with any people, ended up
| being boiling down to "how do we muddle through one more day
| without challenging anyones preconceptions", 95% of the time
| it was tribal antisocial stuff, and no one would speak up
| about it.
|
| Direct example, for posterity.
|
| I don't wanna speak too directly to it, so lets imagine
| Google Division A (hereafter, dApps).
|
| New division lead (ex-dApps) joins dBytes with apparent bias
| against partnering division (dConsumer). Despite the project
| being previously framed as top priority, new lead
| consistently undermines dConsumer in meetings and shows
| little interest in understanding their work. Team adopts
| leader's negative attitude, becoming obstructive and
| uncooperative. I ended up carrying a critical launch,
| virtually alone, for 6 months. At performance review time, my
| boss questions why I didn't get more team involvement -
| despite the hostile environment that prevented exactly that -
| and speaks glowingly about how we need to support peer going
| for promotion based on their excellent job on part Y...which
| they didn't do. They spent 2 days on it then said it was
| impossible. And they were definitively the most cooperative
| because at least they tried, and wouldn't actively be
| aggressive in meetings with the outgroup.
|
| Everything, always, came down to: A) don't cause conflict at
| all, at home, or you will be buried B) we'll bend over
| backwards to accomodate conflict you invent, as long as we
| can clearly define them as an out-group with 0 ability to
| affect us day to day.
| hinkley wrote:
| At prior jobs we had an escape hatch for this: go to a
| fancy coffee shop with the dissenters and have all of our
| bitchfests out of earshot of the muggles.
|
| But it's trickier to coax people still on the fence to come
| out for multiple coffees.
| SynasterBeiter wrote:
| I hate whining like these. Just do your job and get over it. No
| need to be theatrical about it.
| incognito124 wrote:
| A creative outlet is a need, to some. You sound like OP's
| employees.
| baq wrote:
| Let the man _live_ will you. Not everyone wants to be a drone
| for hire.
| hobs wrote:
| If this was a job with a non-abstract input and output process
| they would have OSHA saying it was genuinely unsafe (and
| definitely stupid.)
|
| Many of us build systems to manage automatic actions to take
| care of this stuff, if I had an engineer I had to take 100+
| steps to get something done I would definitely be considering
| 1) What the hell am I paying for and 2) Why the hell am I
| paying for it?
| hprotagonist wrote:
| you know, he had a post about exactly this comment not too long
| ago!
|
| edit: https://ludic.mataroa.blog/blog/the-violent-role-of-
| relentle...
| tgv wrote:
| That's pretty good. I'm somewhat curious now about his
| writing aspirations.
| ludicity wrote:
| I was offered a book deal and turned it down, hah. I read
| in spectacular volumes and books have a very special place
| in my heart. Growing up in Malaysia, most of my English was
| initially mastered from a gigantic pile of Enid Blyton
| books my family had lying around after the British
| occupation. We even give everyone at Hermit Tech a day off
| per week to study non-IT things and let the subconscious do
| some processing on work. Not because I'm a weirdo (though I
| am), but because I genuinely believe this produces a
| higher-quality experience for our clients, and no one can
| stop me from testing things like "five day work weeks are
| too long".
|
| In any case, the book deal had constraints like "no
| swearing", and it was implied they'd find their own artist
| for the cover. I didn't even intend to swear, but I care
| too much to let them assign an editor and impose arbitrary
| constraints. I ended up chatting a bit with Ed Zitron after
| the AI rant article went super viral, and he told me that I
| have the audience to just publish my own books.
|
| So I'm doing just that! I'm starting with ten short
| stories, at the advice of someone in the local Melbourne
| scene that has helped many people publish. Then I'm aiming
| to write one book containing a series of essays on IT work,
| and one fantasy/fiction book. I'm a reasonably good judge
| of popularity, and because neither will be particularly
| angry, I will probably only sell a few hundred copies. But
| I'll have a book with my name on it, which is very special
| to me.
| hipadev23 wrote:
| Complacency is how this shit happens.
| dangerwill wrote:
| Quick question: How many years of industry exp do you have? I
| thought the same way until this year when the burnout got to me
| too. I thought I was too much of a high achiever to get burnout
| and yet I'm in the same boat as the author.
|
| Also, once the people who speak up about a problem leave, all
| you are left with are idiot yes-men in management, old timers
| doing their job as minimally as possible to not to be noticed
| by management, and fresh new engineering grads ready to be
| grist for the mill. When those sorts of people are writing all
| of the code around you, no matter how good you are, you will be
| driven insane.
| brianhorakh wrote:
| Wonderful prose. I am in Melbourne also. I possibly used to work
| at the same place but I'm not sure.
|
| I resigned due to the night terrors caused by the cyber security
| issues I saw everywhere. The more I explored and understood the
| more sleep I lost.
| ludicity wrote:
| Hah, yes, I'm not in cybersecurity but am very close to a few
| people that are. The incompetence is not evenly distributed and
| not as bad as it is in data, but some companies are in terrible
| states, and the stakes are much higher.
| hinkley wrote:
| I worked with an Aussie who was in the US on H1B because the
| aerospace industry was even worse than the status quo in
| Australia. Last I heard he went back. I sincerely hope he
| switched industry verticals.
| zombiwoof wrote:
| Data "engineering" is where all the cool kids go with no clue and
| create insane architectures to justify their incompetence
| hobs wrote:
| As a data engineer I have seen absolutely bullshit pass for
| production, but it doesn't seem that different from all the
| other bullshit I have seen people deploy in my life.
|
| It is one of the few types of jobs I have worked were someone
| credulously offering adding five more layers to fix an issue
| with latency is a normal operating procedure though.
| parpfish wrote:
| data engineer gives data science a run for their money when it
| comes to ambiguous job expectations.
|
| I've seen it mean anything from "distributed computing expert"
| to "knows SQL"
| marcosdumay wrote:
| > to "knows SQL"
|
| There are still ways to go. I've seen it means "spends the
| days filling excel spreadsheets".
| sgarland wrote:
| > knows SQL
|
| If by that you mean, "knows the commands to create, fill, and
| select from a table," then yes. If you mean, "knows how to
| create a performant schema and queries that will serve them
| well into the future," then no, absolutely not.
|
| OTOH, IME data folk are much more cheerful and willing to
| change things than devs when I point out the innumerable ways
| their DB choices are choking them. Devs more often fall on
| the side of "we don't have time for that on our roadmap;
| can't you just fix it?"
| the_af wrote:
| You can replace "data engineering" with "software engineering"
| in that sentence and it will still hold true...
| fifilura wrote:
| Or chef or carpenter or automobile builder or priest or...
| icedchai wrote:
| I've seen some "data engineering" scripts that were complete
| messes and beyond crazy. Some examples: Massively over
| engineered "pipelines" that process a few hundred rows a day,
| but somehow manage to take forever to run. Developers that
| didn't know SQL beyond "select * from table", so they do all
| their summarization in Python. Or, worse, I've seen a Python
| script calling a shell script calling R calling something else,
| several more layers deep, when the same result could've all
| been done in SQL with a few temporary tables.
|
| Oh, then I'm asked to "give this a code review before so-and-so
| does a deployment tomorrow." Uh, it's a little late to address
| any of the fundamentals, but there are hard coded paths
| everywhere...
| bob1029 wrote:
| I recently got a bit of a shocked reaction when I proposed to
| directly load daily files into temporary SQL tables and then
| use merge commands within the database to load the final
| tables. My use of code is essentially a shim between an SFTP
| client and SQL Server in this scenario. Maybe ~200 lines to
| connect, locate the files, run the bulk load operation, and
| then invoke the merge commands. Most of the fun bits are in
| the actual merge scripts.
|
| Once your data is safely inside the database (temporary load
| tables or otherwise), there really isn't a good excuse for
| pulling it out and playing a bunch of circus tricks on it.
| Moving and transforming data _within_ the RDBMS is infinitely
| more reliable than doing it with external tooling. Your ETL
| code should be entirely about getting the data safely into
| the RDBMS. It shouldn 't even be responsible for testing
| new/deleted/modified records. You really want to use SQL for
| that.
|
| You'll also be able to recruit more help if everything is
| neatly contained within the SQL tooling. In my scenario,
| business analysts can look at the merge commands and quickly
| iterate on the data pipeline if certain customers have weird
| quirks. They cannot do the same with some elaborate set of
| codebases, microservices, etc.
|
| One specific thing that really sold me on this path was
| seeing how CTEs and views can make the T part of ETL
| 10000000x easier than even the fanciest code helpers like
| LINQ.
| snidane wrote:
| The architecture is sound - typically called ELT these
| days. Dump contents of upstream straight into a database
| and apply stateless and deterministic operations to achieve
| the final result tables.
|
| SQL server is where this breaks though. You'll get yelled
| by DBAs for bad db practices like storing wide text fields
| without casting them to varchar(32) or varchar(12), primary
| keys on strings or no indexes at all, and most importantly
| taking majority of storage on the db host for tbese raw
| dumps. SQL Server and any traditional database scales by
| adding machines, so you end up paying compute costs for
| your storage.
|
| If you use a shared disk system with decoupled compute
| scaling from storage, then your system is the way to go.
| Ideally these days dump your files into a file storage like
| s3 and slap a table abstraction over it with some catalog
| and now you have 100x less storage costs and about 5-10x
| increased compute power with things like duckdb. Happy data
| engineering!
| jamesblonde wrote:
| You're basically describing the Lakehouse Tables
| architecture. Store your data as tabular data in
| Iceberg/Hudi/Delta on S3. Save a bucket on storage. Query
| with whatever engine you like (Snowflake, Redshift, BQ,
| DuckDB, etc).
| aoeusnth1 wrote:
| Yes, this is the vast majority of my data work at Google
| as well. Spanner + Files on disk (Placer) + distributed
| query engine (F1) which can read anything and everything
| (even google sheets) and join it all.
|
| It's amazingly productive and incredibly cheap to
| operate.
| hinkley wrote:
| Except everyone wants microservices each with its own
| database.
| tgv wrote:
| Some of my colleagues use Microsoft PowerBI, and indeed, they
| upload a few hundred rows of data (and a few hundred columns,
| which get unpivoted in powerbi to a say 40k rows). When they
| upload it, the powerbi instance overloads, and people get
| timeouts and such. That can last up to 20 minutes. I stay
| away from that as far as I can.
| jitl wrote:
| I wonder what company they're describing here. It sounds like so
| many self inflicted problems that that you could undo or set
| right in a couple of weeks if you had the time and latitude to
| make changes across the system instead of being confined to a
| small area of team ownership.
| bartread wrote:
| I worked somewhere that had a lot of this sort of thing going
| on once. You cannot overestimate how hard it is to get anything
| done: politics and organisational dysfunction, not to mention
| that you probably don't have access to half of what you need to
| in order to fix any given problem and are even more unlikely to
| be able to get it, mean there are just huge scads of problems
| that, on the face of them, look relatively straightforward to
| solve but which, in practice, are organisationally impossible
| to solve.
| whstl wrote:
| Yep. I don't work with data engineering, but from hearing war
| stories from them, this could perfectly describe the last
| four companies I worked at. :/
|
| _> you probably don't have access to half of what you need
| to in order to fix any given problem and are even more
| unlikely to be able to get it_
|
| I once sat down with a data engineer to try to fix a specific
| problem they had and that was 100% accurate. They were left
| to die by Ops and CISO.
| Terr_ wrote:
| A framing/question I like to ask use is: "Look for a root
| problem that _ought_ to be fixed through a change in policy,
| politics, or incentives, and the wasteful use of time /money
| is how the company tries to avoid or defer facing it."
|
| For example, Operations might demand that Engineering
| develops an increasingly-byzantine approvals process, to stop
| Sales from over-promising impossible or unprofitable
| projects.
| deergomoo wrote:
| > you probably don't have access to half of what you need to
| in order to fix any given problem
|
| I've found this to be one of my largest day-to-day problems
| even in a relatively functional organisation. Particularly
| when it involves something I can't run on my own machine,
| like an AWS service.
|
| In a previous role I often found myself constructing
| elaborate hypotheses about what was going on inside systems I
| couldn't see into. I'd then need to try to verify it with
| someone on another team, in another timezone, who had the
| access but not necessarily the development background. Which
| usually meant getting on a screen share and asking them to
| click various things I wasn't allowed to. If I was wrong,
| back to the drawing board and start again.
| knallfrosch wrote:
| You're operating in the same company that let this happen in
| the first place.
|
| And -- not to take away from the piece - If noone noticed the
| logs have been garbage for two years, they can't be that
| important.
| pavel_lishin wrote:
| The logs are unimportant until suddenly they're critical.
| moregrist wrote:
| It all seems easy to fix until you end up in a place where
| management process (and thus politics) has become more
| important than outcomes.
|
| The people with experience and knowledge get pushed aside in
| favor of someone who can talk management-speak and (often)
| looks the role. Suddenly meetings where Things Get Decided
| only include managers who don't know what they're doing.
|
| The best is when they make a mess and then get tremendous
| accolades for half-fixing it.
|
| I've even seen it happen within relatively small startups.
| It's a sign of rotting culture, but sometimes you have a
| mortgage to pay and have to get comfortable with the
| situation until you can find a better gig.
| ludicity wrote:
| I wrote a piece last year on how they were running a $500K
| Snowflake bill due to typing one number in wrong, and how I
| noticed within my first few weeks there by literally
| eyeballing something and going "queries take milliseconds so
| you really have to justify everywhere a minute appears in the
| configuration".
|
| The logs are stupid (sorta) but imagine how many other issues
| can exist if we uncover something like this every time we
| open something.
| taberiand wrote:
| Irreducible complexity and load-bearing bugs. Kludge built upon
| kludge built upon kludge. An engineer with the go ahead and all
| the support required would still struggle because the knock on
| effects of any one change cascade out in unpredictable ways.
| Not to mention working in an active environment where, although
| the other engineers support the goals in theory, they still
| need to deliver business requirements for a fickle management
| that doesn't truly understand what lurks beneath the facade -
| they don't have time to do it the right way, and if they try
| they'll just break both the old system and the new.
| tiew9Vii wrote:
| Sadly it sounds like most AU corporate companies I've worked
| at.
| chubs wrote:
| Why are we so crummy? Is it because our talent finds it
| generally easy to migrate to the states for better
| opportunities, because of no language barrier and the
| 'coalition of the willing' visa (iirc)?
| ykonstant wrote:
| This must be one reason (not Australian, but have friends
| there), but more importantly it seems to me the "better
| opportunities" are _much, much better_. Australia seems to
| have worker rights that make most of the US look decent by
| comparison while wages remain far below the latter---and I
| don 't mean just US hot spots: an Australian friend went
| from Sydney to Salt Lake City (!) to get a much better job
| with much higher compensation _and_ better rights
| /environment. In Utah!
| jpmoral wrote:
| > Australia seems to have worker rights that make most of
| the US look decent by comparison
|
| This is the first I've heard anyone say that. Can you
| elaborate? We've got more leave (separate holiday and
| sick leave), none of the "at-will" stuff, right to
| disconnect, less hours, etc.
| BrandonM wrote:
| US software engineer. I have 24 days of PTO, 15 company
| holidays, and 9 sick days. 10 weeks for parental leave
| (16 for moms). $240K salary, $400-600K in annual vesting
| equity. That's private paper equity, but I've already
| been able to cash out $700K and buy a house with cash.
|
| Fully remote. I can expense $120/month for phone and
| internet, and a few lunches each month, too. I can get a
| new laptop and/or monitor sent to me just by asking.
|
| When I do visit the office, the trip is fully expensed.
| Free daily lunch. Coffee and drinks and snacks
| everywhere, free. Private desks in a semi-open office
| with couches scattered around. Lounges with hundreds of
| board games, nearly all of which have seen table time
| during work hours.
|
| Primary projects are tracked in a knowledge-sharing
| system, but I can mostly work on what I want to. I'm
| encouraged to merge small fixes and refactors without any
| ticket-pushing at all. Yelling by managers or anyone
| would not be tolerated.
|
| "At-will" is more FUD than reality in my experience. Most
| companies, when firing or laying someone off, give
| something like 2 weeks of severance for every year of
| service.
| jimby wrote:
| I work at more typical software job for an AUS company
| operating in the US ... AUS workers make less money than
| the US office but they can rollover PTO indefinitely,
| right to log off, etc. I think the main reason people
| come to the US to work from aus is the cost of housing
| anywhere near the cities is exorbitant and the australian
| version of the american dream is unbelievably dead.
| ewuhic wrote:
| How do I get such a job as someone from Europe? Where do
| I search for such a job? Is it Bay Area only?
| BrandonM wrote:
| I wish I could suggest something for you. My path was
| moving to the Bay Area 13 years ago to work for a small
| startup, helping to grow it, then going remote after a
| few years. Startup is a B2B with an ethical technical
| founder, and it had a credible business model from day 1.
| lelandbatey wrote:
| The typical problem is a culture where if you say
|
| "I would like to make this systemically better by addressing
| our second order problems that are causing our very visible
| first order problems"
|
| You will be told that we absolutely do not have time for that.
| The only actions you're allowed to take are fighting the fires
| closest to you, not turning off the pumps that spray the
| gasoline everywhere.
|
| Typically this is only even possible because nothing that
| you're doing is actually used or scrutinized since if it was,
| someone would have immediately noticed that nothing works.
| Usually this is at places running on varying levels of
| investment dollar three-card-montey.
| 3eb7988a1663 wrote:
| I inherited a pipeline like this. It is as if everything is a
| global variable. You cannot "just fix" one thing in isolation,
| because some spooky action at a distance of which you were
| unaware relies upon this insane behavior. Each and every hack
| is the expected input somewhere else in the chain. You have to
| carefully inspect everything downstream of any kind of minor
| adjustment because your cleanup is quite likely to break
| something else.
|
| Immensely frustrating and draining where you can have
| accomplished ~nothing in a full day of work to fix what should
| have been a five minute change.
| ludicity wrote:
| I'm the author. You are exactly correct. Everything was so
| heavily interwoven that it was impossible to tell what would
| happen downstream without making an edit and then tracking
| the changes through dozens of steps through the architecture
| diagram.
| baq wrote:
| `The purpose of the system is what it does.'
| jitl wrote:
| In this situation I'd make a second copy of a bad thing and
| then try to make the copy good, instead of changing
| anything in place. But yeah, I totally get the huge
| upstream fight it must be and I'm not trying to backseat
| drive... just marveling at how the organizational fuckups &
| constraints make it hard to fix obvious problems
| ludicity wrote:
| You're right. I actually did propose this, but the other
| difficulty is getting stuff put onto the Jira board. My
| belief, yet to be verified, is that it was possible to
| deliver everything the executives promised AND perform
| the refactoring without any drop in service, but this
| would have required convincing management that a well-run
| team could 3x productivity. Eventually everyone got tired
| of talking to people that were just used to bad
| performance, so they couldn't envision smooth CI/CD and
| happy workers (as opposed to contented to get paid to
| hang around workers, which is what happy means in many
| cases).
| bartread wrote:
| I'm going to read the rest of this. I'm enjoying it. But,
| simultaneously, part II has me so triggered - it bears striking
| resemblance to repeated situations I've encountered where the
| meaning and content of columns in a relational database were
| overloaded in varying degrees of heaviness (which is a practice I
| absolutely detest) - that I need to take a short break.
| XorNot wrote:
| People do this with system hostnames _a lot_.
|
| And it's almost impossible to get them to stop: the hostname
| should either be a random UUID or a random name from a
| pronounceable list depending on scale (or a syllabic UUID
| thing).
|
| Because every other factor has one answer: you look up the
| other data you need in your CMDB. If that's too hard, you fix
| that so it's easy (DNS TXT records can be surprisingly useful
| here).
| hermitcrab wrote:
| I manage a database for a small local charity. I have set it up
| so that only I can add, delete or change the column structure.
| If someone wants a change, they have to email me and convince
| me (they are fine about this BTW). I'm sure the database would
| be an utter disaster zone by now if everyone was allowed to
| change it.
| klysm wrote:
| I think database schemas deserve to be protected with one's
| life as the holy ground of the system. If the schema is
| fucked, everything else will be fucked too.
| hermitcrab wrote:
| With a database you can lock down the schema. In reality
| though, many data system are composed mainly of people
| emailing in Excel spreadsheets. Good luck enforcing any
| sort of schema there.
|
| My day job is writing a desktop/file-based ETL system. I
| have just added in a schema version feature to cover these
| sort of issues. It was one of the most requested features,
| because most people aren't able to control the schemas of
| the data they receive.
| lifeisstillgood wrote:
| >>> pretending that any of this is more important than hiring
| competent people and treating them well. I could build something
| superior to this with an ancient laptop, an internet connection,
| and spreadsheets.
|
| Ow
| iamthepieman wrote:
| I do not use this term to refer to myself. I respect those who do
| and respect the meaning behind it but am just old enough that it
| feels alien to me 99% of the time.
|
| But I am SO triggered by this piece. I had that intrusive feeling
| you sometimes get when driving where you think, "I could just
| close my eyes and see what happens", "Or that clif is so close
| and the guardrail doesn't really extend far enough"
|
| Only for my career. Like I should just not show up on Monday. I
| should get in the car and drive far away and change my name and
| work at a nice retail joint in a mid-sized town.
|
| I'm going to need to sit and stare into the distance for an hour
| and 3.
| Electricniko wrote:
| > There has been a point in my life where I ended every day in
| the dark, staring at a wall for an hour or two straight, trying
| to figure out why everything felt awful.
|
| From his post about burnout and mental health. Also worth a
| read.
|
| https://ludic.mataroa.blog/blog/on-burnout-mental-health-and...
| SanjayMehta wrote:
| If on Monday morning you're wishing it was Friday evening,
| it's time to quit.
| chubs wrote:
| Many of us have kids to feed! The economy is not bursting
| with jobs anymore since rates rose post-COVID.
| karel-3d wrote:
| No need to quit immediately, just apply for jobs on the
| side.
| Aeolun wrote:
| And do the same thing elsewhere for less money and with
| less social capital?
| tpxl wrote:
| I went from an important cog with low pay but high
| responsibility to a much higher paying job with no
| responsibility. You can too with a bit of luck.
| Aeolun wrote:
| No, I mean, I'm there, but it's anything but fulfilling
| xD it's just that anything else would feel like a
| downgrade.
| sph wrote:
| The frightening thing about serious work-related burnout is
| that three years after quitting, on Monday mornings you
| still wish it was Friday evening.
|
| Any day now I'll be ready for the grind again. Any day now.
| ludicity wrote:
| It took me about six months off to start feeling normal,
| and I think I got out much earlier than most people do.
| And if you read that post, I still clearly let it get
| pretty bad before I left.
| feoren wrote:
| What about if on Friday morning you're wishing it was
| Monday? Like, two Mondays ago? So you weren't quite as late
| on everything?
| whstl wrote:
| As someone who managed to stay productive during a
| burnout despite constant bullying by a yelling CTO: it
| doesn't really help if you deliver on time.
| andrepd wrote:
| Yep, in theory yes, but shame that the bills won't pay
| themselves
| _proofs wrote:
| i don't buy that any situation is so hopeless, you're
| powerless to improve it. at least in the context of this
| field and its line(s) of work.
|
| sounds a lot more like learned hopelessness making it
| harder to respond to stress with radical change because
| of (normal and human) fears of the unknown.
|
| at some point though responsibility for the
| circumstances, the feelings, the stress -- the good, bad,
| and ugly or easy, hard, and nearly impossible -- has to
| be taken.
|
| there's only one life to live. we owe it to ourselves and
| others to do more than -- to try not to -- just "roll
| over and play dead", so to speak.
|
| humans have survived a lot and have adapted to just as
| much if not more.
|
| if i ever allowed myself to even stay at any of my former
| jobs coming up in my life when i was paycheck to paycheck
| because of not making rent or just being flatout broke
| and homeless, i would have not progressed my career, or
| life, in any meaningful way, and just fed the negative
| feedback loop influencing what feels like a miserable
| existence (even privileged as it were).
|
| can't hold myself hostage. and also, i can't hold those
| around me hostage as consequence of my non-action,
| either.
| bryancoxwell wrote:
| > I had that intrusive feeling you sometimes get when driving
| where you think, "I could just close my eyes and see what
| happens", "Or that clif is so close and the guardrail doesn't
| really extend far enough"
|
| L'appel du vide
| albert_e wrote:
| Does the mention of such concepts or acknowledging it is real
| ... put some lisetners (if they are work certain professions)
| under an obligation to refer the person to a mental health
| assessment?
|
| Example: a blog post like this one, with the author's real
| name, that acknowleges it front and center: https://ebb-and-
| flow.blog/2023/07/23/another-scan-lappel-du-...
| josephg wrote:
| Seriously, quit then. It's not worth it. You get one life. How
| many hours on this earth do you want to spend suicidally
| depressed? If you have a really high pain tolerance, maybe you
| can do that for years. How lucky would that be?
|
| There's a polish restaurant near where I live that makes
| amazing food. The owner is always out and about, chatting with
| customers and making sly jokes. Turns out he used to be an
| oracle sql consultant of some sort, and he turned it all in to
| run his restaurant. You can tell he's thriving. I think he's
| got the right idea.
| andrepd wrote:
| Survivorship bias. In an ideal world yes but in reality
| there's bills to be paid and tech (generally) pays really
| well.
|
| Not saying you shouldn't quit, just that it's not so simple.
| josephg wrote:
| I hear you. But also, ... if you're literally feeling
| suicidal because of work, in a sense it really is that
| simple. You aren't doing anyone any favours - not your
| coworkers, your family or yourself - by living like that.
| hackable_sand wrote:
| This is the answer.
|
| Money is money, but money comes and goes.
|
| The work you put into finding a healthy source of income
| is worth every minute.
| hinkley wrote:
| There's literally a 90% chance your restaurant won't
| survive its first year.
| josephg wrote:
| Then do something else! Literally billions of people are
| employed every day doing things that aren't software
| engineering. Pick anything.
| isoprophlex wrote:
| Change something for the better. You are the one who cares the
| most, you are the one best suited to take control over your
| life.
|
| You deserve to feel good. Life is too short to be a cog in a
| broken machine.
| strken wrote:
| It's an almost exact copy of my last few months, right down to
| the 10am start.
|
| Except that all our other senior engineers got laid off and
| there's nobody to pair with, I don't give two fucks about
| bullying because at this point the entire company knows I'll
| quit on the spot if they try, and our problems are mostly that
| the remaining team cannot understand the terrifying eldritch
| decision making process that led to fun little patterns like
| "wrap every API call in a try/catch and then ignore the
| errors".
|
| I am seriously considering doing a TAFE course and becoming an
| electrician.
| Buttons840 wrote:
| They took inspiration from the error steamroller:
| https://github.com/ajalt/fuckitpy
| jauntywundrkind wrote:
| The observability world still regards itself as a system for
| monitoring, but reading (and sometimes seeing) how these systems
| just go so bad continues to drive a conviction that perhaps their
| strategies and tools should become bigger. That they should
| converge with business pipines.
|
| We shouldn't just have wide events/big spans emitted... We should
| have those spans drive the pipeline. Rather than observability
| being a passive monitoring system, if we write code that reacts
| to events we are capturing, then we shuffle towards event
| sourcing.
|
| Given how badly coupled together with shoestring glue & good
| wishes so many systems are, how opaque these pain zones are, it
| feels like the centralization upon existing industry standard
| protocols to capture events (which imo include traces) is a clear
| win.
|
| (Obvious downside, these systems become mission critical,
| business process & monitoring both.)
| halfcat wrote:
| What's this look like in practice? Is this something like
| business process modeling and workflow engines, or something
| else?
| edejong wrote:
| Totally agree. Observability is just another dataset and should
| be modeled, managed and governed as other datasets. Data
| quality controls should be equal or of higher standard than
| regular data sets.
|
| Monitoring, dashboarding and alerting should leverage other BI-
| class tooling.
| halfcat wrote:
| What's the solution to wrangling these data projects?
|
| The author's experience is not far off from my own.
|
| 1. Any solution in place can only be understood by the person who
| created it
|
| 2. _"No, we can't change that because then we'd have to validate
| everything from scratch again"_
|
| And therefore, as the author says:
|
| > _"we 'll continue with the work instead of fixing the critical
| production error"_
|
| I'm honestly not sure how to address it either. With traditional
| software dev we'd write tests, incorporate those into CI/CD, and
| start to course correct. We can use sample data to validate the
| code does what we think it does and that we didn't break it.
|
| But in these data projects, it's not only the code that's
| changing, but the data is also a moving target. You can write a
| test with sample data, but tomorrow your data might change
| because someone in sales added a custom field to the CRM, or IT
| upgraded the accounting software and all of the unique IDs
| changed, or someone upgraded their Excel version, or whatever.
|
| And your code that works on the sample data needs to handle all
| of this, which obviously it can't. You can try to validate the
| data somehow, check the schema, check if the number of rows
| hasn't doubled or halved, and so forth, and then stop it from
| importing until you look into it, but also you can't stop inbound
| data because an exec has a meeting in a few hours and expects
| their report to be updated.
|
| I heard something about "data contracts" that's supposed to
| address this, but it sounds like the next in a long line of buzz
| words intended to get management to buy another data product.
|
| Has anyone worked in this kind of project that went well?
| 3np wrote:
| "We should add a single-source-of truth validation microservice
| that we put in front of all insertions, putting failing
| messages on a separate queue"
|
| And on it goes.
| Liquix wrote:
| https://xkcd.com/927/
| ludicity wrote:
| Author here, and also executive director at Hermit Tech now
| where we do things like this. Your approach has the core of how
| I'd go about it. The contract stuff is legit, though you don't
| need to buy a product for it.
|
| The thing that is hard at big organizations isn't that
| executives need the data for meetings. The issue right now is
| that many organizations are already 3-4 years into building
| their analytics platforms, and Chief Data Officers worldwide
| are trying to prevent their role from disappearing. They're
| already very much "Mom, we have CTO at home" in many companies,
| as evidenced by the fact they're usually reporting to the CTO
| or CFO.
|
| So at this stage, they've already told the business that the
| platform is "ready", and they are onboarding data sources. With
| no way to measure data quality, the only thing visible at the
| organization level is number of data sources onboarded. The
| fastest way to onboard data sources is to have good CI/CD and a
| solid developer environment, but this would probably result in
| slowdown for 1 - 2 months even if you had executive backing to
| bulldoze all objections from the IT department.
|
| That's the sort of thing I can commit my team to as a business
| owner, but most executives don't have the nerve to slow
| delivery down and aren't losing money out of their pocket due
| to the inefficiency - I get to talk with a lot of them due to
| the blog's success these days, and many of them really are just
| employees with more status, with the same incentives. And to
| make it worse, the loss of nerve is actually understandable,
| because the type of team that would build something this bad
| will also waste those two months then still deliver slowly! But
| most people aren't thinking in terms this complex, and yes, I
| know it isn't that complex.
|
| I'm expecting to pick up some work in this area at larger orgs
| in a few years when these leaders rotate out and new leaders
| rotate in and go "what the hell IS this?", but for now we're
| mostly aimed at helping smaller places do it right from day
| one.
| ewuhic wrote:
| Do you have a post describing the "right [way] from day one"?
| Spill your secret sauce.
| ludicity wrote:
| The boring answer is that it's context dependent, but
| fundamentally "do data engineering the same way you do
| high-performance software engineering". Have tests that run
| fast, where fast means "a few seconds when you start" and
| "refactor as you go so the tests keep taking a tolerable
| amount of time". I think Kent Beck suggested 10 minutes in
| Extreme Programming Explained.
|
| We're gradually forming our own, complex opinions in this.
| In the consulting context, this is essentially our product.
| A fascinating realization moving from software to marketing
| is that a sales pitch or marketing strategy can be built in
| a way that isn't entirely dissimilar to code, and that it
| has second-order effects. They aren't the same because...
| they aren't the same, but there's an artistry combined with
| principles to doing it "right".
|
| And then as a consultant there's additional complexity, as
| each team is different. Some are high-performing and need a
| bit of an external jolt. Others need the help the most, but
| are in politicized environments so they're almost
| inaccessible until a new executive comes in who can admit
| there are problems (or indeed, even see that there are
| problems).
|
| Joe Reis has some great stuff in Fundamentals of Data
| Engineering, which includes advice on early objectives when
| rolling out a new practice.
|
| Disclaimer: Joe has hosted me on his podcast, and we are in
| the mutual-marketing whirlpool together. But I've been
| recommending his book long before I met him.
| ewuhic wrote:
| Amazing answer, thank you, straight on point with
| tangible outcome.
|
| Wish I had a LinkedIn and were an Executive, so I could
| connect with you.
| pxc wrote:
| This blog post rescheduled all my appointments, tucked me in,
| sang me a lullaby, then woke me up with coffee and breakfast late
| the next morning. I am healed.
|
| For real, a fun and refreshing read (if also a little haunting).
| marcosdumay wrote:
| The link about Scrum has a link to this:
|
| https://agile2.net/
|
| Can someone, please, tell me this is a joke. Because I can't be
| certain, but it doesn't look like one.
| SSLy wrote:
| This is the real agile2 https://scaledagileframework.com/
| marcosdumay wrote:
| Oh, this is much worse satire. It didn't even put the effort
| to add details into The One Big Diagram.
|
| (But yeah, this one I know is for real. The other one that
| feels like a joke, but looks real is harder to decide.)
| ludicity wrote:
| Author here. To the best of my knowledge, it is real. Sadly.
| bobnamob wrote:
| Jeez, that's almost as bad as the semantic versioning spec
| being semantically versioned
| Muromec wrote:
| Great piece of writing from someone who truly cares about craft
| and suffers from the feeling that this craft is not what they are
| paid for.
|
| Add: for people who sharer the feeling -- you can work in a place
| where velocity isn't all, managers are not assholes and you can
| dedicate yourself to craft.
| baazaa wrote:
| People always say this guy just has had bad luck with his
| employers but I live in Melbourne and work in data and reckon the
| whole industry is a scam.
|
| Like why didn't anyone catch the issue with the logs? Because it
| doesn't matter, every data team is a cost-centre that
| unscrupulous managers use to launch their careers by saying
| they're big on AI. So nothing works, no-one cares it doesn't
| work, most the data engineers are incapable of coding fizzbuzz
| but it doesn't matter.
|
| People always wonder why banks etc. use old mainframes. There's
| like a 0% success rate for new data projects. And that 0%
| includes projects which had launch parties etc. but no-one ever
| used the data or noticed how broken it was. I don't think a lot
| of orgs which use data as core-infra could modernize, the
| industry is just so broken at this point I don't think we can do
| what we did 30 years ago.
| le-mark wrote:
| This jives with my experience at a financial services company.
| I once sat next to the "big data team" and the company 5 year
| plan was all about delivering analytics and ai to customers
| using their data the company housed.
|
| The team consisted of one guy (who had a business degree) and a
| lot of empty cubes they were trying to fill. A year later the
| company had been acquired and the big data initiative had
| evaporated.
| RangerScience wrote:
| I have felt exactly this on regular full stack teams many, many
| times, so it's also not just limited to data teams.
|
| IMO a major factor here is that software engineering is both
| opaque and esoteric - at least with physical engineering,
| there's something people can look at and think they understand.
| baazaa wrote:
| My theory is that data is worse again because at least if
| you're making a website you're expected to end up with a
| website. The process is opaque and esoteric, but the end-
| product is somewhat tangible.
|
| A lot of data projects are moving and transforming data no-
| one cares about. They can fail completely silently, a manager
| can lie and say 'we've successfully built the data platform
| which is going to enable AI analytics' and it'll be like a
| misconfigured S3 or something. No-one's checking the end-
| product or even understands what it's meant to be.
| RangerScience wrote:
| Excellent point... and one I should know. I spent about 6mo
| as a data eng (only one at the startup) and long after,
| found out no one ever had a clue what I was talking about
| in standup. (To be fair, I was self-teaching, and no-one
| else knew anything so)
| tiew9Vii wrote:
| Reluctantly worked on AU data projects for maybe the past
| decade. I don't classify myself as a data engineer, in fact I
| hate data engineering or data related work which is glorified
| ETL and SQL most of the time. They are the worst, broken
| projects I've done, not software engineering in the software
| engineering sense. I don't think I've worked on a good one yet
| despite the potential to be really interesting projects. I
| prefer general software projects doing a bit of everything as a
| generalist, data pays the bills though in AU.
|
| Not seen/heard of this person before but reading this specific
| blog post it all sounds very familiar, it's depressing.
|
| The "CTO" getting on stage taking a bunch of credit and
| everything being a mess or incomplete or lies is very familiar.
| Maybe not CTO but higher management. It's all smoke, mirrors,
| optics, self promoting, it works as these people end up making
| their way up the ladder when the lonely dev trying to do better
| work is just a dispensable cog in the wheel.
| 6510 wrote:
| > Not seen/heard of this person before but reading this
| specific blog post it all sounds very familiar, it's
| depressing.
|
| Someone once told me he, as a form of therapy, rewrote the
| company he worked at in a few weekends. He never mentioned it
| to his coworkers, it was strictly a therapeutic effort. They
| apparently spend years "fixing" things without making any
| progress.
| intelVISA wrote:
| Most apps are trivial for a decent dev to reproduce, I'd
| wager the root problem is rarely the codebase: the org is
| rotting. Years of 'fixes' with no progress is like blaming
| the water for sinking a ship.
|
| Success attracts deadweight who (un)intentionally sandbag
| efforts to reverse this downward trend for their own self-
| preservation. I don't blame them, doubt there's a fix when
| the system requires most people work bullshit jobs instead
| of collecting UBI.
| whstl wrote:
| Bingo. The #1 thing I learned in consulting is that you
| can't build good software if the processes and structures
| are wrong in the first place. Ditto with off-the-shelf
| software.
|
| Something that takes a week in company 1 can take a year
| in company 2 purely because of organizational issues.
|
| Rotting organizations will produce rotting software.
| lelandbatey wrote:
| I have seen data work well, but it only worked well in a
| situation where we had management focusing on two very tangible
| things that even the CEO could verify (since the CEO did know
| the product). Those tangible things were:
|
| 1. Accurate, auditable billing down to per-chargable
| entity/event
|
| 2. Dashboards for each customer that reflect THE SAME numbers
| as we generate in #1, so customers could see relevant info
| quicker than just waiting for the bill from #1
|
| The only reason those things were valued and made a focus
| though was because a HUGE customer threatened to completely
| drop our company because that customer did an audit and noticed
| that we had overcharged them 3% because we were actually
| billing them on estimated numbers. That led to our CEO being
| personally yelled at by a _much_ larger CEO, and our CEO (to
| his credit) didn 't blame us (we'd raised the alarms that the
| bills were estimates and not auditable) but did say "this can
| never happen again, I trust you, do whatever needs to happen to
| make sure this never happens again."
|
| And once we had #1 solid and tight, we were able to leverage
| that solid auditable data to generate solid dashboard numbers
| that _always_ squared with what showed on the bill.
| RHSman2 wrote:
| A CEO worth working for
| denimnerd42 wrote:
| I lead a team on a large data project at an enormous bank,
| hundreds of devs on the project across 3 continents. My team
| took care of the integration and automation of the sdlc
| process. We moved from several generations of ETL applications
| (9 applications) netezza/teradata/mainframes/hive map reduce
| all to spark. The project was a huge cost savings and great
| success. Massive risk reduction by getting these systems all
| under 1 roof. We found a lot of issues with the original data.
| We automated the lineage generation, data quality, data
| integrity, etc. We developed a frame work that made everything
| batteries included. Transformations were done as linear set of
| SQL steps or a DAG of sql steps if required. You could do more
| complicated things in reusable plugins if needed. We had a rock
| solid old school scheduler application also. We had thousands
| of these jobs. We had an automated data comparison tool that
| cataloged old data and ran the original code vs the new code on
| the same set of data. I don't think it's impossible to pull off
| but it was a hard project for sure. Grew my career a ton.
| RHSman2 wrote:
| What was the main reason for your success?
| jonathanlydall wrote:
| Not the person you're replying to, but I would expect that
| a near universal answer to this across all kinds of
| projects (not just software) is effective collaboration and
| communication between stakeholders and teams.
|
| Despite no shortage of technical talent on large projects
| they can still often fail, and it's because building a
| technically impressive thing doesn't matter if it doesn't
| do what business needs.
|
| So it's about making sure you're building the "right" thing
| that delivers on business's actual needs, and the only way
| to find out what those are is through constant and ongoing
| good communication between technical and business people.
| ozim wrote:
| Downside is lots of work business is doing is running
| around with wheelbarrows and they actively sabotage it
| when someone wants to build a conveyor belt.
| moregrist wrote:
| The flip side of this is that the stakeholder has to
| actually care enough to invest in collaboration and have
| enough bandwidth to be able to follow through.
|
| The kind of communication that lets cross-functional
| projects be effective is time consuming, and competent
| people tend to be overworked, no matter what part of the
| business they're in.
| RHSman2 wrote:
| I was fishing for that answer. Glad to hear this is the
| universal answer (not well implemented)
| jorvi wrote:
| Specifically for the financial sector and especially banks
| and government tax departments, they're on a clock.
|
| As time moves on, there are less COBOL engineers. Hell,
| sometimes their systems have been written in a bespoke
| language. There is less and less understanding of why
| something is set up the way it is due to lack of
| documentation. Updates / changes to the code sometimes have
| to wait for 2-3 years because the system isn't flexible
| enough (literally, not as in "this change will take 2-3 dev
| years"). Even code that old contains bugs, but due to the
| age of the code they're inscrutable.
|
| However, whichever new system gets tooled up has to be
| 99.999% flawless, or it could cause serious damage to the
| bank and even its regional market.
|
| When there is that kind of pressure, dev teams are no
| longer considered a cost sink, money flows, and the world
| is possible.
| d0gsg0w00f wrote:
| I think the difference is that technical and business
| leadership at a bank understand that data is lifeblood. Bad
| data will get you on the front page of WSJ and a phone call
| from a regulator in Luxembourg.
|
| For a lot of smaller Internet companies, data is just a
| fluffer. The real business is in image and which VC bbq you
| get invited to.
| neeleshs wrote:
| This story is more an exception than norm.
|
| I know startups that hired data engineers, deployed
| warehouses,DBT, a BI tool and churned hundreds of reports,
| and in one case their DBT project has hundreds of files. No
| one in that company knew why any of it was used.
|
| All said and done the business users wanted three reports.
|
| More often than not data teams are self-serving than anything
| else.
| bob1029 wrote:
| > I don't think a lot of orgs which use data as core-infra
| could modernize
|
| I argue this is a happy conclusion, not a problem to be solved.
|
| What would "modern" bring to a bank except even more pain &
| suffering? Database technology invented in the 80-90s is more
| than sufficient for tracking information at the scale that 99%
| of financial institutions operate at today.
|
| Virtually every core conversion project I've ever heard of has
| been a failure or is currently a burning wreck on its way to
| the bottom.
|
| The only new bank projects that touch data and seem to succeed
| are LOB apps with highly curated experiences that are tightly
| integrated with the actual front/back office business. Having
| buy-in from staff regarding your UX is way more important than
| spinning out a 20 page AWS architecture diagram. The CTO can
| only take you so far through the vendor approval process at a
| bank. Retail operations (i.e. the people who are responsible
| for the brick & mortar branches) typically have substantially
| more pull in these organizations.
| makeitdouble wrote:
| > What would "modern" bring to a bank except even more pain &
| suffering?
|
| In the most simple term, a future.
|
| Except if your bank is literally too big to fail, at some
| point you have to either move on from 80s technology or at
| least bring in an adaptation layer, because your profit
| center have also moved on or you're facing harder
| competition.
|
| A typical example is banks getting merged: there will be a
| fight to see which system stays and which one disappears. If
| you froze your technology 4 decades ago it won't be your
| stack winning. [0]
|
| Another is the evolution legal frameworks: EU countries
| passed laws requiring interoperable APIs to perform standard
| banking operations. Being a customer of a decent bank or a
| fossilized one made a huge difference and the market grew a
| lot more competitive. People would start hedging their bets
| when legacy banks looked too far behind.
|
| [0] The most interesting and recent example of this is Mizuho
| bank just miserably failing at that task to the point the
| gov. intervened and anyone not married to them probably moved
| out.
|
| https://www.mizuhogroup.com/news/2021/06/20210615_2release_e.
| ..
| iamacyborg wrote:
| > If you froze your technology 4 decades ago it won't be
| your stack winning.
|
| I'm unclear whether this is bad for the business or just
| bad for folks hoping to keep their jobs.
| lmm wrote:
| Mizuho is doing great, they're probably the least awful of
| the big Japanese banks. Everywhere is like this, and "old"
| technology doesn't seem to make the places that use it
| appreciably worse.
| lmz wrote:
| Another system migration example (TSB, 2018) from the UK:
| https://www.tsb.co.uk/news-releases/slaughter-and-may.html
| bob1029 wrote:
| > A typical example is banks getting merged: there will be
| a fight to see which system stays and which one disappears.
| If you froze your technology 4 decades ago it won't be your
| stack winning. [0]
|
| In my experience (small/mid-size US banks), the institution
| with more assets or branches usually wins. It rarely has
| anything to do with technology. If a 6 region, 200 branch
| monster comes in and wants to buy some 4 branch relic in
| the West Texas desert, it doesn't matter if the smaller
| institution has achieved AGI and an intergalactic core
| platform. They're almost inevitably gonna be merging their
| records into some old boring IBM system.
| shakna wrote:
| The landscape is a little different over in Australia.
| Most of the Big Four are closing as many branches as they
| can. Branches are no longer a mover or shaker, because
| most Australians never touch cash anymore. [0] Most
| transactions are digital.
|
| Almost as many people pay with card as with phone.
|
| Faster record systems, faster transfers, actually do win
| people over here.
|
| [0] https://www.rba.gov.au/publications/bulletin/2023/jun
| /cash-u...
| finnh wrote:
| I welcome the day when the US stops devoting enormous
| amounts of useful real estate to bank branches. They are
| a sad simulacrum of actual street life, taking up tons of
| space to advertise a bank and contributing to high rents
| that preclude less-profitable small businesses. One step
| up from billboards.
| pletnes wrote:
| I tossed out my credit card because the UX was bad. At this
| point most of the CC services are utilities or commodities.
| Just get another at a bank with better apps and website.
| xelamonster wrote:
| Yup, I'm moving to a new ptimary checking account
| currently because I'm sick of my local credit union that
| is apparently so incompetent they can't handle sending
| email alerts correctly. Also, any bank or credit card
| that won't support Plaid seems not even worth considering
| at this point.
| mixmastamyk wrote:
| Had to look up what plaid was. Think I'd prefer Fednow
| support and/or Aus/NZ style modern banking, that's future
| proof. I see no reason for a third-party to be involved.
| 8n4vidtmkvmk wrote:
| Yes, please, fix the UX. That was my biggest gripe, working
| as an FSR at the bank.
|
| One particular thing was we had to convert transit #s into
| branch numbers regularly. We did this by looking at a sheet
| of paper of course. Eventually I got fed up and wrote a web
| app so you could just punch in the numbers and have it
| instantly convert. I checked and people are still using it 10
| years after I quit, which means nothing has changed and
| they're still using the same god awful software.
|
| They did move some data at some point. I know this because
| they screwed that up too and partially merged my mom's and my
| bank accounts, which is a pretty bad error. Would be worse if
| it was some rando. Speaking of... That's exactly what AT&T
| did.
| chaxor wrote:
| >What would "modern" bring to a bank except even more pain &
| suffering?
|
| It probably depends on what "modern" means here. If updating
| from tons of COBOL to {Julia, Python, Rust, or some other
| well known language} with an update to an SQLite backend (or
| perhaps postgres is acceptable for very specific scenarios),
| that is likely a good choice due to being able to fix old
| cruft and add maintainability for the future. If it's a
| switch to some nosql database backend with everything
| switched to some cypher-based lang or anything that touches
| javascript in any way, it's probably a mistake.
| ericjmorey wrote:
| Why SQLite? Why Julia? These seem like poor options for
| banks.
| neverartful wrote:
| In the case of SQLite, I'd say incredibly poor (to the
| extent that the person who made the decision should be
| fired).
| __turbobrew__ wrote:
| I would like a bank which supports U2F.
| atoav wrote:
| The problem is that most orgs seem to do the wrong thing,
| because the incentives of the higher ups don't align with ehat
| is good for the org.
|
| E.g. if you are a bank ideally you'd like all your processes
| automated and streamlined with extremly transparent data flows
| etc -- and you want as many of the banks employees to be
| proficient in these systems and constantly work on improving
| the systems within an controlled environment.
|
| In practise this is not the kind of thing that allows single
| managers to come across like heroes -- so it doesn't happen
| that way and you get island solutions with duct-taped
| connections between.
| ludicity wrote:
| I also get to spend a lot of time with executives thanks to
| the blog's success, and part of it isn't just incentives,
| it's pure confusion. People have no idea what they're buying.
|
| I also get invitations to "sponsor events" now, since people
| see "director" on LinkedIn and think I have way more money
| than I do. Their business model seems to be flattering
| executives by inviting them to events where they can network
| with other rich people, then ask me for "sponsorship" money
| so that I can go into the room and brainwash them with my
| marketing material. I might even try it at some point to see
| if that's an accurate read.
| disgruntledphd2 wrote:
| If you want to make your consultancy a success, you
| probably should attend some of these and see if they help
| you get business.
| ludicity wrote:
| We're bootstrapped so we unfortunately can't fling money
| at sponsorships. Or rather, we can, but it would cut into
| our runway quite a bit, and we have more promising
| avenues to pursue. If we acquire bad clients, the type
| that would let themselves be brainwashed and who status-
| seek by attending these events, that's just going to be
| as dumb as a regular job but without the luxuries
| afforded to employees.
| rocqua wrote:
| As matt levine likrs to say. High level finance is mostly
| about seating charts.
|
| In other words, it's about status, not money.
| stackskipton wrote:
| >I also get to spend a lot of time with executives thanks
| to the blog's success, and part of it isn't just
| incentives, it's pure confusion. People have no idea what
| they're buying.
|
| So, 43 years later and Putt's law is alive and well.
| cratermoon wrote:
| Putt's Law: "Technology is dominated by two types of
| people, those who understand what they do not manage and
| those who manage what they do not understand." From the
| book _Putt 's Law and the Successful Technocrat_,
| published in 1981. An updated edition, subtitled _How to
| Win in the Information Age_ , was published by Wiley-IEEE
| Press in 2006
|
| <https://en.wikipedia.org/wiki/Putt%27s_Law_and_the_Succe
| ssfu...>
| ludicity wrote:
| Author here. I now know some places in Melbourne that have a
| good success rate on projects. Some of them are so small as to
| be invisible and rarely hire. One uses two specific independent
| recruiters or internal referrals. As far as I know, they are
| extremely profitable because the competition is a joke.
|
| For many organizations, the success rate is indeed 0%. A Group
| of Eight university (our top 8 universities nationwide), for
| example, sent me a job description a few months ago where they
| misspelled the word engineer, and left change tracking on in
| the Word document. This allowed me to walk through the profiles
| of the people running their data projects, and it was super
| obvious that many of the people involved aren't going to do a
| good job. They could have saved millions by having a random HN
| person eyeball the CVs of their chosen leadership team.
| svilen_dobrev wrote:
| hey. Congratulations on your decision.
|
| i think it all goes deeper in overall culture/attitude there.
|
| i was in Melbourne in 2012.. with idea to relocate wholesale,
| 2nd time. Worked 2 months at some "startup", that fired me
| when i finished the task given.. Seems it was cheaper to hire
| "permanent" then fire, rather than take someone on 2 months
| contract. So that's one red light on the dashboard.. There
| were other redlights from overall "society", feeling
| something-is-wrong but i did ignore them for quite a while -
| people have become evil, etc..
|
| Then i started going around places and mailing my cv here or
| there (with 22y of experience making software, by that
| time),.. ibm, ernst&young, you-name-it.. to no avail, and
| more red-lights flashed on me.. And one day visited some kind
| of meetup, organised/held in some wellknown company.
| Seemingly it was kind-of "hiring" event or so, we grouped in
| teams of 3-4-5 people, with half from company, and other half
| outsiders.. and went solving some problems of theirs. Or that
| was the "label". Any solution that any of outsiders
| suggested, was shot down, with somewhat vague reasons, that
| at the end started to sound like "if we solve this there'll
| be no job tomorrow". And Smile :) Lots of smiles. Empty ones.
|
| That was one of the Last red lights on my dashboard. Whether
| it was a financial balloon pressing everyone so they only
| smiled and did _nothing_ , in order to pay the mortgage, or
| something else, i don't know. Next day i watched
| Sacrifice/1986/Offret by Tarkovsky, and.. bought a ticket
| out. Discontinued my oz dream. For good.
|
| quoting meself, from 2007-8: "with time, places change
| people. Other way happens noticeably only while coming in -
| or switching on."
|
| have fun
| nyarlathotep_ wrote:
| >> I live in Melbourne and work in data and reckon the whole
| industry is a scam.
|
| You needn't live in Australia to reach that conclusion.
| chubs wrote:
| Do you think perhaps the problem is rooted in people being
| dishonest, and honest people are driven mad by it all and self-
| select out? The dead-sea effect?
| ludicity wrote:
| It's so many things. Dishonesty, lack of technical
| competence, political pressure, hype, organization structure,
| and incentives.
|
| If I had to summarize though, it's that the median
| performance in any field will be at much lower levels than
| outsiders expect, and some fields with hazier results have
| this level set very, very, very low, especially when they're
| hyped up. But also that the market is actually at least a
| little bit efficient, but over long time scales. I think
| there's a 50%+ chance that the role of Chief Data Officer
| begins to die off, but also that it'll be replaced by
| something silly.
| DeathArrow wrote:
| >Because it doesn't matter, every data team is a cost-centre
| that unscrupulous managers use to launch their careers by
| saying they're big on AI.
|
| If you have lots of data flowing and have full teams "working"
| 24/7 on it, does it really matter if that data is junk and that
| is not processed in a meaningful way? You can still ask AI to
| generate some nice looking charts with big numbers to show to
| investors. Investors like nice charts and big numbers. Or so,
| some businesses people think.
|
| But in all reality the investors will ask questions like: how
| will this solve problems for customers, how do you intend to
| sell this to customers, how much does it cost, how will this
| generate me money. Unless those investors plan an early exit by
| finding other, more gullible investors than them, kind of like
| knowingly investing early in a Ponzi scheme.
| CalRobert wrote:
| Ten years ago data engineering was another discipline in
| software engineering, like backend or frontend. Somewhere along
| the line the term was co-opted by "I can maybe barely string
| together some untested airflow pipelines" and it means
| something much different now.
| akdor1154 wrote:
| Agree.. I can tell you at least one Melbourne-based Flybuys
| retailer calculates your points with an unholy daily-scheduled
| stored procedure in Snowflake SQL, because.. big business
| dysfunction reasons lead to the data team being assigned to do
| it, and the data team didn't actually have any software
| engineer roles in it.
|
| At least it has tests.
| jpmoral wrote:
| Do you mean the points earned in-store that are then sent to
| Flybuys to add to your total? Or, god forbid, do they do
| their own total?
| akdor1154 wrote:
| The former, in-store and online (which is beginning to
| touch on those business dysfunction reasons).
| photonthug wrote:
| > it doesn't matter, every data team is a cost-centre that
| unscrupulous managers use to launch their careers by saying
| they're big on AI. So nothing works, no-one cares it doesn't
| work
|
| Yes. Lots of times the most important asset for these companies
| is actually contractual obligations in terms of exclusive
| access to data or customers. It doesn't matter if the product
| works, you'll have to buy the company to build a different one
| that does. But the (broken or nonsensical) product pushes up
| the value of mergers and acquisitions. If leadership completely
| makes shit up then they might go to jail, so, they burn X
| million on "work" and cloud spend as part of an elaborate
| argument that it should sell for 10X.
|
| > the industry is just so broken at this point I don't think we
| can do what we did 30 years ago.
|
| Well no, it's never been easier to do high quality engineering,
| but mbas are in charge. They don't think like philosophers or
| scientists and don't traffic in common sense.
|
| For anyone questioning their life / career choices because of
| this, it's not about you. An individual working in an
| environment like this can still be a craftsman of integrity if
| they focus on small problems and solve them well, but you need
| to be able to get satisfaction from that, not from some overall
| mission (which again, is probably fake). If you're most
| motivated to work directly on architecture, unification, etc,
| and want to change lots of things then you will probably be
| miserable.
|
| But if you're feeling shitty about the whole thing, it might
| help to realize that the actual nuts and bolts of
| adtech/martech data pipelines are much the same as the ones for
| cancer research or particle physics or climate science, so one
| can at least try and get transferable skills if circumstances
| are currently holding you hostage. Data isn't a bullshit job.
| Leadership and management that just want to play games is the
| problem.
| djoldman wrote:
| This kind of thing is pretty typical.
|
| The older the company is, the more likely one finds this morass.
|
| It won't change absent powerful technical leadership.
| holden_nelson wrote:
| I went down the rabbit hole of this blog after reading this post.
| This person's blog is amazing. I particularly appreciated this
| piece: https://ludic.mataroa.blog/blog/quitting-my-job-for-the-
| way-...
| sph wrote:
| _" and still you will lay upon your death bed thinking that you
| may fend off the Reaper if you could but Estimate how long
| dying will take!"_
|
| What would I give to be able to compose sentences like this
| one.
| snidane wrote:
| Looks like the classic mistake of every data team. Every single
| office person works with data in one way or another. Having a
| team called 'data' just opens a blanch check for anyone in the
| organization to dump every issue and every piece of garbage to
| this team as long as they can identify it as data.
|
| That's why you build data platforms and name your team
| accordingly. This is much easier position to defend, where you
| and your team have a mandate to build tools for other to be
| efficient with data.
|
| If upstream provides funky logs or jsons where you expect
| strings, that's for your downstream to worry about. They need the
| data and they need to chase down the right people in the org to
| resolve that. Your responsibility should be only to provide a
| unified access to that external data and ideally some governance
| around the access like logging and lineage.
|
| Tldr; Open your 'data' mandate too wide and vague and you won't
| survive as a team. Build data platforms instead.
| _jonas wrote:
| I'm excited for LLM applications that can setup,
| monitor/validate, and optimize data pipelines at scale. Seems
| possible soon given that SQL and most data records aren't
| intended to be human-friendly
| FridgeSeal wrote:
| Once again. Because clearly this needs to be said loudly and
| repeatedly.
|
| That is a _technical solution_ to a fundamentally _social_
| problem.
| sph wrote:
| Ah, the hubris of youth.
| spencerchubb wrote:
| When LLMs can do the following, they might be able to fix data
| hell:
|
| - Negotiate with different teams to figure out what a field
| means
|
| - Be told that a field should be converted from one format to
| another, but oh wait it's causing errors somewhere downstream
| because it was told the wrong instructions
|
| - People come to you with some issue about the code you
| maintain, and you dig enough to realize the root cause is
| another team's code
| hinkley wrote:
| - explain how they came up with "12" as the answer.
| hermitcrab wrote:
| And now we have two problems...
| hinkley wrote:
| Ten problems. Regex is for wusses who fear danger. LLMs are
| where real men turn.
| sourcepluck wrote:
| Either this is a joke comment, or you haven't seen
| https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...
| codethief wrote:
| Largely unrelated, but "data hell" reminds me of this classic
| from Silicon Valley: https://m.youtube.com/watch?v=YPgkSH2050k
| hinkley wrote:
| "I don't want to live in a world where other people make the
| world a better place better than we do."
|
| Mike Judge did such a great job of making that character into
| Dollar Store Steve Jobs.
| lucidguppy wrote:
| > because we don't have any tests,...
|
| Right there ^^^
| lucidguppy wrote:
| > I could build something superior to this with an ancient
| laptop, an internet connection, and spreadsheets. It would take
| me a month tops.
|
| ^^^ Then do it... and then strangler fig the original.
| ahoka wrote:
| You can't as it will immediately draw out the leeches who will
| hurl "best practices", strategic partnerships with scam
| software vendors and compliance check ticking bullshit at you
| until you back off.
| ludicity wrote:
| Author here. This is essentially the case for many clients,
| especially governemnt. But we're bootstrapped, so we can
| afford to consult and use what we've learned to build tooling
| that only targets a small handful of sane clients.
|
| I've sought advice from various people on this, some who are
| famous-ish or quiet sales powerhouses in the US. My question
| was "What are executives buying when they hire consultants?",
| and the answer is consistently "Comfort". No one is actually
| comforted by Deloitte, KPMG, whoever.
|
| The moment I had confidence in an ethical consulting practice
| is the moment someone said "I don't even know where I would
| hire good consultants". This was someone with 30 years or
| something absurd of industry experience, including mentoring
| people that went on to become staff engineers. After
| processing that, I realized I don't know where to hire a
| consultancy that isn't going to bait-and-switch me with
| mediocre talent. They obviously exist, but they probably can
| only support something like 1 to 10 clients each.
|
| Even Thoughtworks, a place that I used to hear mentioned
| positively, was flagged by the CTO of a >$1B company over
| lunch last week as "shifting to bait-and-switch" tactics.
|
| tl;dr Pretty sure you can absolutely do better and make a
| living off that, but you have to think very carefully, do
| research, read a bunch of sales/marketing books, have great
| communication skills and at least adequate engineering
| skills. I still don't know if I have some of those, but if I
| fail it'll be a skill issue, not because the problem is not
| tractable.
| erulabs wrote:
| As a regular old "platform engineer" I fight to ignore "data
| platform" tasks. There's no target to hit, it's just moving sand
| around a sand box.
|
| If you want an answer to a specific question, we can spin up a
| read replica and a Metabase and write a query in an afternoon,
| cool. I'll get you a chart, we'll move on. If you want "a data
| analytics platform to enable blah blah blah" I'm out, I can't do
| it. My eyes won't focus, my hands stop moving.
|
| Developers sometimes tell me stuff like "Kubernetes is too
| complex", "jeez React is a pain". I send those quotes to my
| friends stuck writing 195 step DAGs to transform log files from
| s3 into s3 so they can eventually land in s3 - ah yes but they're
| parquet somewhere in between, and that matters for some reason.
| We laugh together, but I can see it hurts them more than I
| intended.
|
| Life is too short to faff about doing nothing. Go join a company
| with less than 100 engineers and learn to be happy again. Let the
| enterprises burn, we'll all be better for it.
|
| Anyways this was a fantastic piece, I hope this person writes
| their book after all.
| spencerchubb wrote:
| > my friends stuck writing 195 step DAGs to transform log files
| from s3 into s3 so they can eventually land in s3
|
| sounds similar to what I do! on my team, it does actually serve
| a purpose, it's just hard to see the forest from the trees
| sometimes
| hermitcrab wrote:
| >Go join a company with less than 100 engineers and learn to be
| happy again. Let the enterprises burn, we'll all be better for
| it.
|
| But but then I won't be able to lease a new panzerwankmobile
| every 2 year to impress people I don't like!
| hinkley wrote:
| In police dramas there's always the judge who refuses to sign a
| warrant and accuses the cops of going on a "fishing
| expedition".
|
| That's half of Big Data to me. Maybe more. No idea what we're
| looking for but we will tell you after the checks clear.
| LAC-Tech wrote:
| This has such strong Australian vibes.
|
| AU dev scene is not great. Really heavy with POs, and PMs and
| CTO's without the background.
| 0xbadcafebee wrote:
| I've worked for like a dozen companies full-time. Most people
| don't know what they're doing. I always thought 'impostor
| syndrome' was a projection of general insecurity. But I've
| started to think it's actually the subconscious saying _" I'm not
| sure what's right or wrong, please consult an expert."_
|
| I have a fantasy of quitting my job to write books on the
| [modern] theory and practice of information systems engineering.
| Not 'how to write software', that's been done; I mean all the
| forms of engineering around software/information systems. In my
| dream, I write the books, everyone reads them, and starts doing
| their jobs right.
|
| But then I remember, I, a person arrogant enough to believe he
| knows how to do things right, still can't get shit done right.
| Maybe if I were a one-man company, I could 'do everything right',
| and feel good about the result. But I depend on an entire company
| of people to do the right thing, in the right way, at the right
| time. That's hard even with the best people. No company is made
| up of the best people. It's always a mix of the best, worst, and
| in-between.
|
| Strangely, a company can put out a decent product, despite the
| company being a tire fire. This is some comfort when you get
| older. You realize that everything being shit is okay, as long as
| the bills are paid. I have PTSD from when the thing that paid the
| bills was on fire, every week, for years. Lately at every job I
| have, I internally panic and scream at how horrible everything
| is. Because I'm haunted by what _might_ happen. But it 's not
| happening yet. So I muffle the screams, smile and nod along with
| the stand-up-meeting-cum-status-update.
|
| The sad thing is, I forget that it's okay that the stand-up is
| shit. I forget that I'm still getting a fat paycheck just to sit
| in meetings that could have been an e-mail. I forget that,
| despite the company bleeding cloud costs [no savings plans, RIs,
| serverless, right-sizing, etc], we seem to be making a profit.
| Despite the terrible designs, bad process, ineffective
| leadership, absentee management, lack of security, and all the
| rest, the bottom line is fine. The shit is fine. Currently, and
| probably for the unforeseeable future.
|
| I get craftsmanship. I'm a crappy woodworker. I enjoy making
| things well, and getting better at it. But our jobs are not fine
| woodworking. Our jobs are construction. We are banging rusted
| nails into shitty, twisted, racked, cupped, knotty-ass studs. If
| we're _lucky_. Yeah, this building is going to be shit. But
| somebody 's still going to pay for it. And there'll be another
| job after. If we really wanted fine woodworking, we never would
| have taken this job, and we know it. We'd be struggling to sell a
| cabinet that took us two 80+ hour weeks, too tired to appreciate
| its beauty, too defeated by flaws only we notice.
|
| So let's stop beating ourselves up. Let's stop beating each other
| up. We don't, can't, won't, find meaning in this monument to
| mediocrity. No comfort from the pain zone. No pride to take home.
| But we are paying the bills, with more left over than most have.
| No broken backs and long hours. No lack of health care, no abuse
| from customers or the public. Not even that big a worry about job
| security. We are the lucky ones. We are blessed with a golden
| shovel. So let's do like those blue collar laborers we often
| idolize, and get to this annoying, bloody awful work that we are
| blessed with.
| torginus wrote:
| I just wonder what would happen to society if everyone worked
| like software engineers do.
|
| For example, what would happen if carpenters just eyeballed
| every measurement, and just shrugged their shoulders when the
| walls didn't line up. If they found out the way the wiring was
| planned will not actually end connecting where they thought it
| would and just shrugged and did it anyway.
|
| What if they had hour long meetings about how to drive in
| screws and their workdays consisted of putting up a beam, and
| then going home because they think they've done enough for the
| day.
|
| If all of society was ran like that, it wouldn't be running for
| long.
| icedchai wrote:
| 100% this. It's still sad to experience this though, day after
| day. I've seen fundamental things done so badly, no time to fix
| them, we're given no choice but to continue "construction" on
| top of a smoking garbage pile. Nobody wants to hear "start
| over."
| hermitcrab wrote:
| That seems a rather defeatest attitude. I was fed up with
| working for other people 20 years ago. I started my own 1-man
| software company and write software how I think it should be
| written. The software isn't perfect, but I'm proud of it. I'm
| not rich, but I do fine financially. It isn't a viable path for
| everyone, but it is something to consider if you hate your job.
| 0xbadcafebee wrote:
| I am defeated. I think a lot of us are. It's wonderful that
| you have your own company, but I don't have the grit and
| self-discipline for it, and I'm still trying to scrape up
| enough for retirement. A younger me might've liked to try
| your path, if I'd heard more of those success stories growing
| up.
| hermitcrab wrote:
| I'm sorry to hear that. It's definitely not for everyone.
| ludicity wrote:
| I'm the author. This is the track we're on. We also named our
| company "Hermit" so there's something extremely serendipitous
| about this comment, hah.
|
| I definitely think it isn't viable for many people, but I've
| also met people who could totally do it that are scared
| because they're around a lot of people with ability/financial
| constraints that they simply don't have.
| hermitcrab wrote:
| Best of luck. One word of advice: marketing. ;0)
| int_19h wrote:
| The other aspect of this is that it's easier to be the cog when
| you're tasked merely with implementing shitty decisions. You
| can just disregard the rationale and focus on the list of
| things you're told to do.
|
| But as you (hopefully) progress in your career, the expectation
| is that you are increasingly the one _making and justifying_
| those lists, which is much more soul crushing and harder to
| "background" and pretend that it doesn't matter.
| tofflos wrote:
| > Like why didn't anyone catch the issue with the logs?
|
| I see questions like these a lot and every time I feel that
| people immensely underestimate the effort required for curating
| data. In my experience data can only ever be as good as what it's
| being used for and in this story the logs haven't been used for
| this purpose before so they're not going to be any good.
|
| It's some sort of data variation on the second law of
| thermodynamics - entropy is winning. Going in with the
| expectation that things should be better will only lead to
| frustration.
| epgui wrote:
| This is not a data curation issue though, it's a basic o11y
| issue.
| andrewflnr wrote:
| > o11y
|
| Seems to be "observability", for anyone else seeing that for
| the first time.
| downrightmike wrote:
| It pops up every 8 months or so for me
| t420mom wrote:
| I feel like there's a whole new generation of tech workers that
| need to read _Zen and the art of motorcycle maintenance_
| ludicity wrote:
| Author here. This is one of my favourite books of all time. In
| fact, my two favourite books are ZAMM and The Black Swan, both
| of which I hated on my first read when I was 19.
|
| I recently re-read it for the third time while on holiday and
| was taking notes on pages worth including in a book review. I
| ended up with about 1/3rd of all pages logged and decided no
| review was better than just telling people to read the book,
| but I might write a blog post on gumption traps.
| nathancspencer wrote:
| _of course, we 're serverless, because how can you hurt yourself
| without a cutting-edge?_
|
| Brilliant
| zbyforgotp wrote:
| Bullshit jobs once again. I don't know. These companies are
| complex systems.
|
| He is writing as if the engineers all knew how to fix the
| systems, but were just powerless to do that. But I've also seen
| projects lead by engineers that only added to the overall
| complexity.
|
| There is a paradox in this - the people who seem the most
| confident about fixing the systems usually only make things
| worse. Chesterton fences and stuff.
|
| This article triggers me because everybody who reads it will
| always believe that they would fix the mess if only they got the
| power, but in practice when they get power they would only add
| new complexity to the whole mess.
| xena wrote:
| People generally do have some idea how to fix complicated
| systems have have endemic problems. The reason they don't is
| that the company considers the capitalism going up faster NOW
| to be way more important than it going up faster in the future.
| mrlonglong wrote:
| This makes me very glad I now write software that drives
| hardware.
| paulsutter wrote:
| Understanding that IT projects are difficult gives us more
| empathy. Gartner says that 80% of corporate IT projects are
| considered failures. McKinsey says that 17% of large projects
| fail so badly that the companies existence is threatened.
| Standish group says only 10% of projects succeed.
| hinkley wrote:
| All large successful projects start as small successful
| projects.
| sethammons wrote:
| > Like why didn't anyone catch the issue with the logs?
|
| Because there were no automated tests. If the company needs
| something to work, that thing needs a, preferably automated,
| test.
| pards wrote:
| > The word enterprise means that we do this in a way that makes
| people say "Dear God, why would anyone ever design it that way?"
|
| Thank you for this phrase; I'll quote it at every opportunity.
| hinkley wrote:
| I'm working on an SDLC app that will end up with inspirational
| sayings as interstitials once I run out of bigger
| features/desperately want to procrastinate.
|
| I'll stick heavy hitters from Goldratt, Fowler, Feynman et al
| in there, but there's going to be a "dark humor" and "snarky"
| category and this will definitely go into one of them, along
| with some Ambrose Bierce.
| tokinonagare wrote:
| > At two of the four businesses I've worked at, the most highly-
| performing engineers have resorted to something that I think of
| as Pain Zone navigation. It's the practice of never working
| unless pair programming [...] The fear and dread comes from a
| culture where people feel bad that they can't work quickly enough
| in the terrible codebase
|
| Exactly why I burned-out at work, worked at most 2 hours per day
| on a good day and finally was ejected from the project after a PM
| that graduated last year from school noticed and went after my
| head. Author is a wizard for describing the situation this well.
|
| It's been 3 days I've been free from the tyranny of Jira and
| project managers, and I worked more on my personal projets than I
| did in a week at my former workplace.
| hinkley wrote:
| > The word enterprise means that we do this in a way that makes
| people say "Dear God, why would anyone ever design it that way?"
|
| I feel this comment in my bones.
| kayo_20211030 wrote:
| Good Grief! Prolix?
|
| > coated in grass which rends those who tread upon it like a
| legion of upraised spears,
|
| and there's more. Get to the point, whatever it is.
| dcminter wrote:
| Some of us enjoy the journey.
| hinkley wrote:
| This is the "I saved my company half a million dollars in about
| five minutes" person who got in trouble for saving them half a
| million dollars.
| ludicity wrote:
| You are the only person in months that has called me anything
| other than "the AI rant guy" and I just wanted to say I
| appreciate you. <3
| hinkley wrote:
| I am an AI agnostic so even if I remembered that was you too,
| I'd only say it to our tribe.
|
| I came into the industry exactly as a previous AI Trough of
| Disillusionment was in full swing so I've been vaccinated.
| sampo wrote:
| The "AI rant": https://news.ycombinator.com/item?id=40725329
| sourcepluck wrote:
| I had read and very much enjoyed the "AI silence or you'll be
| Piledriven into next week" post, without clicking on anything
| else on the author's blog. It was click link, read whole thing,
| love it, send it to one or two people, move on.
|
| Very happy to see this here, realise it's the same person and
| that this is "a thing", and then to rollick in the author's
| backlog. A joy! Raucous real-life laughter has exploded from me
| on numerous occasions along with most articles. I think I've read
| 5 in a row there, and my brain is buzzing happily.
|
| Thank you to the author for having the courage to write about
| real experiences. A breath of fresh air. I look forward to future
| books and articles, and reading more previous work, and cross my
| proverbial fingers hoping they can keep it real in the face of
| what will presumably be an avalanche of grifters looking to leech
| off the attention.
| refulgentis wrote:
| for fellow readers: https://ludic.mataroa.blog/blog/i-will-
| fucking-piledrive-you...
| sourcepluck wrote:
| And, adjacent, in my opinion, readings like this piece,
| entitled "Hallucinating sense in the era of infinity-
| content":
|
| https://www.documentjournal.com/2024/05/technical-images-
| fil...
| refulgentis wrote:
| This is really, really, fucking good, thank you.
| hermitcrab wrote:
| I did suggest this excellent link to a well known data
| science newletter, but they said it was a bit too profane for
| their readership!
| jacobyoder wrote:
| "Suffice it to say that while people are sincerely trying their
| best, our leaders are not even remotely equipped to handle the
| volume of people just outright lying to them about IT."
|
| I've tried to come up with some heuristic to determine whether or
| not a team is competent, good, or doomed. I've been exposed to
| all over the last... 8-10 years, and one of the key things I've
| noticed is the ratio of competent/skilled developers to the
| unskilled ones is a big ... indicator(?). Predictor?
|
| Colleague of mine has been working with a team - dev team has
| ranged from 5-8 people over the last few years. Few people seem
| to have any grasp of programming at all. Only two people - my
| colleague and one other - have ever taken projects from ideas to
| delivery, or even taken features from requests to successful
| rollout of already functioning software.
|
| The arguments that people get in to there - days or weeks of
| people 'researching' whether or not OAUTH 'really' requires
| 'refresh tokens' or whether it's really supposed to be a JWT.
| Management has some notion of 'every voice is legitimate and
| should be heard - we don't support bullying' and so on.
|
| If you have a team of 10, and 1 or 2 people are simply bad at
| having the ability to think somewhat abstractly, you can survive.
|
| If that number hits, say, 4-5... the team will struggle. A lot.
| You can keep things going, but it will be slow. And everything
| becomes a battle.
|
| If that number becomes 7 or 8, and you only really have 1-2
| developers who are actually competent developers... things will
| continue to spiral downward.
|
| On the other side - I worked with a team of about 8-10 people on
| a 6 month contract. The larger org had another 40 or so folks,
| handling other projects, and support. Onboarding was _great_ - I
| pushed production code in the first week. Everyone on the team
| was competent, including the juniors. I had more development
| experience, but they had more company experience, and it was
| really a relatively enjoyable engagement overall.
|
| It was refreshing to be able to ask anyone on the team questions,
| and either get a workable answer, or an "I'm not sure, let's
| check with XYZ" to get working answers. The "oh, yeah, it's ABC"
| when ABC is clearly not the answer stuff never happened. People
| committing code and pushing to production _without ever having
| run the code at all_ - I 've experienced that - didn't happen -
| that's happening to my colleague.
|
| The problem with a plurality of tech-incompetent folks in a tech
| group is that they honestly can not determine that they aren't
| competent. The only examples of competence are in the minority,
| and tend to not be trusted (even though that minority is the only
| portion that turns out working/functional code).
|
| Leaving ends up being the only option in those cases. My
| colleague is only at his place part time, and has hung around
| because they've gone through some restructuring where new folks
| were brought in, and... you hope that things might get better in
| a few months, then realize they don't.
| javajosh wrote:
| Tempting to write a response piece, "Get me into data hell, I
| need the money."
| ludicity wrote:
| In Australia, data jobs have a high floor but a low-ish ceiling
| at most places. Salaries start at around A$100K if you know how
| to interview, but 30 years of diligent practice only nets you
| around A$200K, and this would be considered an extreme success.
|
| If you're already good, just have coffee with executives/leads
| at places that take engineering seriously and you'll earn well,
| but there's no need to enter data hell as an employee at the
| typical company.
| ramshanker wrote:
| Acting against your own better Judgement, I have to take that
| fight/ urge every few days. ;)
___________________________________________________________________
(page generated 2024-11-03 23:00 UTC)