[HN Gopher] Get me out of data hell
___________________________________________________________________
Get me out of data hell
Author : pavel_lishin
Score : 120 points
Date : 2024-10-31 19:01 UTC (2 days ago)
(HTM) web link (ludic.mataroa.blog)
(TXT) w3m dump (ludic.mataroa.blog)
| jaygreco wrote:
| I really like the author's writing style here. The quips about
| the tea especially.
| remram wrote:
| It reminds me of the classic series on /r/talesfromtechsupport
| by airz23. It's uncanny.
|
| They would almost always feature a "I sip my coffee. It tastes
| like <feelings at the time>" Example:
| https://www.reddit.com/r/talesfromtechsupport/comments/203qx...
| ctippett wrote:
| > of course, we're serverless, because how can you hurt yourself
| without a cutting-edge?
|
| A beautiful epigram.
| storafrid wrote:
| Yes, but I'm surprised that they attribute "cutting-edge" to
| Lambda. It's about as old as Docker.
| salt-thrower wrote:
| Beautifully written and fun to read. Blog posts like this give me
| a boost of mental strength to keep going during my worst episodes
| of burnout.
| reverius42 wrote:
| > I've even degraded team morale because I've convinced some of
| the engineers that things should be better, but not management,
| so now some of the engineers are upset.
|
| Oof, that hits a little close to home.
| SynasterBeiter wrote:
| I hate whining like these. Just do your job and get over it. No
| need to be theatrical about it.
| incognito124 wrote:
| A creative outlet is a need, to some. You sound like OP's
| employees.
| baq wrote:
| Let the man _live_ will you. Not everyone wants to be a drone
| for hire.
| hobs wrote:
| If this was a job with a non-abstract input and output process
| they would have OSHA saying it was genuinely unsafe (and
| definitely stupid.)
|
| Many of us build systems to manage automatic actions to take
| care of this stuff, if I had an engineer I had to take 100+
| steps to get something done I would definitely be considering
| 1) What the hell am I paying for and 2) Why the hell am I
| paying for it?
| hprotagonist wrote:
| you know, he had a post about exactly this comment not too long
| ago!
|
| edit: https://ludic.mataroa.blog/blog/the-violent-role-of-
| relentle...
| hipadev23 wrote:
| Complacency is how this shit happens.
| brianhorakh wrote:
| Wonderful prose. I am in Melbourne also. I possibly used to work
| at the same place but I'm not sure.
|
| I resigned due to the night terrors caused by the cyber security
| issues I saw everywhere. The more I explored and understood the
| more sleep I lost.
| zombiwoof wrote:
| Data "engineering" is where all the cool kids go with no clue and
| create insane architectures to justify their incompetence
| hobs wrote:
| As a data engineer I have seen absolutely bullshit pass for
| production, but it doesn't seem that different from all the
| other bullshit I have seen people deploy in my life.
|
| It is one of the few types of jobs I have worked were someone
| credulously offering adding five more layers to fix an issue
| with latency is a normal operating procedure though.
| parpfish wrote:
| data engineer gives data science a run for their money when it
| comes to ambiguous job expectations.
|
| I've seen it mean anything from "distributed computing expert"
| to "knows SQL"
| marcosdumay wrote:
| > to "knows SQL"
|
| There are still ways to go. I've seen it means "spends the
| days filling excel spreadsheets".
| jitl wrote:
| I wonder what company they're describing here. It sounds like so
| many self inflicted problems that that you could undo or set
| right in a couple of weeks if you had the time and latitude to
| make changes across the system instead of being confined to a
| small area of team ownership.
| bartread wrote:
| I worked somewhere that had a lot of this sort of thing going
| on once. You cannot overestimate how hard it is to get anything
| done: politics and organisational dysfunction, not to mention
| that you probably don't have access to half of what you need to
| in order to fix any given problem and are even more unlikely to
| be able to get it, mean there are just huge scads of problems
| that, on the face of them, look relatively straightforward to
| solve but which, in practice, are organisationally impossible
| to solve.
| whstl wrote:
| Yep. I don't work with data engineering, but from hearing war
| stories from them, this could perfectly describe the last
| four companies I worked at. :/
|
| _> you probably don't have access to half of what you need
| to in order to fix any given problem and are even more
| unlikely to be able to get it_
|
| I once sat down with a data engineer to try to fix a specific
| problem they had and that was 100% accurate. They were left
| to die by Ops and CISO.
| knallfrosch wrote:
| You're operating in the same company that let this happen in
| the first place.
|
| And -- not to take away from the piece - If noone noticed the
| logs have been garbage for two years, they can't be that
| important.
| bartread wrote:
| I'm going to read the rest of this. I'm enjoying it. But,
| simultaneously, part II has me so triggered - it bears striking
| resemblance to repeated situations I've encountered where the
| meaning and content of columns in a relational database were
| overloaded in varying degrees of heaviness (which is a practice I
| absolutely detest) - that I need to take a short break.
| lifeisstillgood wrote:
| >>> pretending that any of this is more important than hiring
| competent people and treating them well. I could build something
| superior to this with an ancient laptop, an internet connection,
| and spreadsheets.
|
| Ow
| iamthepieman wrote:
| I do not use this term to refer to myself. I respect those who do
| and respect the meaning behind it but am just old enough that it
| feels alien to me 99% of the time.
|
| But I am SO triggered by this piece. I had that intrusive feeling
| you sometimes get when driving where you think, "I could just
| close my eyes and see what happens", "Or that clif is so close
| and the guardrail doesn't really extend far enough"
|
| Only for my career. Like I should just not show up on Monday. I
| should get in the car and drive far away and change my name and
| work at a nice retail joint in a mid-sized town.
|
| I'm going to need to sit and stare into the distance for an hour
| and 3.
| jauntywundrkind wrote:
| The observability world still regards itself as a system for
| monitoring, but reading (and sometimes seeing) how these systems
| just go so bad continues to drive a conviction that perhaps their
| strategies and tools should become bigger. That they should
| converge with business pipines.
|
| We shouldn't just have wide events/big spans emitted... We should
| have those spans drive the pipeline. Rather than observability
| being a passive monitoring system, if we write code that reacts
| to events we are capturing, then we shuffle towards event
| sourcing.
|
| Given how badly coupled together with shoestring glue & good
| wishes so many systems are, how opaque these pain zones are, it
| feels like the centralization upon existing industry standard
| protocols to capture events (which imo include traces) is a clear
| win.
|
| (Obvious downside, these systems become mission critical,
| business process & monitoring both.)
| halfcat wrote:
| What's this look like in practice? Is this something like
| business process modeling and workflow engines, or something
| else?
| halfcat wrote:
| What's the solution to wrangling these data projects?
|
| The author's experience is not far off from my own.
|
| 1. Any solution in place can only be understood by the person who
| created it
|
| 2. _"No, we can't change that because then we'd have to validate
| everything from scratch again"_
|
| And therefore, as the author says:
|
| > _"we 'll continue with the work instead of fixing the critical
| production error"_
|
| I'm honestly not sure how to address it either. With traditional
| software dev we'd write tests, incorporate those into CI/CD, and
| start to course correct. We can use sample data to validate the
| code does what we think it does and that we didn't break it.
|
| But in these data projects, it's not only the code that's
| changing, but the data is also a moving target. You can write a
| test with sample data, but tomorrow your data might change
| because someone in sales added a custom field to the CRM, or IT
| upgraded the accounting software and all of the unique IDs
| changed, or someone upgraded their Excel version, or whatever.
|
| And your code that works on the sample data needs to handle all
| of this, which obviously it can't. You can try to validate the
| data somehow, check the schema, check if the number of rows
| hasn't doubled or halved, and so forth, and then stop it from
| importing until you look into it, but also you can't stop inbound
| data because an exec has a meeting in a few hours and expects
| their report to be updated.
|
| I heard something about "data contracts" that's supposed to
| address this, but it sounds like the next in a long line of buzz
| words intended to get management to buy another data product.
|
| Has anyone worked in this kind of project that went well?
| pxc wrote:
| This blog post rescheduled all my appointments, tucked me in,
| sang me a lullaby, then woke me up with coffee and breakfast late
| the next morning. I am healed.
|
| For real, a fun and refreshing read (if also a little haunting).
___________________________________________________________________
(page generated 2024-11-02 23:00 UTC)