[HN Gopher] I accidentally saved my company half a million dollars
___________________________________________________________________
I accidentally saved my company half a million dollars
Author : softskunk
Score : 648 points
Date : 2023-10-30 14:15 UTC (8 hours ago)
(HTM) web link (ludic.mataroa.blog)
(TXT) w3m dump (ludic.mataroa.blog)
| j0hnyl wrote:
| Once upon a time I uncovered a bug that recovered $4MM/year in
| revenue. It was swept under the rug to protect the team and
| executives that let the blunder continue for as long as it did. I
| didn't get a raise, but I made some allies and got to coast for a
| while.
| chasd00 wrote:
| I did something similar early in my career but it very much
| went noticed. The system I wrote to correct the bug was
| basically a MITM attack against pharmacy rx transmissions, it
| re-priced prescriptions just before they went to the insurance
| carrier's system because the pharmacies would never apply price
| updates (i worked at a small independent pharmacy chain, they
| didn't have a central dispensing system). In my infinite genius
| I named the system after my current crush. Corporate liked it
| so much they made an annual award named after my system... so
| therefore after my crush but i was too embarrassed to tell them
| the real origin of the name. That was 25 years ago but, to this
| day, every year a trophy gets made with the first name of my
| old crush printed on it and handed out. heh if she ever knew
| she'd be mortified.
| busterarm wrote:
| Very early in my career I discovered some mission-critical
| network devices at the hedge fund I worked at that would have
| caused a network loop if either of the two machines were ever
| rebooted. The two servers were related to some feed data and I
| had noticed that they were cabled together oddly based on how
| their function was described to me.
|
| The estimated cost of downtime for these systems was something
| like $7 million per minute. I had raised the issue to a couple
| of the staff responsible for the machines and to the networking
| team but was completely dismissed because "there is no way we
| would have hooked them up that way" and because I was the FNG.
|
| I then raised the issue again at the weekly group meeting
| because it seemed important -- somebody was dispatched to check
| visually and came back to confirm what I said. It was a big
| deal -- the networking team had about 2 weeks of emergency work
| to do to resolve the issue cleanly.
|
| EVERYONE was angry at me. Even though I had just averted a
| catastrophe for the company, I made everyone look bad by doing
| it and particularly because of my status/position on the team.
| It was an important lesson learned.
| tehlike wrote:
| Keep doing the right thing.
| busterarm wrote:
| It's 20 years later and I've made my career off of pointing
| out fundamental mistakes made by very smart people. ;)
| (just with more skill and tact)
|
| 90% of what I do is ask dumb questions as if I'm completely
| clueless.
| tehlike wrote:
| I know the feeling. I'm similar in that regard. Debugging
| complex things by asking dumb questions is extremely
| fruitful.
|
| Also asking dumb questions to newer engs teaches them to
| think different aspects of a problem themselves.
|
| One of the things I'm hoping I could keep alive in my
| child
| phkahler wrote:
| >> EVERYONE was angry at me. Even though I had just averted a
| catastrophe for the company, I made everyone look bad by
| doing it and particularly because of my status/position on
| the team.
|
| People can be excessive in both taking credit and placing
| blame. An appropriate and helpful way to frame this is "the
| system was configured incorrectly but nobody noticed because
| the problem never actually happened. It's a good thing the
| new guy had time to go through things and spotted the problem
| before it ever happened." No need to crucify the team or
| exaggerate the value of the new guy.
| mablopoule wrote:
| On his book "The Secrets of Consulting", Gerald Weinberg
| advise against improving more than 10% of performance, and if
| so, of hoping to have any credit.
|
| Just like the article, his reasoning is that if you improve
| performance _too much_ , it makes management/the team look
| bad for not doing it before, while a smaller improvement in
| performance make management looks good.
| biugbkifcjk wrote:
| I'm only just realizing now why I was treated so strangely
| after discovering a pretty severe security issue that had
| been in our software for about 7 years. I had only been in
| the team for about 12 months when I discovered it.
| koliber wrote:
| The Secrets of Consulting", Gerald Weinberg advise against
| improving more than 10% of performance, and if so, of hoping
| to have any credit. Just like the article, his reasoning is
| that if you improve performance too much, it makes
| management/the team look bad for not doing it before, while a
| smaller improvement in performance make management looks
| good.
| davedx wrote:
| Incredible.
|
| Meanwhile big tech thinks the way to reduce costs is to wholesale
| fire 1/2 their company.
| pavel_lishin wrote:
| For what it's worth, OP also recommended that approach.
| mcv wrote:
| Yeah, but corporations rarely fire only the incompetent
| people. Quite often it's the best paid engineers who get
| fired first because that saves the most money. And when
| there's a voluntary get-out program, it's the unappreciated
| competent people who are the most eager to leave.
| goalieca wrote:
| The two are not exclusive. Companies can run hundreds of
| programs many of which are not productive and actually hurt and
| frustrate engineering teams. Just in the same way government
| can hurt the environment by throwing up a lot of red tape on
| green initiatives, companies can hurt quality and productivity
| by spending thousands of hours on things like employee annual
| reviews.
| pavel_lishin wrote:
| This person is a good writer, I'm looking forward to seeing what
| else they've written.
| bcjordan wrote:
| It is really what you want out of a hacker-stuck-in-corporate
| story.
|
| There was a tale maybe 10+ years ago about someone who
| automated their job with a script or Excel sheet or macro and
| didn't tell anyone about it. Having a hard time tracking it
| down again, anyone remember what that was?
| pavel_lishin wrote:
| Google reveals two things, followed by a bunch of shitty
| blogspam:
|
| - Last year, there was a post on Reddit, with an HN
| discussion as well:
| https://news.ycombinator.com/item?id=29994776
|
| - In 2017, there was a question about whether someone should
| tell their employer that they've automated their job:
| https://workplace.stackexchange.com/questions/93696/is-it-
| un...
| orzig wrote:
| I've found LLMs to be good for those sorts of vague "remember
| that thing like this?" searches. Bing (with GPT) or even Bard
| with potentially better search integration are both worth a
| try.
| alex7734 wrote:
| In a sane world this would have to be satire, but unfortunately I
| believe every single word of this
| tgv wrote:
| A writer cannot make up so much shit. You need a board of
| managers for that. Effing hell.
| vwoolf wrote:
| This reminds me of some of Dan Luu's stories,
| https://danluu.com/nothing-works/
|
| _Likewise with chip software tooling; despite it being standard
| to outsource tooling to large EDA vendors, we got a lot of
| mileage out using our own custom tools, generally created or
| maintained by one person, e.g., while I was there, most simulator
| cycles were run on a custom simulator that was maintained by one
| person, which saved millions a year in simulator costs (standard
| pricing for a simulator at the time was a few thousand dollars
| per license per year and we had a farm of about a thousand
| simulation machines). You might think that, if a single person
| can create or maintain a tool that 's worth millions of dollars a
| year to the company, our competitors would do the same thing,
| just like you might think that if you can ship faster and at a
| lower cost by hiring a person who knows how to crack a wafer
| open, our competitors would do that, but they mostly didn't._
| cybernoodles wrote:
| I saved Amazon $10MM as an intern back in 2012. If only I could
| have seen 1% of that.
| shepherdjerred wrote:
| I also saved Amazon a ton of money and saw nothing from it!
| gottorf wrote:
| On the other hand, employees aren't liable for the company's
| losses and debts, so it works out in the end.
| TuringNYC wrote:
| In trading/PM roles in finance, before clawbacks became
| more popular, employees would regularly get the upside and
| avoid the downside.
| searchableguy wrote:
| Are clawbacks common at trading companies?
| foxyv wrote:
| > employees aren't liable for the company's losses and
| debts
|
| Never been laid off during a recession or had your pay
| frozen and bonuses cancelled during a hard time? Employees
| risk a lot more than most stock holders by working for a
| company. On average, stock holders are way more
| diversified.
| throwaway98797 wrote:
| losing your job is not the same as losing capital
| mao_tse_tung wrote:
| Correct, is way worse. Proletarians don't have any
| capital to lose. Capitalists do. So if they lose their
| capital, they can just be like the rest of us. If we
| (proletarians) lose our job, we risk poverty and death.
| anonymouskimmer wrote:
| > So if they lose their capital, they can just be like
| the rest of us.
|
| Nah, if they had enough capital to live off of before
| then, they are far more likely than us to fail upward
| into a management job.
| foxyv wrote:
| No, it is wayyyy worse. Although there is a lot of
| capital investment in a job. You are also investing the
| most valuable thing you own.
| geodel wrote:
| > Employees risk a lot more than most stock holders by
| working for a company.
|
| hmm, they can perhaps reduce the risk by not working for
| a company. They can just be stock holders or launch their
| own company, that way whatever may happen they will never
| get fired.
| birdyrooster wrote:
| > They can just be stock holders or launch their own
| company
| foxyv wrote:
| In order to become a stock holder or launch a company,
| you need to have capital. Capital is hard to acquire when
| wages are being actively suppressed by a cabal of
| employers. It is also hard to hold on to capital when
| your health care system is intentionally designed to
| strip away generational wealth from workers. Then to add
| a cherry on top, you lock the higher wage jobs behind an
| additional investment that can only be funded by non-
| dischargeable loans. Then you have the rent seekers, both
| literal and figurative. Landlords, insurance companies,
| toll roads, etc...
| martin8412 wrote:
| Pay frozen? Absolutely not. The company would be
| undergoing bankruptcy by the end of day if not all
| employees agreed to it.
| vdqtp3 wrote:
| Pay frozen != no pay
|
| Pay frozen == no pay raises
| 93po wrote:
| This isn't a risk the employee takes on as a result of
| doing business. It's a result of the company _choosing_
| to do this while also still making profits. So it 's not
| really a risk, it's just mistreatment.
| deepsun wrote:
| > Employees risk a lot more than most stock holders by
| working for a company.
|
| Uhm no?
|
| You apparently never been a business owner. Employees get
| their wages, and even can legally enforce them. If a
| business go down, owners eat the losses and envy their
| employees.
|
| I've been on both sides. Being a business owner is much
| riskier.
| dvt wrote:
| > On the other hand, employees aren't liable for the
| company's losses and debts, so it works out in the end.
|
| How can anyone seriously type this? If you fuck up bigly
| enough, you will 100%--without fault--get sacked. Again,
| not even talking about long tails (bad economic conditions,
| layoffs, etc.).
|
| This is under totally normal situations: if you lose the
| company money, you will be fired. As a bonus, you also lose
| unvested options or equity. These kinds of posts are
| exactly why engineers have garbage bonuses compared to
| finance even though they probably generate an order of
| magnitude more value.
| chii wrote:
| > If you fuck up bigly enough, you will 100%--without
| fault--get sacked
|
| which is to be expected - making a big mistake might not
| be something that can be forgiven and overlooked
| (depending on the magnitude of the mistake).
|
| But you will not lose capital as an employee, since you
| did not put in capital to lose. Your time would still
| have been paid, up to the day you are fired.
|
| Therefore, you obviously have no incentive to take on a
| risk that can result in a mistake (but which the reward
| you take no part in). You just do your assigned job, and
| whether it saves the company money or not, as long as you
| can cover your ass, you're golden.
|
| Unless the company incentivize you to save money - for
| example, via a bonus through hitting a target or
| achieving some goal that was set.
| dvt wrote:
| > But you will not lose capital as an employee, since you
| did not put in capital to lose.
|
| The conversation is a lot more complicated because
| there's an opportunity cost, you lose time (your time is
| finite, company time is infinite), you lose reputation,
| and so on. Besides, your argument is a bit weak as it's
| not like hedge fund managers put up the cash themselves,
| either.
|
| My point is only that value-generators should be rewarded
| as such, and it's a bit weird that engineers are totally
| cool with not getting a piece of the pie.
| gottorf wrote:
| > My point is only that value-generators should be
| rewarded as such, and it's a bit weird that engineers are
| totally cool with not getting a piece of the pie.
|
| Software engineers are some of the best-paid labor in the
| world with great benefits and workplace conditions. They
| often receive equity as a compensation, even when the
| salary is still vastly above many other lines of work.
| They are absolutely getting a piece of the pie, and in
| much greater proportions than almost any other economic
| activity.
|
| You may be discounting the value of capital, management,
| sales, and other roles in a successful software-related
| business.
|
| The remuneration that labor and employees receive is
| never going to be in line with the value that they
| generate, precisely because the former group doesn't take
| any risk. They don't invest any personal capital and they
| aren't liable for anything. They can walk away any time,
| sometimes voluntarily, sometimes not. In return, they
| work fixed hours and get paid on a routine basis. The
| owners receive only what remains above and beyond all
| that, which could be great profits, just breaking even,
| or even losses.
|
| > there's an opportunity cost, you lose time (your time
| is finite, company time is infinite)
|
| Everyone everywhere loses time, because time passes
| whether or not you choose to do anything with it.
| Employees aren't unique among economic entities that they
| face opportunity costs.
|
| > it's not like hedge fund managers put up the cash
| themselves, either
|
| This is actually a good example to dive into. Hedge funds
| are typically paid "2/20", meaning 2% of assets under
| management every year whether or not there are any gains,
| and 20% of any gains above some benchmark. It's similar
| to, say, a commission-based sales role that gets paid a
| certain fixed salary and a percentage of sales they make.
| Whether or not 2/20 is "fair" is solely up to those who
| buy their services, since there is a competitive market
| of providers of fund management (the "employee") and
| providers of capital (the "employer").
|
| And in some situations, the "employers" do in fact lose a
| lot of money, while the "employees" walk away; the
| limited partners of Melvin Capital, for example, lost
| many billions of dollars, all while Melvin Capital itself
| continued to charge the 2% management fee.
|
| And within hedge funds itself, there are again employees
| who receive a stable salary and maybe some performance-
| related bonuses on top of that, versus the principals and
| owners who have personal capital invested. When LTCM blew
| up, for example, it's estimated that its owners lost
| $1.9B[0].
|
| [0]: https://en.wikipedia.org/wiki/Long-
| Term_Capital_Management
| anonymouskimmer wrote:
| > But you will not lose capital as an employee
|
| It depends on how much you mess up. Mess up large enough
| as an employee and you can end up sued by your former
| employer. Losing a lawsuit is losing capital. A probably
| not comprehensive list of reasons an employer can sue an
| employee, not all of which are because of negligence or
| malfeasance: https://www.mylawteam.com/employment/can-an-
| employer-sue-an-...
|
| Depending on the state you can also have your pay docked
| (if that's not a capital loss, at least for
| transportation costs, then I don't know what is):
| https://www.avvo.com/legal-library/employment-
| law/paycheck-d...
| robertlagrant wrote:
| You've completely missed the point. Businesses can lose
| money for all sorts of reasons. Owners have to eat the
| losses while keeping on paying salaries.
| anonymouskimmer wrote:
| No they don't, they can instead fold the company and sell
| it off in parts.
| robertlagrant wrote:
| That's absolutely the last resort and worst thing to
| happen for them.
| anonymouskimmer wrote:
| Sure. They could instead split the company and dump all
| of the poorly performing assets and debt into the split
| off company and on the bond holders.
|
| https://en.wikipedia.org/wiki/Texas_two-step_bankruptcy
|
| https://www.businessinsider.com/corizon-health-
| bankruptcy-ye...
|
| > If successful, Corizon's Two-Step would avoid a much
| wider range of liabilities than previous companies who've
| used it -- not just injury lawsuits, like J&J, but the
| routine debts to vendors that companies rack up every
| day. If the company succeeds, it provides a "roadmap for
| eliminating virtually any unsecured liability owed by any
| corporate entity, regardless of whether that entity is
| solvent," Ian Cross, a Michigan civil-rights attorney who
| represents multiple prisoners who have sued Corizon,
| wrote in a procedural objection in April.
|
| or do a leveraged buyout in which: https://www.investoped
| ia.com/articles/markets/111015/10-most...
|
| > The goal of leveraged buyouts is to make a large
| acquisition _without committing much capital_ investment.
| tshaddox wrote:
| Who is liable for the company's losses and debts, I wonder?
| tiffanyh wrote:
| Sounds like you should move into sales if you want a % paid
| on value you help a company.
| ipaddr wrote:
| We're all in sales
| shepherdjerred wrote:
| It just feels strange to save a company a huge sum of money
| (by my own initiative) and not see a penny.
| tiffanyh wrote:
| Flip side, are you ok if your company docked your pay if
| they found you wasteful?
|
| E.g. why didn't you turn off that temp m3.xlarge
| instance. $X gets docked from your pay.
| rvnx wrote:
| In some way, I made Amazon get a lot of profit by choosing to
| purchase on their platform, and saw nothing from it, though I
| directly contributed to their profit and could deserve a %.
| microtoodle wrote:
| I once saved a company $20k in infra costs and saw nothing of
| it.
|
| In fact the team was pretty upset that they'd budgeted that
| money for infra already and it'd have been better spent instead
| of waiting till next year to re-budget it.
| busterarm wrote:
| I once talked AWS into a 5-digit refund for something that
| was our team's mistake.
|
| It wasn't necessary though because we had a committed spend
| target to reach and we just had to figure out how to
| legitimately spend the money somewhere else. :(
| WediBlino wrote:
| My company paid a consultant PS25K to increase efficiency. He
| recovered about 5 minutes a job.
|
| I wrote a tool that saved about two hours a case, in total this
| saved about PS500k. I got a free case off beer.
| kakwa_ wrote:
| You are lucky, I got nothing for heavily suggesting a
| ~1M/year saving.
|
| The only "personal" reward I get from that is: whenever I
| feel guilty for not having done much in a given day, I remind
| myself that by this action alone, I've saved my company
| several times what I would ever cost them.
|
| Helps with self-esteem, but I don't think my company see it
| that way.
| switch007 wrote:
| Are you a consultant now?
| oh_sigh wrote:
| I saved/earned Amazon $25M/yr also back in 2012. Because the
| project that was supposed to compute price matching/most-
| favored-nation status for vendors was never actually
| implemented. Despite it being in the standard contract for
| vendors for years, no one ever noticed that we never adjusted
| prices based on it. My own initiative noticing the problem, my
| own design and implementation, as a L3(or whatever the fresh
| grad role level is), and I got zip for it. Big part of my
| reason for leaving the company. I didn't expect people to fall
| to their knees and worship me, but it seemed like a project
| that should be a big part of a promotion, but I was passed over
| multiple times.
| rybosworld wrote:
| TLDR: Bureaucracy is everywhere.
| jack_riminton wrote:
| "this whole department, like many departments, is some sort of
| weird political PsyOp to get executives promoted. It's cosplaying
| as a real business and the board thinks the costume is
| convincing."
|
| Came for the engineering, stayed for the blisteringly on-point
| observations of corporate life
| andrewstuart2 wrote:
| At my previous $JOB I started calling this Promotion-Driven
| Development because it's everywhere. Take the easy problems
| that look good and you can finish quickly, get promoted, hand
| off the facade to team that has to actually solve the real
| problem, and repeat.
| dmitrygr wrote:
| This is very much how Google is on the inside. And that's not
| surprising. People respond to the incentives presented. If
| you spend years fixing hard problems that actually affect
| users, and don't get promoted, you'll almost certainly become
| jaded, and start doing things that are pointless but do get
| you promoted. Because that's how you make more money. And
| that's the actual reason to be working.
| andrewstuart2 wrote:
| And yeah, it shows via ye olde Google graveyard. Honestly I
| do think it's just a hard problem not to have, because the
| world tends to work on visibility combined with innovation.
| You can't solve hard problems and make money if nobody uses
| your solution because they don't know it exists. But I
| struggle with the balance because yeah, I'm a bit jaded.
| syntaxing wrote:
| I think the crappy part is that it works most of the time.
| shalmanese wrote:
| The purpose of a system is what it does. _All_ development is
| promotion driven development, different people can either
| choose to be clear eyed about this or not and, subsequently,
| good at it or not.
|
| Your job is the venn diagram intersection of an organized
| system of rewards and punishments from both inside and
| outside of the firm intersected with your own goals and
| aspirations over time. Anything not in this intersection is
| _not a job_ , it's something else we like to delude ourselves
| into thinking is a job to preserve our egos.
|
| More specifically, your worry about how distorted incentives
| lead to poor quality software is _not your job_ unless it
| _is_ your job to worry about said incentives. People keep
| hoping to go work for one of the magically sane company where
| all the incentives are correct and everyone is doing the
| quote-unquote "right" thing at all times. Such a company
| does not exist because the very concept is incoherent, every
| system of possible incentives will come with a different set
| of tradeoffs and the art of operating a firm is to pick from
| amongst a bunch of shitty options for the least shitty one.
|
| People who do not accept this will be perpetually unhappy and
| bitter about how others obtain undeserved success because
| they simply did the unsporting thing of playing the game they
| were asked to play. You can either be one of those people or
| you can receive the radical acceptance of what a job is.
| Spivak wrote:
| I like to think I'm pretty good at this game, swung two
| promotions and raises by working every request from outside
| the team out of band to the detriment of my main flow of
| work -- "everyone loves you and your work is constantly
| noticed by $upper_management." Wow that's crazy, it's just
| nice to be appreciated I guess wink wink.
| em-bee wrote:
| _People who do not accept this will be perpetually unhappy_
|
| or they go start their own business to get away from this
| nonsense.
|
| when i deal with clients i actually have to deliver
| something because i am not getting any promotions.
| shalmanese wrote:
| Client work is its own unique hell of misaligned
| incentives. You've just replaced one boss with a dozen
| bosses, all of which have the power to "promote" or fire
| you.
|
| I'm not saying that any one individual can't find a job
| that is the unique perfect blend of incentives _for them_
| , indeed, the entire point of this framework is that
| alignment between your goals and the incentives of your
| job is the most powerful lever an employee can pull. But
| that incentive structure will necessarily make others
| working there deeply unhappy as its unique choice of
| tradeoffs is just shitty in a way invisible to you.
| em-bee wrote:
| _all of which have the power to "promote" or fire you_
|
| that's fair, because i also have the power to fire them.
|
| loosing one client when i have a dozen others is not a
| big deal.
|
| with so many bosses i can focus on the good ones and
| reject working with the bad ones. i can also unilaterally
| raise my fees, especially for bad clients.
| OkayPhysicist wrote:
| Not all jobs are promotion driven, at all. My current dev
| job is for a small family biotech business, and the only
| path upwards from where I currently am would involve church
| bells and a flower girl.
| shalmanese wrote:
| But there are still things you need to avoid that would
| get you fired if your performance dropped and there are
| still things you can do that would improve or decrease
| your job prospects on the open market if the company ever
| goes under or is forced to let you go. I'm using
| "promotion" in a more abstract sense than literal
| promotions.
| mring33621 wrote:
| I do not like it. And I personally (mostly) fight against
| it. But, I believe that this person is correct.
| chunkymilk wrote:
| My last place also called it promotion-driven-devevlopment.
|
| It got to the point where some dude built a system that
| completely floundered but got him a promotion. He then re-
| wrote it so it sucked slightly less and got another
| promotion. Last I heard he was working on v3.
|
| Never mind that someone with actual domain knowledge would
| have either not built the system (since it didn't _really_
| need to exist) or would have built a much simpler/more
| reliable system to get the job done.
| its-summertime wrote:
| He did it like that because thats what he was asked to
| do, metrics-wise
|
| His boss is happy with it because it looks good for him
| too
|
| The company is happy because it allows them to sell
| milestones to themselves and or customers.
|
| Customers are happy because the sales-people took the
| right person out to a fancy dinner.
|
| I don't feel like I could fault anyone acting their part
| in the play.
| chunkymilk wrote:
| Can't disagree. He's was doing what the incentives in
| place said he should be doing.
|
| I would find it boring/tedious to keep rebuilding the
| same thing since I'm more of a solve the problem,
| operationalize the solution, then move on person.
| hansvm wrote:
| Ah, but he "de-risked" the project by getting it out the
| door quickly, he started "delivering immediate value"
| despite the rough edges, and "has the technical acumen to
| optimize it further."
| dbalatero wrote:
| I got downleveled in a job offer recently after being
| interviewed about a project which I labored to make as
| simple and efficient as possible. They said after careful
| evaluation, the project wasn't technical enough to merit
| the title I wanted. Live and learn!
| rvbissell wrote:
| This has to be one of the most insightful comments I've
| ever read.
| lowbloodsugar wrote:
| Sure. If one is at a company that is entirely run by
| incompetent sociopaths, then get with the program, right?
|
| But not all companies are sociopath boondoggles. Some are
| companies where people are _trying_ to do the right thing
| (make money, grow business), and management would _prefer_
| to be successful. In those environments, we still get
| "promotion-driven-development" because, sadly, most
| managers are not software engineers and can be successfully
| bullshitted. And it's not even always malicious: you take
| someone who is a great people person but a mediocre
| engineer (huh, that correlates a lot), and they really
| think that this pile of steaming shit idea is a good idea,
| and their genuine enthusiasm convinces management who also
| don't know better. This mediocre engineer really wants to
| do good but has no idea _why_ their idea is fucking herpes
| - because they are not _competent_ enough to understand
| why. Dunning Kruger time. And of course the competent
| engineers are all Autistic and come across as rude,
| disparaging assholes in neurotypical management 's eyes,
| and everyone gets herpes except mediocre engineer who gets
| promoted (possibly into management).
|
| In that situation you have to have "... a handful of good
| engineers and going totally rogue, we outperformed the
| entire department pretty effortlessly."
|
| All of my major career jumps have involved "going rogue",
| and having the outcome being recognized. And one time I was
| basically fired for it, and the work buried by a
| psychopath, and the remaining team did it anyway and
| delivered the solution, averting a product-ending scaling
| cliff. YMMV.
|
| But not all companies are like that, and in some places you
| can point out to management that the idea is bad, and
| management agrees. In my current job we had someone pitch a
| fabulous promotion opportunity that was not merely a total
| waste of time but also fundamentally missed (didn't even
| attempt to identify) the root cause of the problem it was
| trying to solve, and fortunately management agreed when we
| pointed it out. Yay!
|
| So I don't think it's as hopeless as you make out, at least
| not everywhere. I'm having fun right now, and getting paid
| enough (it's never enough).
| gedy wrote:
| > People who do not accept this will be perpetually unhappy
| and bitter about how others obtain undeserved success
|
| This is a good comment, however a giant issue is most
| companies will not say this out loud, and in fact say the
| opposite.
|
| So while it's fine if you're savvy enough to see through
| this, it's not really fair to smirk at people who don't and
| who honestly take what leaders say at face value.
|
| It's like a weird western version of Japanese office
| culture where you need to know the wordless rules, but it's
| not wordless and you need to ignore the BS said while
| feigning agreement.
| ludicity wrote:
| Author here! I wrote on something very similar earlier a
| few weeks ago, down to a reference to Japanese office
| culture (though not the exact same part of it).
|
| https://news.ycombinator.com/item?id=37806870
| f1shy wrote:
| >> work for one of the magically sane company where all the
| incentives are correct and everyone is doing the quote-
| unquote "right" thing at all times.
|
| Between the "perfect" company, and the shitshows described
| here, there is a lot A LOT of shades...
|
| I used to work in a company which was far from perfect, but
| it was great to work there. Until the CEO changed...
|
| I just search for a company where thing are just noy
| insane. I've been there, couple of times, and my current
| position is kind of ok...
| mprovost wrote:
| I've seen this over and over with cloud projects. People
| absolutely get promoted/raises based on optimising/cutting
| cloud budget. So there's no incentive in doing things
| efficiently from the start. You're much better off to build a
| vastly (but maybe not obviously) inefficient first version
| and then go back later and show how you "saved" the company
| tons of money by turning off a bunch of unused services etc.
| It's harder to hide racks of powered off servers which is why
| this is so prevalent in the cloud.
| ludicity wrote:
| Fascinating! I'm young enough to have almost entirely
| worked on cloud services, but that certainly explains the
| prevalence of this kind of behavior.
| davidgerard wrote:
| * Save 50p: cost-cutting genius
|
| * Spend PS5,000: prudent investment in the future
| jmetzmeier wrote:
| For many businesses the faster time to market is worth the
| additional starting costs. Doing it quickly and optimizing
| cost later has been an explicit desire for a lot of
| organizations I have been a part of.
| over_bridge wrote:
| Some of the best ladder climbers I've seen are amazing at
| pitching projects. They get the business case done, fully
| funded, themselves promoted to run it - and then immediately
| leverage the new title into a job somewhere else.
|
| I've been on the receiving end of one of those projects and
| once you dive in without the person who pitched it, you see
| how hollow and nonsensical it really is. But by then its too
| late. The consultants who wrote it are all paid and the
| executives who attached their names to it as sponsors keep it
| limping along for years, draining everyone's will to live.
|
| The other guy meanwhile is doing the same thing again at a
| bigger company having a great time.
| npsomaratna wrote:
| Almost sounds like a piece from the BOFH
| neilv wrote:
| > _The entire thing is stitched together by spreadsheets that are
| parsed by Python, dropped into S3, parsed by Lambdas into more
| S3, the S3 files are picked up by MongoDB, then MongoDB records
| are passed by another Lambda into S3, the S3 files are pulled
| into Snowflake via Snowpipe, the new Snowflake data is pivoted by
| a Javascript stored procedure into a relational format... and
| that 's how you edit someone's database access. That whole
| process is to upload like a 2KB CSV to a database that has
| people's database roles in it._
|
| Sometimes it's hard to distinguish resume-driven development from
| iterative-StackOverflow-driven development.
| syntaxing wrote:
| I'm genuinely curious what a unit test for something would look
| like.
| yonixw wrote:
| // TODO
| jacquesm wrote:
| Checksum on the resulting csv with a parallel implementation
| of the whole pipeline ;)
| syntaxing wrote:
| I don't work with large databases so pardon my ignorance.
| Is there typically a "unit test" bucket you run it on or do
| you just put in test entries on a production bucket?
| hightrix wrote:
| We tend to only test what is being tested. So, _most_ DB
| calls are mocked in our unit tests. For stored procs or
| other tests that need to be run on a DB, we use a test DB
| that is setup to mirror production.
|
| I'd bet there are a 100 different answers to your
| question though. This is the way we handle it.
| jacquesm wrote:
| Normally you'd fire up a separate environment, mock the
| process and see if it produces the expected results. By
| the time you put 'test entries in a production bucket'
| there are so many lines crossed that it likely won't end
| well even if the tests do pass.
| tommek4077 wrote:
| That is probably quite straight forward and of course they
| have 100% coverage.
| FartyMcFarter wrote:
| That would be an integration test, not a unit test.
| wredue wrote:
| We usually use very complicated UIPath flows to "test" these
| things.
|
| If that doesn't exist yet (it usually doesn't), we test
| manually, but only core workflows.
| topaz0 wrote:
| Of course the answers on stack overflow are partly a result of
| resume-driven answerers.
| baz00 wrote:
| Everything I look at these days looks like this. And most of
| the time it doesn't even solve the initial problem statement
| but everyone is too naive to even realise that.
|
| The worst thing I've seen is a stack that parses out a file and
| loads it into a DB. So someone sends us a file via an expensive
| SFTP+S3 thing in AWS. That is then picked up by some scheduled
| task using a proprietary in house scheduler process running
| inside kubernetes. This proceeds to download the file to the
| local pod. Then it makes tens of thousands of API calls to
| match up data which cranks the CPU up on a huge database
| server. This breaks all the other jobs running. Then it writes
| another file out to S3, consuming 17GB of RAM in the process.
| Another process picks that up and then batches it and inserts
| it into the DB with no transactional stuff around it.
|
| The original process this replaced was a copy into a temporary
| table and then a bit of transaction-wrapped SQL that took about
| 20 seconds to import + run. They improved that to 7 hours and
| reduced the success rate from 100% to about 80%
| foobiekr wrote:
| I know of an engineer who built a work queue by having a
| chain from an app to kafka to a processor to Kafka to a
| database writer.
|
| Literally instead of a table.
|
| This stuff is everywhere. Microservices made it worse and
| half legitimized it.
| JohnMakin wrote:
| This must be common or we worked at the same company, seen
| this exact pattern.
|
| It went like:
|
| App -> DynamoDB -> Kafka Connect Sink Process -> RDS ->
| Kafka
|
| The reason for all the middle processes were because teams
| couldn't agree how to structure their data and the first
| app would dump literal nonsense sometimes so the Kafka
| connect process's job was to clean it and dump any of the
| nonsense they pumped into it. Pretty sure there was a
| gnarly log aggregation layer in the middle somewhere too
| IIRC.
| Groxx wrote:
| I've seen _so_ many people spend weeks if not months
| "working" to avoid doing a trivial database migration.
| Database fear is overwhelmingly powerful in a lot of people
| it seems.
| znpy wrote:
| > Database fear is overwhelmingly powerful in a lot of
| people it seems.
|
| Database are still fairly poorly documented when it comes
| to administrative work.
|
| There is an incredible amount of tutorial, books and
| courses on how to write sql queries and stuff... But
| there is almost zero content on how to properly
| administer a database.
|
| I mean, from novice admin to DBA-level capabilities.
|
| I said all this before and i'm ready to write this again:
| i think there's a good market space for dba-style
| courses.
| KMag wrote:
| But, going the other way, I worked for over a decade on
| Goldman Sach's SecDB system. It's a quirky steampunk
| alternative future that branched from our light cone around
| 1995. There's a globally distributed eventually consistent
| NoSQL database tightly integrated with a data-flow
| gradually-typed scripting language (and a very 1990s feel
| 16 color IDE). I'm sure in the late 1990s/early 2000s
| (before globally distributed NoSQL was popular and before
| gradual/dynamic typing had a resurgence) it was more like
| discovered alien technology than steampunk alternative
| future. (Also, with source code being executed from
| globally distributed immutable database snapshots,
| deployment is much nicer than anything else I've used to
| date. After release testing, set a database key to point to
| the latest snapshot, and you're deployed.)
|
| There's a service that watches the transaction log of your
| regional replica so that you can make long-poll HTTP
| requests that return when any change matching your filter
| is committed. (Edit: usually the HTTP result handler is
| used to invalidate specific memoized results in the data
| flow graph, letting lazy re-evaluation re-fetch the
| database records as needed.)
|
| It makes a lot of sense for a financial risk system, where
| you end up calculating millions of slight variations on a
| scenario. The data flow model with aggressive memoization
| makes this sort of thing much cheaper.
|
| However, I saw plenty of systems written where you'd
| attempt to write your request to the next key matching some
| regex (and retry with the next key if it already existed),
| where your request would contain some parameters and the
| database key and/or filesystem path where results should be
| written.
|
| Under-experience with databases easily results in rewriting
| a database using message queue/bus. Under-experience with
| message queues/busses easily results in rewriting a message
| queue/bus using a database.
| ljm wrote:
| I think I've seen enough complexity created by engineering
| teams given total autonomy, with hands-off leadership, that
| I'd prefer a much more constrained approach. There should
| still be autonomy, of course, but proposals for new tech,
| languages and paradigms should only be considered with due
| diligence.
|
| The most unpleasant codebases I've dealt with are ones that
| have suffered from a lack of strong leadership, and they
| are almost uniquely microservice setups that pull in
| everything but the kitchen sink, usually because it's just
| trendy to use it. Monoliths can get pretty damn ugly too
| but at least it's contained in one single codebase.
| specialist wrote:
| Repeating myself...
|
| Just two examples from my prior gig (fashion e-commerce).
|
| #1 Our hottest dataset (db of current products) stored in
| DynamoDB. Core dependency for all our code. Easily fits in <
| 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed
| to persuade the team to start transition from DynamoDB to
| Redis.
|
| #2 Tiny (vs micro) service that munged some URLs. Blocker for
| an important campaign. Prior team of 4 churned for a year,
| was no closer to delivery. Spring, ORMs, CI/CD pipelines, the
| works. I spent a week unraveling the requirements (repeated
| facepalm). A second week banging out a trivial nodejs thing.
| (My team preferred nodejs, which was their prerogative.)
| Really trivial. I felt _so bad_ for the biz dev people who 'd
| been dying to get this functionality for so long.
| dmd wrote:
| I am currently working with a US government system for
| downloading public scientific data. You select some data you
| want to download and add it to a shopping cart. Check out,
| and select 'create database'. This generates your own copy of
| an Oracle database, with your own credentials and hostname
| and db name. Connect to that and construct a query against a
| table that has some metadata about studies you're interested
| in. Using the identifiers from that table, join with a LIKE
| against another table for s3:// URLs. (There are no primary
| keys and the other table's column is not exactly the same;
| you need to use a LIKE. This is all documented.) Those s3
| URLs point to a CSV which contains another identifier which
| you use to download manifests which contains links to a web
| page created on-the-fly which contains to the s3 files to
| download. By the time you've done all this, your access has
| likely expired and you must start over from scratch.
| alliao wrote:
| voted for the Kafkaesque of 2023
| dmd wrote:
| (I assume this is all a reflection of
| https://en.wikipedia.org/wiki/Conway%27s_law )
| blantonl wrote:
| I'm going to take a deep breath, and move on to the other
| stories further down.
|
| So far I'm feeling pretty good about what I've developed
| over the years.
| oooyay wrote:
| You can actually make a very comfortable career of Senior and
| Staff by learning to identify this kind of work/system and
| proposing ways to simplify it. These kinds of systems, as the
| author pointed out, are incredibly expensive and inefficient,
| but look readable on an architecture diagram.
| lainga wrote:
| > You can actually make a very comfortable career of Senior
| and Staff by learning to identify this kind of work/system
| and proposing ways to simplify it.
|
| Where?
| tomrod wrote:
| IT departments, typically, though occasionally there are
| whole companies that work in "technology" where this type
| of work can be found.
|
| I said the above as a jest, but seriously, simplification
| of complex stacks has been a good consulting gig.
| foobiekr wrote:
| Most large companies. There is a stark difference between
| the distinguished engineers and the tier below them in
| terms of asking people to stop doing things badly.
| oooyay wrote:
| I've typically worked in SRE and platform engineering work
| and that's where I've gotten exposed to these kinds of Rube
| Goldberg machines. Make a short list of them when you find
| them and then use them as a hit list during "cost cutting".
| Most people don't want to touch these systems because they
| look big and expansive and generally "work". They're just
| very poorly optimized.
|
| Dare I say, any time I see a function as a service my brain
| immediately drifts to inspecting the cost implications of
| said process.
| mring33621 wrote:
| As opposed to all those people that make similarly
| comfortable careers in middle and upper mgmt by identifying
| simple systems and complicating them beyond recognition?
| joshuahutt wrote:
| It's a beautiful symbiosis.
| solarkraft wrote:
| These roles aren't opposed at all, they greatly benefit
| each other :-)
|
| Bringing what used to be the privilege of upper management
| (wasting massive amounts of resources while getting paid
| handsomely) down to software developers.
|
| It's that trickle-down effect people talked about, right?
| oooyay wrote:
| Hah, yes. I will say that while I understand the general
| disdain here, as I grew more senior in my career I realized
| the world takes all types. There are "doers" who will rush
| to an end goal that's highly prioritized and then there's
| "optimizers" who come fix that mess up into a durable,
| cost-effective system. Some people are gifted enough in
| knowledge and have the right business priority to do both
| at the same time, but usually they're required at different
| times.
|
| Anecdotally, optimization tasks ( _in this brain_ ) are
| multitudes easier than _innovation_ tasks. I spend a lot of
| time thinking about how to do things differently whereas
| optimization utilizes many lessons I 've learned over and
| over again with well-trodden patterns. That's to say, I'm
| grateful for the doers :)
| blitzar wrote:
| > learning to identify this kind of work/system and proposing
| ways to simplify it
|
| "I dont think you are fitting in here at MegaCorp"
| avgDev wrote:
| "You are being negative, the system is great, we don't like
| that kind of attitude here".
| nine_zeros wrote:
| > "You are being negative, the system is great, we don't
| like that kind of attitude here".
|
| I want to see more lines of code, not less.
| ljm wrote:
| "We need this to scale to hundreds of millions of users
| across many regions"
|
| "But we have no users at all right now"
|
| "But we might have hundreds of millions of users in future"
| ludicity wrote:
| I am the author and this has caused me psychic damage.
| mateo411 wrote:
| You should take a vacation.
| baz00 wrote:
| Tried it. Nope. You can get people to acknowledge it but
| because it's not a fun project or doesn't involve an upsell
| you can bill the clients for, it'll go in a product backlog
| for a decade or two.
|
| I don't care any more. I'm just there to tell people what's
| shit and then laugh when it explodes in their face.
| pastage wrote:
| That is what skunkworks are for. You just deliver on time
| and you are fine.
| baz00 wrote:
| What is this deliver thing? I haven't done anything
| productive for years.
| claytonjy wrote:
| The easy part is choosing a better end-state; anyone can
| do that, and for any of these Rube Goldberg machines at a
| large-ish company, several people likely have.
|
| What makes someone a staff+ is finding a path to
| iteratively evolving towards that end-state without
| breaking anything along the way and while having progress
| to show off at each step.
| foobiekr wrote:
| I see stuff like this every day. It is a natural consequence of
| people who only "develop" by gluing things together. God help
| them if they'd actually have to write some core function
| themselves.
| pphysch wrote:
| The worst kind of "DevOps engineer" that doesn't really
| understand development, operations, or engineering.
|
| But hey, they can run some docker and git commands and piddle
| around an AWS GUI, which means they are highly technical.
| candiodari wrote:
| You don't give enough credit to organization chart and
| project driven engineering.
|
| When developing anything:
|
| 1) you don't get to touch anyone else's code. And another
| department's code? Something another manager's team manages
| ... that amounts to treason. Never for any reason. MAYBE if
| they've totally abandoned it and you absolutely need it
| (but only during unpaid overtime)
|
| 2) you don't get to spend ANY time on anything outside of
| the current project or JIRA ticket. Any time at all. So
| really, NOT optimizing anything is faster and cheaper. Just
| look at all the spreadsheets made!
| wredue wrote:
| >piddle around in an AWS GUI
|
| I've had enough calls with the "Senior/Technical Lead Azure
| Cloud Engineers" telling them exactly what they need to do
| that me and them really really don't get along.
|
| I don't do any of that shit and even I can muddy my way
| through it, but these people cannot. The real kicker of it
| is how much these people make.
|
| And you know how there are those people who, every time you
| need to work with them, they answer a teams call and then
| "need to get to my computer, give me 5" and their status is
| perpetually set to away? I don't want to RTO at all, but
| dealing with this team almost makes me think I'm wrong
| about that.
| vsareto wrote:
| >It is a natural consequence of people who only "develop" by
| gluing things together. God help them if they'd actually have
| to write some core function themselves.
|
| That's on the industry for not training and gating well. It
| would be nice to have glue/plumber positions so expectations
| are not out of line too.
| latexr wrote:
| That reads like the KRAZAM Microservices sketch.
|
| https://www.youtube.com/watch?v=y8OnoxKotPQ
| neilv wrote:
| I've used that video to explain to business people. It's
| watchable, and communicates important ideas of what a
| poopshow this can easily become, without having to talk about
| real partners/teams close to home as problems.
| Lacerda69 wrote:
| KRAZAM is a prophet and must be protected at all cost
| ludicity wrote:
| At my workplace (the one in the post), whenever one of the
| good engineers asks about how something works and it's one of
| these spaghetti-balls, we chorus "It's the design of our
| backend, okay?"
| gruturo wrote:
| :O
|
| And I thought my unholy xmllint -xpath (bad stuff, lots of
| slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/
| -e /ohyeahthistoo/somethingelse/ | grep something | while read
| AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e
| "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort
|
| was as bad as things get. I need to get my horror code game up.
| I mean, not only is the code awful, its very purpose is
| horrifying (XML to CSV with some transformations, bit of math,
| all without being able to use any external sources due to
| security, only what's in a baseline RHEL7 (soon 8, yay!) ).
|
| I promise I'll rewrite it in python at some point.
| tshaddox wrote:
| I think it's "iterative-StackOverflow-driven development" most
| of the time, and that actually _causes_ the increased
| popularity of those resume keywords.
| kmfrk wrote:
| ... And a partridge in a pear tree.
| feoren wrote:
| This sounds like absolute hell. This is everything I hate about
| modern software development.
| crooked-v wrote:
| The more experience I have, the more I start to think the
| Omnigres people are right about "just put literally everything
| into Postgres".
| toasted-subs wrote:
| You have no idea how unbelievably annoying it is to work in a
| company that doesn't a well defined architecture. Every "buzz
| word" service should be easily justified.
|
| This is why I hate recruiters, I can't even tell you how many
| times I've had a recruiter call me saying they are looking for
| service XYZ. The same concept rephrased in my resume. I have to
| rewrite my resume just to satisfy these people? No thanks.
| neycoda wrote:
| So, what tech service can I add to that bloated pipeline as a
| middle-man to get a fraction of a penny per transaction?
| Vicinity9635 wrote:
| I feel like this might be case of data engineers.
|
| They're not usually software engineers. They're _tool users_
| not _tool makers_.
|
| So they'll cobble things together to accomplish the task, using
| only available tools and never anything custom that would do it
| task much more cleanly, because they understand data, not
| software. They're not computer scientists or programmers,
| they're just users. And we all know what that means.
| entropicdrifter wrote:
| Agreed. I've been "the backend engineer who works with the
| data engineers" for several years now and I've seen their
| general trend of re-inventing the wheel the hard way a number
| of times.
|
| I've spent the majority of my career building better tools
| for data-related tasks, then winning over my users by showing
| off performance and productivity gains.
| scruple wrote:
| I stepped into a Data Engineering Lead role in 2019. Stepped
| out of it in 2021. My team was the first in the org to really
| approach data engineering and we were all software engineers.
| I'm told that the systems we built have largely been replaced
| by Rube Goldberg machines pieced together by the folks who
| came after us.
|
| Those replacement systems aren't even working, they're
| failing to deliver on the same simple data pipelines that we
| had working by the start of 2020. They're cobbled together
| using a million little AWS pieces and Docker and k8s... I'm
| glad that I left that role when I did, we were being pushed
| by a new-hire with a fancy Data Engineering VP title to do
| all sorts of asinine things. I went and looked just now and I
| see that he's Senior VP at a different company, he started
| there this summer. Onward and upward!
| Agentlien wrote:
| I feel like I've never seen anything even reminiscent of this
| bad in the twelve years I've worked as a software engineer. I
| really want to believe this pipeline as described is satire.
| Yet, somehow, it does not quite seem that way. This scares me.
| But also somehow explains why some companies contain so
| incredibly much more engineering staff than I can possibly
| explain looking at their output.
| BonoboIO wrote:
| Resume Driven development - RDR
|
| Never heard that before, but that's so on point.
| trealira wrote:
| > I'm not sure what the original estimate was, but I think it was
| intended to cost something like 200K for a year of operations,
| but we were now close to a million dollars.
|
| > ...
|
| > I return to work the following Monday. I suspected that this
| would save a bunch of money, and guess what, our projected bill
| dropped from a million to half a million dollars, and everyone is
| losing their fucking minds.
|
| Wow, so they're still over budget by 300k dollars. This is a
| funny story, but the company sounds incompetent.
| NoMoreNicksLeft wrote:
| > This is a funny story, but the company sounds incompetent.
|
| I've seen "typical" spelled this way several times on Hacker
| News. Is it a British thing or something like that?
| plagiarist wrote:
| The author seems to agree with you, they are still wildly
| incompetent with their response to this.
| digging wrote:
| This is such a good article. My skin was crawling.
|
| What really scared me is that I couldn't identify any of the
| issues raised in my own organization, even though we often run
| into similar, smaller-magnitude problems caused by a blindness to
| obvious mistakes. It makes me fear I, too, am blind to massive
| bleeding wounds. Here's hoping they actually don't exist.
| tehlike wrote:
| The whole post is gold.
|
| - managers asked how it was possible we saved that much without
| help from them
|
| - asked to prepare slides
|
| - asked many times on how it happened
|
| - had to roll it out slowly to make it look like they did it over
| time incrementally vs one small toggle
|
| - asked for a raise due to impact and did not happen.
|
| Sir, for your sake, apply to a FAANG or something, you'll be at
| least taken care of better.
|
| Also please implement Twitter card metadata in your blog so it
| looks better on twitter :)
| pavlov wrote:
| _> "please implement Twitter card metadata in your blog so it
| looks better on twitter"_
|
| Didn't Xitter recently remove the display of all external site
| metadata except the image?
| tehlike wrote:
| Yeah people use image interestingly these days though
|
| See, for example:
| https://twitter.com/simonw/status/1717768637799706922
| herpdyderp wrote:
| Maybe my friends are all just bad devs but I don't hear of
| people getting treated well at FAANG anymore. (Though I haven't
| heard anything about Netlifx recently.)
| tehlike wrote:
| Maybe they are busy working - after while layoffs everyone is
| working a bit more to say the truth
| y-c-o-m-b wrote:
| I'm at a FAANG, still getting treated like shit. The
| problems outlined in the article are 1000x worse in a FAANG
| and you'll have the same 99% of the people that have drank
| the proverbial kool-aid telling you everything is great and
| it's supposed to function like this. At least you get paid
| more for it though.
| tehlike wrote:
| True, and not true. It's nuanced. It depends a lot on the
| org.
|
| I have moved from admob (google ads) to robotics to
| facebook, and worked with different orgs, and have seen
| differences.
| Tao3300 wrote:
| I think we just keep the N in there so it's less awkward to
| say.
| delecti wrote:
| Facebook is now Meta, Google is now Alphabet, Netflix is
| now just another independent movie/TV studio. FAANG has
| become a term entirely independent of what it originally
| stood for (also "MAAA" doesn't have the same ring to it).
| Groxx wrote:
| I think it works. Just think of it as a family with a
| troublemaker kid.
|
| MAAA! THEY'RE DOIN STUPID STUFF AGAIN AND THEY WON'T LET
| ME USE MY COMPUTER!
| hnuser123456 wrote:
| MAAAM or MAMAA
| petters wrote:
| You don't hear "Alphabet" a lot. MAGMA works
| tubthumper8 wrote:
| With the rebranding of Facebook, we could go for MANGA, but
| dropping the "N" would still have an interesting result.
| jihadjihad wrote:
| It would be quite the GAAF
| CrimsonChapulin wrote:
| Im surprised no one asked to automate an email to management of
| the status for the rollout of changes.
|
| "We'll need the percentage complete as well and a summary of
| the savings expected. It needs to be sent out every Friday at
| noon."
| nfRfqX5n wrote:
| At least at fang you can write a mostly plaintext document
| instead of PowerPoint. Downside is you would probably have to
| do it before making the changes, have it reviewed by the entire
| team and get "alignment"
| tehlike wrote:
| Yes and no, changes based on the org
|
| But you would get handsomely paid.
| munk-a wrote:
| Skip FAANG and go straight to a company that'll value you!
| Justsignedup wrote:
| Lol, if I wasn't given a raise, I'd agree to everything, and at
| the start of my presentation say
|
| "hi everyone, I wanted to walk you how we got to a half-a-
| million dollar savings, basically I spent a day looking at how
| terrible the original infrastructure was deployed and removed a
| code test feature that was causing the problems. This was just
| complete oversight from every aspect of the development,
| management, testing, and everything. Overall this code is as
| bad as it can possibly get, and we just launched it. And
| basically I was told not to say any of this because it makes
| everyone look bad, so I was to roll this out gradually to make
| it seem like managers were doing some sort of work."
|
| Then drop the mic and walk off stage.
|
| Honestly the amount of give-a-fucks I would have lost would
| have been a lot. And this is coming from someone who's done
| this for almost 2 decades and cares about his job because bills
| to pay, kids to feed.
| whartung wrote:
| Back in the day I worked at a company that if you came up
| with some long term cost saving measure, they gave you a
| bonus of 10% net savings for the first year.
| dugmartin wrote:
| A co-worker in the early 90s (he was a tech writer) told me
| of a cost savings device he invented in the 80s at Texas
| Instruments to fix a process where occasionally a mirror on
| a very expensive piece of military camera gear got
| scratched (I think it was during field disassembly). It was
| basically some forceps with more metal welded on to make
| them longer and that allowed you access via a different
| route than where the mirror was installed. TI gave cost
| savings awards as a percentage of money saved and he did
| very, very well with that little invention.
| tehlike wrote:
| That's a nice incentive.
| soitgoes511 wrote:
| They sure don't do that anymore... At least not at any of
| the Dallas factories. Even patents reap little money. You
| practically sell your soul when you join the company and
| all your ideas are theirs.
| dugmartin wrote:
| This was in Dallas, I believe. But this was in the 80s
| before the Peace Dividend when there was a lot of defense
| money sloshing around. I met him in Chicago in the 90s
| and he, and a lot of TI folks, had left Texas as the
| defense related work dried up. The team I was on in
| Chicago was working on the flight data recorder for the
| F-22 which still had funding.
| soitgoes511 wrote:
| Oh... I believe it.. Times have changed though. They cut
| back on travel, corporate credit cards, etc. We would
| have no less than 2 cost marathons per year of all day
| meetings.
| Justsignedup wrote:
| I would say blanketed bonuses are a bad idea, or I'd make
| some partnership deals with other engineers. However, this
| was a pretty clear cut "give this man a medal" situation,
| in a clearly toxic company. This is why my reaction is what
| it is.
|
| There was an old thedailywtf post about how a company
| thought they'd incentivize fidning and fixing bugs.
| Suddenly every engineer had a QA buddy, and they'd make
| like 50 spelling errors, which QA will find, and
| enigneering will quickly resolve. They took down the bounty
| within a week.
| duskwuff wrote:
| > There was an old thedailywtf post about how a company
| thought they'd incentivize fidning and fixing bugs.
|
| That might have been this Dilbert comic:
|
| https://i.stack.imgur.com/bQOvF.png
| qohen wrote:
| There's a name for this anti-pattern, the Cobra
| Effect[0]:
|
| _The term cobra effect was coined by economist Horst
| Siebert based on an anecdotal occurrence in India during
| British rule. The British government, concerned about the
| number of venomous cobras in Delhi, offered a bounty for
| every dead cobra. Initially, this was a successful
| strategy; large numbers of snakes were killed for the
| reward. Eventually, however, enterprising people began to
| breed cobras for the income. When the government became
| aware of this, the reward program was scrapped. When
| cobra breeders set their now-worthless snakes free, the
| wild cobra population further increased._
|
| (There may be some question as to whether these events
| actually occurred or not, but there are similar examples
| of documented pest-control campaigns (and others) on the
| Wikipedia page[0] where similar things happened).
|
| [0] https://en.wikipedia.org/wiki/Perverse_incentive#The_
| origina...
| jimmySixDOF wrote:
| I have seen higher splits than that. You have to remember
| this is bottom line cost output saving so the 500K if you
| look at it on the sales side is like completing a +/- $2M
| deal in terms of net margin added to the business.
| sshine wrote:
| I did save one company enough money to finance my salary
| for the year I worked there, just by removing dead servers
| provisioned but no longer used. Seems like my predecessors
| were blind to both cost and the past.
|
| I felt bad for leaving so soon, but good for not having
| cost them a dime.
| BonoboIO wrote:
| Can work great, but watch the incentives. Don't
| overoptimize a business process and later fix it with a 10%
| bonus.
| quest88 wrote:
| While this feels good to imagine, the social fallout would be
| disastrous.
| yjftsjthsd-h wrote:
| It really depends on your priorities and operating
| environment, but there are many people who would be leaving
| anyways if they save the company that much money and were
| given exactly zero reward, and at that point plenty of
| people would be happy to burn bridges on the way out. Like
| I said, it really depends on the environment and your
| priories.
| tehlike wrote:
| Tech is a small circle. Unless it's something obviously
| wrong, you may want to be at least milder :)
| quest88 wrote:
| The thing about burning bridges is you don't know when
| you'll need them.
|
| Let's say 5 or 10 years later you're applying to a job
| where one of these upper-level people now work. How do
| you want them to remember you? The know-it-all who wasn't
| a team player and kind of an asshole? Or the engineer who
| gets things done and has demonstrably shown to land
| impact and value, an engineer the exec would consider
| lucky to have?
|
| Some of you will say you wouldn't want to work for one of
| these executives again. But people change, incentives
| change, the environment changes. Have you ever made a
| technical decision you later regretted?
|
| And maybe you don't work for them. Maybe you're applying
| to a different company where someone knows these previous
| upper-level management folks and they ask about you. How
| do you want that recommendation to come across? "That
| engineer was an asshole.", "That engineer was amazing, I
| wish we could have kept them. We made a big mistake by
| not trying to keep them.".
| mst wrote:
| You are -mostly- right but there is also the factor where
| a specific sort of negative reaction can actually
| function as a recommendation later.
|
| I've definitely missed out on work sometimes due to
| having a reputation for being about as subtle as a brick
| to the face with no lemon, but I've also -got- certain
| pieces of work as a direct result of being criticised and
| somebody who heard the criticism thinking "if he annoyed
| that person that way, he's probably serious about doing
| the right thing."
|
| I would, however, suggest that probably I would've done
| better overall if I'd toned it down a bit.
| mcv wrote:
| Would it? I mean, the people you're throwing under the bus
| would hate you, but the top of the company should love
| because a) you saved them a ton of money, and b) you
| identified a pile of incompetence in their company.
|
| And who's going to fire you for this?
|
| The people telling you to roll this out slowly are doing so
| mostly to protect themselves from having their incompetence
| exposed and to appear useful. Protecting them will help
| them steal your credit and will get them promoted.
| quest88 wrote:
| It would, I replied to someone else below to explain why.
| But in general, it's best for your future career to be
| remembered well.
| digging wrote:
| That sounds like a very nice and theatrical outcome, but
| in reality nobody wants to believe they've been horribly
| wrong about their understanding of things, which means
| the higher-ups are going to be talking directly to the
| managers to get their side and figure out how they can
| show _you 've_ misrepresented the situation. They're not
| going to fire managers _en masse_ because the new guy
| said "Everyone here is dumb and cowardly," even with
| evidence.
| bcherry wrote:
| this assumes that the "top of the company" is a shining
| tower of competence, blissfully unaware of how
| incompetent the people below them are.
|
| more likely, the top of the company is just as
| incompetent but has lucked / gamed their way into
| promotions anyways.
|
| So this engineer throwing his managers under the bus may
| be good for the company in theory, but to the senior
| management this is a red flag. they don't want engineers
| who do this - they want engineers who give credit to
| their managers. if they promoted this engineer and fired
| everyone else, he'd come for them next.
| munificent wrote:
| This is a deep misunderstanding of human psychology.
|
| _> the top of the company should love because a) you
| saved them a ton of money_
|
| The top of the company is probably already rich. Being
| richer is great, but one of the few things rich people
| generally won't burn to run the money-making engine
| hotter is _their own sense of prestige and entitlement_.
|
| _> b) you identified a pile of incompetence in their
| company._
|
| Yes, and you told that fact to people who were already
| responsible for identifying that, which means you just
| told them that _they_ are incompetent too. And that
| transitively works its way all the way up the org chart.
|
| You would make the whole chain of command lose face and
| do so _in front of the rest of the chain of command._ It
| would be career suicide.
|
| Every manager would rather silently waste money than be
| made to look like a fool. Because the money comes out of
| the _business 's_ bank account, not theirs, but looking
| stupid affects their _personal_ reputation.
| d3w4s9 wrote:
| I'd consider that if I already have a job offer from another
| company starting next week. Otherwise it is self destruction.
| choilive wrote:
| "Got management material written all over him"
| grecy wrote:
| I worked for a large telco that operates very, very similarly
| to the company in the article.
|
| About once a year or so one of the stand-out engineers that
| had the weight of the world on their shoulders would get
| burnt out and frustrated enough to do exactly what you
| suggest above.
|
| Literally everyone would just look around awkwardly, leave
| the meeting and never talk about it again. All of middle
| management already know all of this, the only way they keep
| their jobs is by never talking about it, and just ignoring
| anyone that does. The VPs and President only know what those
| below them feed them.
| mparnisari wrote:
| > Sir, for your sake, apply to a FAANG or something
|
| Noooooooooo
| datadrivenangel wrote:
| Why are corporations so allergic to competence?
| yetihehe wrote:
| Competent people in corporation could show how uncompetent the
| rest of corporation is, so they are found and eliminated before
| they can do too much damage to management.
| SillyUsername wrote:
| That last line isn't a joke.
|
| I've been on the receiving end of middle management because
| I've been able to fix things in the business that have always
| been broken but never got fixed until I worked on them.
|
| Management will claim it's their work, will give you just lip
| service, will not use your name higher up the hierarchy and
| will actively down play their own mistakes whilst blaming the
| rest of the department or developers (e.g. like building a
| project with wrong requirements they actually provided, then
| missing the deadlines because stakeholders demand changes).
|
| Where that has happened I quit and then they just go back to
| the status quo.
| ikekkdcjkfke wrote:
| Hominids in suits
| prmoustache wrote:
| They are not eliminated, they are assimilated.
|
| A company is like a chain, as strong as its weakest link.
| There is not much gain to have some links much stronger than
| the others.
|
| The famous "I've learnt my lesson" which means you really
| reached the same low level of incompetence as everyone but
| secretly thinking you are more competent than all the others.
|
| The funny thing is that among all those incompetent peoples /
| idiots mentionned there are probably smart people who just
| learnt their lesson years/months and adjusted to the weakest
| links in the chain.
| yetihehe wrote:
| They are "dealt with". One way or another, competent people
| are stopped from being a problem for management, either by
| being eliminated (fired), or tempered down (not
| assimilated, more like "converted") so they are no longer
| competent, but that's a small difference for me.
| osmsucks wrote:
| This is called the Tall Poppy Syndrome.
| rekoil wrote:
| Possibly hot take; everyone wants to be a manager because
| managers make more, but you can't have a bunch of managers
| unless you hire a bunch of people, and hiring a bunch of people
| that are all competent is hard.
| r00fus wrote:
| Is it possible that the corporate veil is essentially a cloak
| of invisibility for scam artists?
| barryrandall wrote:
| I think it's just the result of assembling a large number of
| humans. Beyond some size, relationships get replaced with
| internal politics and aggregated individual shortcomings become
| organizational pathology.
| NovemberWhiskey wrote:
| Corporations are not allergic to competence; they're allergic
| to bomb-throwing. Except in times of tremendous strife or
| revolution, incremental change is the best that can reasonably
| achieved in large organizations. The definition of competence
| in a large organization includes understanding the concept of
| Chesterton's Fence, and knowing what the Overton window is.
| layer8 wrote:
| Why do seemingly competent people like TFA's author continue to
| work at such a broken company, is what I wonder more.
| ludicity wrote:
| I'm not sure what TFA means (the first article), I appreciate
| being called competent, though the truth is that I'm just
| less blundering than some other people.
|
| I'm here because there's some path-dependence in careers. I
| started at a mediocre company due to not having a permanent
| work visa, and have been clawing my way up. I should write
| something else on this, but I've also realized clawing my way
| up was unnecessary - it turns out that while I've had some
| skill atrophy from working at these places, good engineers
| recognize someone that isn't going to cause a spreadsheet
| dumpster fire, so I should have just jumped to one of the top
| rungs on the ladder years ago.
| runamuck wrote:
| Thank you for writing a hilarious and painfully true slice of
| office life.
| world2vec wrote:
| Like staring at a mirror, this post made me cringe with how
| relatable it was. Guess I have a whole blog to read now.
| ChrisMarshallNY wrote:
| Too bad Scott Adams self-immolated.
|
| This is Dilbertland, at its finest.
| dudeinjapan wrote:
| My goal is to make my company the opposite of this one.
| Groxx wrote:
| > _I am asked to write some PowerPoints, which include phrases
| like "a careful statistical analysis of user usage patterns
| indicated an opportunity to more effectively allocate resources",
| implying that nothing was wrong, we just needed to collect more
| data before deciding not to let the expensive machines idle all
| day._
|
| Yep.
|
| If you don't make them realize it was a Hard Problem(tm) that was
| only solved by their smart hiring, funding, and task-deciding,
| you might shatter their whole world view.
| cainxinth wrote:
| TLDR: They changed the idle time settings for running queries in
| their Snowflake database. Originally, they were set to idle for
| 10 minutes after every query, but most queries only took about 2
| seconds to run.
| 0cf8612b2e1e wrote:
| I know nothing of Snowflake pricing, but how big is their
| infrastructure that leaving it on for an extended amount of
| time could see that much savings?
|
| To put it another way, to rack up $500k spend for 30 days of
| constant provisioned time:
|
| $500k/(30*24) = $694/hour
|
| Presumably, with the 10 minute blocks, there was idle time
| where the spend was zero, so the instantaneous amount would be
| higher.
| datadrivenangel wrote:
| $58 per hour over a year on average. Sounds like a big big
| warehouse. For scalability.
| ludicity wrote:
| This is approximately right. They also almost have one
| warehouse per two data engineers. I've been told there's a
| reason and usage analysis has been done by actually smart
| people, but it just sounds incredibly suspicious to me.
| SillyUsername wrote:
| I did something similar.
|
| Success and super efficiency was rewarded with additional work,
| including hinting I should help other departments (apparently a
| joke).
|
| I received no extra remuneration despite asking, yet my company
| continues to hire new staff weekly.
|
| I've learnt my lesson, just like this author.
| mcv wrote:
| You didn't accidentally save half a million, you deliberately and
| intentionally saved them half a million, but now you regret it.
| That's not the same thing.
|
| Large organisations are so woefully inefficient that I'm
| surprised they're able to compete at all, but they have a ton of
| money and economy of scale and all that, and along the way
| there's more than enough money to waste millions on stupid
| nonsense and inefficiency and nobody really cares.
| photonbucket wrote:
| In my mind, there is where inflation really comes from. For
| every wage paid that wasn't useful, the resulting product
| becomes more expensive / less profitable. If we were able to
| optimize out all the waste we'd probably have deflation while
| computers and business processes keep getting more efficient
| digging wrote:
| I mean waste is probably _a_ source of inflation but lots of
| it just comes from unrestrained profit-seeking behavior. If a
| company believes it can raise its prices without impacting
| sales, it will do that. That 's why so many massive
| corporations posted record-breaking profits during the recent
| bouts of inflation.
| anoxor wrote:
| Government shutdowns that made it more difficult or
| impossible to shop outside of big stores or online would be
| as big or bigger issue here.
| digging wrote:
| It wouldn't explain simultaneous mass layoffs.
| ElectricalUnion wrote:
| Except that generalized deflation is one of the largest and
| widespread economic disasters that can be brought about; you
| are disincentivizing immediate consumption to on the basis of
| future larger consumption, you're disincentivizing investing
| in the economy instead of just holding capital - both things
| that will make a economy stop working in the long term.
|
| It's the reason why most central banks really want some
| minimal level of inflation going on at all times.
| Vicinity9635 wrote:
| ...no.
|
| Inflation is actually very simple: More money is added to the
| system than value produced. That is, money is printed faster
| than value is created through labor. Stop and think for a
| second. How do we have trillions of USD? Banks _create_
| money. And they 're doing it faster than ever before.
|
| Now you can absolutely have price gouging _at the same time_
| , but the two are independent of each other, even though
| combined the affects are worse for the price gouged.
| Vinnl wrote:
| It would be helpful if you'd point out what term _is_
| appropriate. Given the reference to "my country", odds are
| that they're not a native speaker. (I can't tell from the
| English though, but I'm not a native speaker myself.)
| throwaway290 wrote:
| Another way to look at it, you intentionally saved the company
| half a million and you got paid what you think was the fair pay
| for it. It seems like the author might think it's too much
| money but the irony is that someone is clearly undercharging
| (helping skew salaries in the entire industry lower, of course)
| while execs get bonuses. There's nothing to regret if you can
| make ends meet, great job.
| akira2501 wrote:
| > I'm surprised they're able to compete at all
|
| They typically don't. When the government doesn't enforce it's
| own rules, these companies just buy up competition. They
| strangle the market and they get more inefficient all at the
| same time.
| pgraf wrote:
| In my opinion this red tape is why startups are more successful
| than large organisations... At least in the beginning
| CobaltFire wrote:
| I feel this entire post.
|
| My career record (US Navy) for cost savings was something over
| $50MM. Every time I did something I had to do PowerPoints,
| present to Flag Officers, etc. Hell, once I almost got hugely
| punished because I didn't let my boss take the credit (he had no
| desire to because he had zero idea what it was I even did).
|
| Note that some of that was as a Lean Six-Sigma Black Belt doing
| Enterprise projects (I hate every single bit of terminology in
| that entire godforsaken sentence), where it was literally my JOB
| to save the DOD as much money as possible with the least
| disruption possible. Those were the absolute worst years of my
| career. That period of my career was my reward for just going
| rogue and fixing things that saved millions.
|
| I'll echo the last part of that post: Beware of doing good things
| at work; the reward is rarely compensation and is usually more
| work for the same pay.
| tehlike wrote:
| Sometimes I think of joining government org (3 letter agency)
| to find areas to improve $ efficiency on, or make decent direct
| contribution to US as a thank you for accepting me as an
| immigrant.
|
| I'm an engineer in a FAANG, who worked directly in money flow,
| and have had experience in diverse areas where it could come
| handy.
|
| Then I start thinking finding the right person to work with &
| right area to start at is probably 95% of the job, then give
| up.
| wslack wrote:
| As someone who does this sort of work (but posting
| personally, not officially), you're 100% right. There are
| groups of already-networked folks you can join like USDS or
| 18F that are already connected to the problems. The post
| reminds me of some of the stuff you might see in gov, though
| I've never seen someone save money and be punished like this.
| bigsassy wrote:
| You'd want to start with the United States Digital Service
| (USDS). They're basically a consultancy inside the executive
| branch to build tech solutions for the various 3 letter
| agencies. You basically do a "tour of duty" for a year or so
| (however long you want), tackling one of these solutions. And
| they go out of their way to hire folks like you, from FAANG
| and other tech companies, so you work with good people.
|
| https://www.usds.gov/
| BonoboIO wrote:
| Is the salary competitive?
| busterarm wrote:
| > Six-Sigma Black Belt
|
| I'm so triggered by this phrase. The worst (to work for and
| performing) company I was at hired SSBBs like crazy and none of
| them really did anything or made a damn bit of difference, but
| if you took the courses you were fast-tracked for success as
| far as promotions went.
| CobaltFire wrote:
| I was not at all popular among my "peers" in that world. I
| had too much common sense, and didn't make changes just to
| make changes. I was extremely effective, but since I didn't
| play by the rules I was ostracized internally.
|
| That said, it did give me the opportunity to get the hell out
| of that community and back into IT/CyberSec.
|
| In all honesty, there are absolutely good concepts and tools
| in that world. The issue is that the people who gravitate to
| that world have no ability to provide any value themselves so
| don't actually understand how to USE those tools properly.
|
| Everything is a nail when all you have is an ENTIRE TOOLBOX.
| syntaxing wrote:
| It's interesting because I found Mech design engineers with
| green belts to usually be really sharp. I only met like one
| black belt design engineer and he pivoted careers a decade
| ago. But most black belts I met in QA were managers. Most of
| the technicians that did the work weren't certified but lived
| and breathed the products for decades.
| busterarm wrote:
| In that industry it makes sense. We were a services
| platform -- basically a massive, distributed call center.
| CobaltFire wrote:
| My experience was that the tools are absolutely useful.
| Green Belts are those who learned the tools, but stayed at
| their day job. Black Belts are when you pivot into doing it
| full time.
|
| Much like people who underperform in technical fields often
| pivot to Management, you see the same in the pivot to
| Process Improvement. That creates the issue of the lowest
| performers at the actual job being the ones most inclined
| to pivot into that position.
|
| There are always good ones (I'm biased, but think I was).
| The issue is that they are a relative rarity due to those
| (and I'm sure other) factors.
| syntaxing wrote:
| The pivot part rings so true, especially "Engineering
| Services", aka process and infrastructure control. They
| often lead the engineering change committee too. I
| shudder just recalling those meetings.
| unicornmama wrote:
| > I'll echo the last part of that post: Beware of doing good
| things at work; the reward is rarely compensation and is
| usually more work for the same pay.
|
| Let's rephrase this in a less nihilistic way: Understand your
| organization's values and culture. And beware of doing good (or
| bad) things in a way that goes against your organization's
| values and culture.
| CobaltFire wrote:
| Speaking as someone who spent over 20 years in the DOD:
|
| There isn't a way to make things better in the DOD that
| doesn't go against the culture. Period. My literal JOB was to
| make things better and it was the worst time of my career.
| That was with Flag Officer backing and independent authority.
| I got a chest full of shiny candy and a pathological distaste
| for it all after a few years of that.
| esafak wrote:
| Bigger organizations tend to be more pathological; they
| attract and shelter sociopaths. So try to find a smaller
| one that is compatible with your values.
| CobaltFire wrote:
| Instead of finding a new org I retired and am doing my
| own thing.
|
| Figuring out what said own thing is is proving a bigger
| challenge than anticipated, but I'm enjoying it.
| koliber wrote:
| How did you go about figuring it out? I'm somewhere on
| this difficult path and would love to hear a good story.
| Contact details are in my profile if you'd be willing to
| share.
| CobaltFire wrote:
| I'm about six months into figuring it out and only now
| getting a feel for who I am without all the pressures of
| making a living in my chosen career.
|
| I'll drop you a line via your profile contact info and
| maybe we can have a chat!
| PH95VuimJjqBqy wrote:
| That's not the takeaway at all.
|
| The takeaway is don't be good at things you don't want to be
| asked to do.
| rvbissell wrote:
| > Hell, once I almost got hugely punished because I didn't let
| my boss take the credit
|
| ... I'm mentally tripping over that part. Is that normally
| expected? ... By recursion, it seems that POTUS should get all
| the kudos, all the time.
| CobaltFire wrote:
| In the military it's extremely common for someone above you
| in the Chain of Command to get the credit; it may be your
| boss, or several levels up.
|
| Some of that credit may or may not spill over to you, but the
| general thought process is that they are responsible for what
| happens in the command so they get the credit. Good ones
| avoid that and push the credit to where it belongs, but then
| they don't promote. This leads to an incentive to take credit
| from those under you in order to get promoted.
| rvbissell wrote:
| Right, but someone /other than your boss/ was mad that you
| didn't arrange for your boss to receive the credit. That's
| the WTF detail.
| CobaltFire wrote:
| Correct; the fact that my name was on the presentation
| that went before Congress, vice my CO's, indicated to
| them that I went AROUND my CO.
|
| In reality, this is certainly possible. It just would
| have been a career killer. My CO was the one who sent
| that presentation up with my name front and center. He
| had to defuse the situation.
|
| Also, and this is absolutely relevant, I was not a
| Commissioned Officer. I retired as a Chief Petty Officer,
| though I had a degree (Nuclear Engineering) and multiple
| "graduate level" certifications (PMP, LSSBB, CISSP, etc.,
| etc.). There is only so far competence and capability can
| take you without rank in the US military. I chose to exit
| rather than move over and promote, which was the right
| decision for me.
|
| I apologize for lacking the context in my previous
| comments; I often forget people don't understand the
| intricacies of the arcane ways of the military unless
| they subjected themselves to it.
| quacked wrote:
| It's like a dark-mirror version of a really good principle
| in leadership philosophy, which was outlined in _Starship
| Troopers_ by Robert Heinlein. "Everything that happens
| under your command is your responsibility." If a man fails
| to do his job in the correct way, it's because you failed
| to make a system that would train him properly, and
| possibly because you allowed the wrong man to be hired.
| Technically, every success is also your responsibility, but
| a smart leader spreads blame for success and privatizes
| blame for failure. In reality many leaders pretty much do
| the opposite.
| CobaltFire wrote:
| Starship Troopers is on many recommended military reading
| lists for career professionals for very good reasons.
| DavidPeiffer wrote:
| I had an industrial engineering professor who did a full career
| in the military, focused on energy efficiently.
|
| Paraphrasing, "The great thing about the US Government is it's
| so big, if the optimal solution would save 30%, but you find a
| suboptimal solution that only saves 29%, nobody will notice
| because you're still saving tens or hundreds of millions of
| dollars of savings."
|
| One of his big projects was on energy efficiently in remote
| military camps. The cost of fuel per gallon was around $100
| delivered to parts of Afghanistan. They had tons of portable
| generators running at ~20-40% of capacity. If memory serves,
| ~70% is most efficient. By some combination of building a small
| or connecting a few tents to a single generator, they were able
| to very significantly reduce the fuel required and improve the
| quality of service.
| count wrote:
| >Note that some of that was as a Lean Six-Sigma Black Belt
| doing Enterprise projects
|
| >(I hate every single bit of terminology in that entire
| godforsaken sentence.
|
| Checks out. :).
|
| A super convenient way to do cool stuff like that in the DoD is
| to do it for the most senior Flag you can get interested in it
| directly. Projects at SECDEF or SECNAV offices
| work...differently than outside, as there isn't really an 'up'
| left for most of the folks involved (in my experience - most
| are extremely focused on getting the job done and/or the geo-
| strategic problems).
| CobaltFire wrote:
| During the time I worked in that position I was one office
| removed from COMFRC, which was the Flag I generally reported
| to via my CO (who was the one who put me in that position and
| was highly supportive, and who ended up as COMFRC in later
| years).
|
| Anything that was directly interfacing with that office was
| great; as soon as I was detailed to do something downline
| from that office it got painful despite reporting to that
| Flag.
|
| I wasn't in a position to go higher without spending more
| time in the service, and that was a non-starter for me.
| Vicinity9635 wrote:
| > _Note that some of that was as a Lean Six-Sigma Black Belt
| doing Enterprise projects (I hate every single bit of
| terminology in that entire godforsaken sentence),_
|
| _brother!_
|
| > _Beware of doing good things at work; the reward is rarely
| compensation and is usually more work for the same pay._
|
| I got lucky and learned this in college working for the
| helpdesk. Privately I know the best value I really bring is
| experience in avoiding clusterfucks before they even happen,
| and I'll be johnny-on-the-spot during production things (during
| the workday, I'm not ops), and then slack off as much as I
| possibly can since the reward for high performance isn't more
| pay, it's just more work.
| justin_oaks wrote:
| And this is the difference between competence and incompetence in
| tech. We can argue about whether 10x developers exist, but one
| thing I hope we can agree on is this: Certain problems can't be
| solved no matter how many incompetent people we throw at the
| problem.
| digging wrote:
| This is really what people should be looking for in a hire. If
| this person joins my team, can I reasonably expect them to
| bring some sense into our code/infra/schedule? Nonsense creep
| is constant in a big enough organization, and on-ramping is
| slow, but asking "what does this even do?" is invaluable (wait,
| no, it can be quantified!)
| plagiarist wrote:
| We should be realign around "0.1x developer" terminology. Like
| assume many developers are in the 1x or 2x range, but then
| there are plenty of 0.1x developers writing scripts that start
| with 500 lines of comment.
|
| Then job postings would have to be, "we're looking for
| competence and adequacy. We want to pay you a normal amount and
| get our money's worth."
| orliesaurus wrote:
| i loved the style of writing! what do you call it? Realism?
| Sarcasm? Idk
|
| It had me stuck reading until the very end - usually that never
| happens!!
| pedro_hab wrote:
| I had a friend in a similar situation, he explained the issue and
| how he would fix it.
|
| I was baffled and urged him to be careful and assume he was wrong
| that it had to be wrong.
|
| Anyways, he is still fixing it, getting some people to validate
| it.
|
| I would think he will save at least hundreds of thousands a year.
|
| But seriously, why a company that spends millions of dollars in
| an area would not hire an expert to try to save some money in it
| is beyond me.
| marcos100 wrote:
| Because "we don't have the budget".
| esafak wrote:
| That "we" may be too low on the org chart, and the qualified
| manager may not be aware of the problem.
| freeopinion wrote:
| Why would a manager of a multi-million dollar department reduce
| the department to a single-million dollar department? That
| would reduce the prestige of their own job. I've seen managers
| who were overstaffed by double work hard to double their staff
| again. And the only staff that actually accomplished anything
| were punished and culled.
|
| It doesn't make sense until you realize that they operate in an
| environment where there is no competition that can disrupt
| their business model. If their costs go up, their rates go up.
| There is never an incentive for costs to go down.
| ludicity wrote:
| Because it's monopoly money or something man, I don't know.
| knodi wrote:
| you mean someone costed your company half a million dollars for x
| amount of time.
| dpifke wrote:
| The author's gallery of poignant HN commentary on their writing
| is _chef 's kiss_: https://ludic.mataroa.blog/compliments/
| jjkaczor wrote:
| Huh, I wonder which of the "big-4" advised them on their
| practices?
|
| Am willing to bet it was 'KTMJ'... (Batman reference, but...
| strangely similar to one of the "big-4")
| ludicity wrote:
| You actually got it. If it isn't a bother, I'd actually like to
| know _how_ you guessed this, as I thought all the Big 4 were
| roughly the same.
| asow92 wrote:
| > I saved my company half a million dollars in about five
| minutes. This is more money than I've made for my employers over
| the course of my entire career because this industry is a sham. I
| clicked about five buttons.
|
| I'm sorry, but this needs a privilege/gratitude check. You are
| guaranteed your salary, and you're welcome to take on the same
| level of risk your company is by starting your own. If you think
| it's so easy go ahead.
| skeaker wrote:
| I don't understand your point. According to the article it was
| in fact that easy. The issue was due to hangups in bureaucracy.
| j45 wrote:
| Great post on why LLMs will have Management Consultants looking
| for their future. Average skill is highly profitable. One skilled
| senior and 20 juniors at an affordable rate is always most
| profitable.
|
| Developers - Always do this as bonus paid by contingency of saved
| money and a signed off scope if it works out, especially if
| you're d looking into it on your own time.
|
| Specifically if you like you can get signatures from everyone on
| the hierarchy on how much money or time this will save, cost or
| make them.
|
| Do it enough times and the right kind of CEp/President will tell
| you to stop bringing the business case to them for approval and
| just do them if they make sense and you have your backup.
|
| You will enter a side door, bypassing most politics and c-levels
| (and maybe triggering some new), reserved for people who say let
| me see what's possible instead of coming back with reasons why it
| can't be done.
|
| From there, paint a picture of what if you built a team of only
| doers, minus talkers across the organization.
|
| :)
|
| Tech should never report into Finance, the group that can't even
| tame spreadsheets.
|
| Yes, business owners pay.
| isoprophlex wrote:
| > I ask management for a 30K raise after saving 500K and my
| message is still unread. I suspect I will eventually receive
| either nothing or 5K.
|
| Only one thing left to do. Leave 'sleep(600)' calls all over the
| place and resign.
| znagengast wrote:
| I have to play devils advocate here because for every one of
| these cases, there is probably a dozen of similar stories where
| the ambitious new guy actually did nuke the system with a risky
| friday release and then logged off for the weekend xD
| ludicity wrote:
| Let me hit you with the extra spicy take - since the department
| doesn't really produce business value anyway, taking the system
| offline for a week would have dropped the bill to 0 and been
| the greatest possible cost saving.
|
| Yes, management would have killed me, but I would have been
| absolved at the Pearly Gates.
| cameron_b wrote:
| I love this for purely self-appeasing reasons.
| this machine kills imposter syndrome
|
| or at least it helps. Having a background in solid CS theory from
| High School, and having a degree in Art, I find it very hard to
| apply for engineering roles, and my mixed bag of experience often
| lands me in Support Engineer / application admin / integration
| roles, fuming tremendously when the people with SwEng / Developer
| titles fumble on with implementing some JavaScript change for a
| feature I need in Service Now for my application's customers.
|
| It is incredibly reassuring that it is not simply my organization
| that is hamstrung by the pretense of complexity, when really
| someone just made it complicated to make it seem important.
| 23B1 wrote:
| I accidentally cost my company half a million dollars, but it's
| not really my fault HR misplaced a comma.
| thefourthchime wrote:
| I saved my company $1 million a year a couple of months ago by
| noticing that there was an S3 bucket that kept growing and
| costing 80k a month.
|
| I poked around and realized that there was a system that we
| weren't using anymore that was copying files to the bucket I
| reached out to the stakeholders and they turned it off and we
| deleted the files.
|
| The higher-ups didn't seem to really care, My boss's boss told me
| to reach out to another team that should've caught this and that
| was about it.
| Foobar8568 wrote:
| This is what is going on to a Saas provider we are using. I
| told them about but they shrug it off..VC money I guess :D
| nickdothutton wrote:
| The unsaid part is that this kind of situation is everywhere. It
| only really hits home when (IMO) when you've had a couple of
| decades in the industry. We have totally squandered the abundance
| of resources in many sites/situations/use-cases. Whether it be
| IOPS or threads or memory or network bandwidth and latency. Yes
| premature optimisation is the root of all evil, but the other
| side of that coin is pretty ugly looking too. Not even because of
| monetary waste in many cases, but because of unnecessary
| complexity and fragility. For what? A slightly higher level of
| abstraction? A bit more interoperability? A marginal gain in some
| other metric?
| bitwize wrote:
| Great! That's worth (let's see... carry the three...) a 1% raise!
| yudhiyudhi wrote:
| I know of an intern (with a master mentor) who did a one char
| change and saved our company many millions - the change was to
| update the threshold of data size of when to not compress small
| messages - this threshold was updated after many many years - old
| threshold was based on old compute and network costs.
| throwaway-jim wrote:
| is this larping?
| joshuahutt wrote:
| Now extrapolate this out and realize that 25-50% of our global
| tech economy is basically waste.
| grvdrm wrote:
| This post hit right something I dissected in my last role.
|
| In short, I was revamping/enhancing a property insurance policy
| pricing system as the lead from the business side (i.e. portfolio
| manager/actuary).
|
| However, much of the development team was truly incapable of
| writing software in any useful way, so I inevitably dipped into
| C# and SQL to help diagnose issues as we (as a team) worked on
| the enhancements.
|
| My business coworker and I found something that remains funny to
| this day: code littered with references or attempts to use the
| .NET TPL library to make the platform do more things "in
| parallel," but without actual knowledge of using such a library
| in practice.
|
| Nothing worked in parallel! There were blockers all over the
| place! As someone else said, TPL might well have been a //TPL TO
| DO.
|
| We suspected that TPL was there as one of those "look what I can
| do" intrusions that received praise from more senior devs/mgmt.
| even if it never actually did what it was supposed to do.
| siliconc0w wrote:
| At $large-scale company this happens pretty regularly if you just
| look for expensive things and ask:
|
| * "Can I just remove or slowly deprecate this?" I.e this job
| generates data no one uses, or alerts people just bin-bucket or
| has been already replaced by faster/ better. * "Can I cache
| this?" * "Can I run this at a lower-priority/off-peak/less
| available?" * "Can I reduce the frequency or move processing to
| deltas?"
|
| This is just the easy stuff you can usually do in a few lines of
| change without even getting into basic optimization rewrites.
| Most internal things at $large-co are built because someone
| thought they might be useful, they might get promoted and using
| that hypothesis and moved on, but few things are actually
| continuously validated as still generating value > their cost.
| piperswe wrote:
| Repost of https://news.ycombinator.com/item?id=38064086, should
| probably be merged
| andrewstuart wrote:
| It's always the cloud costing ridiculous amounts of money.
| pavlov wrote:
| It costs a million dollars to run a service that essentially
| "uploads a 2KB CSV to a database" on Snowflake?
|
| And you can cut this in half by changing some defaults in
| instance lifetime?
|
| I'm starting to understand why Snowflake's market cap is
| something like $50B. This sounds like a nice money-printing
| business if you can convince enterprises to use it.
| ludicity wrote:
| I should clarify, the cost was coming from computers that
| engineers used to write queries against our database - the 2KB
| CSV was just how we managed their permissions, and the cost of
| that was negligible.
|
| The immense cost was coming from someone writing a query that
| translates to "I need one row of data" and then we get billed
| like $10-20 in idling compute. With multiple computers and
| several full-time SQL modellers, it adds up very, very quickly.
|
| As the guy that just has to keep Snowflake running smoothly and
| _isn 't_ paying for it, it's a really nice product. I would
| still prefer something else on principle because it isn't open
| source, but eh, I guess it reduces my stress at work.
| ynx wrote:
| > I ask management for a 30K raise after saving 500K and my
| message is still unread. I suspect I will eventually receive
| either nothing or 5K.
|
| Pain I feel very acutely. As an employee, I saved a FAANG company
| billions of dollars in potential GDPR fines after they carelessly
| declared 'mission accomplished', weeks before the deadline, and
| never even got a 'thank you', much less a raise.
| jb3689 wrote:
| Whenever this happens, I really have to wonder about all of the
| people I call "good" on my team. Like surely someone gave a shit
| enough to know this is how it works, right? ...right?
| ilc wrote:
| Reason number 543895 why I like small company life:
|
| When you find shit like this, people will go "my bad" at the
| worst, and you all are happy the company has more money. Small
| teams and small firms have their downside, but honesty and
| transparency, especially when it comes to cost savings... tend
| not to be one of them.
| RadixDLT wrote:
| if you want to learn more about consulting firms watch John
| Oliver https://www.theguardian.com/tv-and-radio/2023/oct/23/john-
| ol...
| greedo wrote:
| I'm on the tail end of my career (hopefully), so I can share some
| unsolicited advice for younger workers.
|
| 1. Remember who you work for. Yourself. Not your boss, not your
| grandpa who always wanted you to become a writer. You only have
| so many breaths before you get your ticket punched, so make them
| count.
|
| 2. Don't seek work as something you should love. Those should be
| your loved ones, and hobbies etc that stimulate you. Sure there's
| a small minority who get to do what they love, but over time love
| can turn to disgust.
|
| 3. When starting a job, figure out the organization's incentives.
| Some orgs want change, some will fight any change. Figure out
| what the org wants and your life will be much easier. My job is
| like the Maytag repairman, kind of waiting for stuff to break. I
| refer to it as being a digital janitor.
|
| 4. Nothing you build digitally will last. Hell, between linkrot,
| bitrot, and the heat death of the universe, nothing lasts. So
| don't expect what you've built to last, or to hold value. It'll
| all be re-factored/re-engineered/re-architected and become
| obsolete.
|
| 5. Large organizations are toxic and often sociopathic. So are
| small orgs. If you can find a way to start your own business, try
| to avoid becoming that way. Good luck, most businesses either
| fail or become that way. But at least you have more choice in the
| matter.
|
| 6. Find a mentor (sometimes referred to as a rabbi) who can
| provide guidance to you when times get tough. Don't choose a
| rabbi from your reporting chain. You will find yourself adopting
| a worldview that might not be aligned to your own interests (in
| other words, the bastards will manipulate you). This rabbi can be
| a friend, or just someone you vibe with. Be careful in choosing
| your rabbi, and take care of this relationship.
|
| 7. Illegitmi Non Carborundum
| ludicity wrote:
| Great advice, especially on finding a mentor. I hadn't thought
| about the importance of finding someone outside your reporting
| chain.
|
| Illegitmi Non Carborundum.
| lsh123 wrote:
| Long time ago in a galaxy far far away, I replaced the need for N
| * (Oracle license + Sun server) with a simple perl script (< 100
| lines) and one Sun server for a total cost saving in $300M+ /
| year. While doing it, I also invented map-reduce (it was before
| Google time).
|
| The problem was to calculate bunch of stats from web servers logs
| (e.g. 10 most popular pages). The original solution was loading
| it all into Oracle database running on multiple servers since
| logs were huge. And then running bunch of SQL queries. Rinse and
| repeat daily.
| arein2 wrote:
| good management would fire half the org
| Ikatza wrote:
| > They hired some incredibly talented people to make this happen,
| and then like five times as many idiots.
|
| I can relate to this.
| arein2 wrote:
| In my company we worked with a platform launched on local
| machines that had an admin console where you could execute java
| code. Pair that with almost everyone not turning on the firewall
| and all engineers connected to the same wifi network, anyone
| could do whatever they wanted.
|
| I showed a demo how easy it is to read private ssh keys to the
| head of infrastructure, and after some months people could
| connect to network only using custom credentials (ldap) which was
| good, but also asked us to install "spyware" that among other
| things checked the firewall. I never installed the "spyware" but
| nobody pushed me. I didn't think I somehow prevented a disaster
| or did some heroic deed because everyone in the company was
| professional and nobody would exploit this. But of course I
| didn't tell about this to anyone except the infra because such
| information should not be disclosed until is fixed. And once is
| fixed why disclose it?
|
| I really miss the Mac checkbox to enable the firewall. On linux I
| use nftables which is really powerful, but with so many
| possibilities it is easy to miss something during configuration.
|
| I observed a lot of senior engineers don't have sufficient
| network knowledge. A lot of people on linux don't use the
| firewall which is really bad if you work on shared wifi.
|
| Also when running docker images, if you map a port when using
| docker run (ex. docker run -p 80:80), docker will automatically
| add firewall rules and bypass the enabled firewall, exposing that
| port publicly.
| afro88 wrote:
| Sounds like something out of a Gene Kim book.
|
| Definitely a pattern through the places I've worked. The ones
| that are smart, have good intentions and go a little bit rogue
| tend to make the biggest impact.
|
| Executives are too concerned with risk, and honestly, it playing
| out this way suits them better: if someone goes rogue and it
| works, they can claim the win, if it doesn't, they can blame and
| reprimand.
| geephroh wrote:
| "My team has spun this as a huge cost saving, when really we just
| applied a fire extinguisher to the pile of money that we had set
| alight."
|
| I love this so much
| mateo411 wrote:
| I think the author should take a vacation.
| mikewarot wrote:
| This kind of thing has been going on forever. Once upon a time, I
| walked into a room full of VT100 terminals, and bored CS students
| waiting for their compiles to finish. (It was a weekend near the
| end of term) I took a look at the system, and realized they had
| pushed all the compiles into a batch queue, but that queue
| defaulted to BELOW interactive priority, so any keystroke
| anywhere had higher priority... so all the people checking their
| position of the compile job in the queue, slowed it down even
| more.
|
| Over the next 15 minutes, I kept bumping the priority of the top
| job in the queue up, an hour later everyone had their work done,
| and went home. I had the room to myself. Rogue sysadmin for the
| win. ;-)
___________________________________________________________________
(page generated 2023-10-30 23:00 UTC)