[HN Gopher] Accidentally Saving the Day
___________________________________________________________________
Accidentally Saving the Day
Author : ahiknsr
Score : 76 points
Date : 2022-05-24 16:11 UTC (6 hours ago)
(HTM) web link (essays.jwatzman.org)
(TXT) w3m dump (essays.jwatzman.org)
| uvnq wrote:
| dang wrote:
| " _Please don 't pick the most provocative thing in an article
| or post to complain about in the thread. Find something
| interesting to respond to instead._"
|
| https://news.ycombinator.com/newsguidelines.html
| high_byte wrote:
| To clarify the ambiguity - he is white.
| https://twitter.com/jvwatzman
|
| Make what you want of it...
| [deleted]
| bartread wrote:
| Many years ago I worked at a company where I just didn't feel
| particularly useful or valued. I was working on a peripheral
| client-side Java app that was basically just used as a visual
| configuration for the meat of the system that was all on the
| back-end.
|
| There were a bunch of more senior developers who I hugely
| respected and learned a lot from, including this guy who I'll
| refer to as $SMART_DUDE who joined at about the same time as me.
| He knew his stuff inside out when it came to C++ and I remember
| being absolutely fascinated by a talk he gave on memory
| management and different types of pointers one lunchtime. It's no
| exaggeration to say that I literally still refer to things I
| learned in that talk today, almost exactly twenty years later.
|
| There were some pretty big personalities in that company and I
| was seriously intimidated by my line manager, who always seemed
| like he was pissed off with me. I spoke to him when he
| interviewed me, and then barely ever until my performance review
| came round nearly a year after I'd joined the company. I remember
| speaking up in a meeting once, him snapping at me, and I just
| shut the fuck up.
|
| He surprised me in the performance review. We had 5 levels of
| bonus. I thought I was decidedly mediocre in his eyes but, out of
| nowhere, he gave me not the top level, but one down, which was
| only slightly less than the top level (we had a reasonable degree
| of transparency about percentages involved).
|
| Still, I didn't think much of it and had started looking around
| for other roles. One of the motivators is that one or two people
| I really respected had or were leaving, including $SMART_DUDE,
| who'd handed in his notice. There was him and another guy, who
| I'll call, $SMART_DUDE2, who I thought - and still maintain -
| were just the real deal. I thought they were better than the
| people they worked for, _way_ _waaaaaaaay_ better than me, and
| better than everybody else in the company, although, to their
| credit, I think the people they worked for also though the same.
|
| The other motivator was that, although I was earning slightly
| more, and on top of this the company had covered my rent for the
| first 12 months, I was still financially worse off than I had
| been where I'd lived before on the lower salary (higher cost of
| living in the new area).
|
| Anyway, in the meantime I started hooking up our client with the
| backend using XML rather than CORBA. Not even using SOAP. I just
| realised there was an endpoint that we could communicate with
| firing XML documents back and forth according to a protocol that
| we agreed in a late afternoon/evening design session that would
| allow us to load and update whole trees of objects, rather than
| just individual properties (surprise, this was mind-blowingly
| slow), without having to make any major code changes on the back
| end. Really not that big a deal.
|
| Whilst this was all going on I landed a contract. It paid
| something like 1.8x what I was earning after tax even with
| something like two months holiday factored in. It could have been
| more but I was so overawed by the low end of the pay band that
| when they asked me what I wanted I said PSY and they accepted
| without an argument. I was an idiot. Still, because I thought
| absolutely nobody would believe me, when I handed in my notice I
| told the company my new role would pay 1.35x my salary. I was an
| idiot for the second time. If I'd not been an idiot at all I
| could probably have got 2.2x.
|
| It was at this point my line manager - bear in mind by this point
| (14 months in) I could still count the number of times we'd
| spoken on one hand - dropped the bombshell of, "Oh no, I mean
| you're really great. I think when you leave we'll feel the loss
| more than $SMART_DUDE. Let me see what I can do on salary." WTF?
|
| I still maintain that $SMART_DUDE was a lot better and smarter
| than I was or will ever be, and I'd hire him in a flash -
| literally no hesitation other than to ask him if he might not get
| a bit bored by the work we're doing.
|
| I still don't really get it, even now, but I suppose I delivered
| pretty consistently during the time I was there, didn't make a
| fuss (anyone who knows me now would find that very hard to
| believe - there's a history here that's too long and boring for
| HN), and solved a business problem that they had quickly and
| pragmatically without worrying too much about whether it was the
| best option from an engineering standpoint.
|
| My advice: you don't need to be super-smart, but be just a little
| bit hungry, able (and stubborn enough) to deliver, and don't
| stand too much on engineering ceremonies.
| conductr wrote:
| I feel like this is a sweet spot of sorts as careers go. People
| hire you unproven and often have little or no real expectation.
| If you can deliver at all, even just by being consistent and
| not making a fuss then you will be rewarded. You've also got
| something to brag about now and it's noticed by other
| employers. Your employer knows if they don't pay you, someone
| else will. They probably also feel like you're rather underpaid
| for the work you put in recently and can justify bonuses and
| such rather easily. Furthermore, if you leave they have a big
| interruption and void to fill. They have to roll the dice on
| another unproven or pay for someone experienced. They're
| lacking leverage to keep you.
|
| In this way, it's not unusual to 2x or more your income in
| those first couple years across many different roles (even non
| technical). This is the reality of becoming what is often
| referred to as junior->senior.
| unnouinceput wrote:
| At word "CORBA" the hair on my back lifted up. I hated that one
| with every fiber of my being 20 years ago when, on a job, I had
| to deal with it. I avoid it like plague after leaving there and
| I am so glad is a forgotten technology none uses anymore (maybe
| except some legacy projects, God have mercy on poor souls that
| have to do maintenance on those projects).
| the_only_law wrote:
| Ugh I had to do some work for the COBRA subsidy during COVID
| and to give you and idea how horrid this was, the frontend
| was an Access DB sourcing data from SAP.
| ajkjk wrote:
| I did this at my first job. A director had a problem and no one
| was sure if a solution was practical. He was chatting about it in
| our team area and I said, oh, I can see a way to make that work.
| A few months later it was shipped and I basically got a promotion
| out of it.
| feoren wrote:
| > I basically got a promotion out of it
|
| What? Your contribution to your company was actually
| _recognized_? I 've never heard of that happening before. Where
| do you live? I'm guessing it's not the USA.
| xingped wrote:
| Hah, yeah, basically my thought process reading the article
| and the parent. Since when do people get
| promoted/recognized/raises for pulling a rabbit out of a
| magic hat? Certainly not anywhere I've worked and I've done
| exactly that a fair few times.
| [deleted]
| wildmanx wrote:
| I appreciate the story and the pride he must have felt. But this
| is the entirely wrong message. It perpetuates that the superstars
| are those jumping in last second in some heroic effort to save
| the day.
|
| This is not how projects should work, and we should stop
| perpetuating these heroic stories. The _actual_ superstars are
| all those people doing the less-than-glamorous grunt work.
| Planning things _on time_. Getting the infra ready, stable, well-
| tested, documented. Making schedules that have margins. The
| people carefully designing things so they are robust and don 't
| break left and right. Thinking through their systematic testing.
| In short, preparing everything so that heroic acts are _not_
| necessary.
|
| Imagine the same story but somebody two months earlier had voiced
| a concern that Jabberwocky and ChaChing wouldn't play well
| together. Pushing for the APIs to be harmonized so that they
| _could_ play together and integrate. Pushing all this _against_
| pressure that they are just delaying everything and should not
| worry so much and be more positive and agile. Imagine they had
| succeeded and everything had gone smoothly. This would have been
| _much_ better for the company (since the existential risk wouldn
| 't have existed in the first place). But that somebody wouldn't
| have ended being the superstar with the promotion and the bonus.
| It would likely have stuck to have been the guy just delaying
| things with their worries. _That 's_ what needs to change in sw
| eng culture.
| necovek wrote:
| And not to be discounted, all those prideful references to long
| days and late nights: another symptom of bad planning and
| management.
|
| It also puts a bunch of people who have actual life commitments
| (like kids, health issues, family issues or whatever else) at a
| disadvantage simply because somebody can afford to spend the
| extra time on something. And if those are the people who
| properly plan and execute, they get a double whammy.
|
| The moral of the story should not be that when there is an
| opportunity to make a significant contribution by exchanging
| your life for your work you do that, but that you should be
| able to excel at work and be recognized for it without
| exchanging your life for it.
|
| Sure, the author of the post does not have anything to do with
| most of the management failures, except the failure to not work
| extra hours for a very artificial deadline (one does not need a
| hero to save face if you've messed up yourself -- exposing the
| failure is usually a net win).
| aidenn0 wrote:
| Firemen will always get more praise than people who change the
| batteries in smoke detectors.
| dilyevsky wrote:
| False. People who quickly and cheaply build houses that kill
| everyone inside in case of small fires get all the praise,
| not firemen or whoever changes batteries (usually nobody)
| morelisp wrote:
| I think there's two stories intertwining here.
|
| One which you've zeroed in on is "don't require heroism". We
| have a lot of these stories already. (Unfortunately also a lot
| of exciting stories of "heroism".)
|
| The more interesting one though, is that sometimes high-impact
| work is "just there" to be done, if you're looking. IMO this is
| also really what "hero" stories are about: something goes so
| wrong that either people are finally forced to either _look_
| for once, or forced to bring in someone who will look.
|
| As a senior engineer, for the past couple years I've tried to
| pitch parts of what I see "just there" intentionally, either
| before it becomes a crisis or just as a potential win.
| Sometimes people _still_ don 't see it; maybe they're stuck in
| "one year of experience, ten times" or sometimes they're just
| built out of stop energy. But the best way I've found to detect
| promising junior engineers who will benefit from mentorship is
| to find who does bite, and more often respond "what if" instead
| of "but".
| jwatzman wrote:
| > The more interesting one though, is that sometimes high-
| impact work is "just there" to be done, if you're looking.
|
| Author here. Yes, this was exactly my point!
|
| > But the best way I've found to detect promising junior
| engineers who will benefit from mentorship is to find who
| does bite, and more often respond "what if" instead of "but".
|
| Indeed. I've mentored a bunch of otherwise-competent
| engineers trying to get to the next level, and my advice is
| similar: ask yourself "ok, now what's next". After doing the
| cool thing, there's inevitably half a dozen other things
| around it, and things might be obvious to _you_ (as the now-
| expert in the cool thing) but not others.
| necovek wrote:
| > Author here. Yes, this was exactly my point!
|
| Your original example shows you doing extra hours and long
| days to achieve this: if this was your point, the example
| is terrible.
|
| I do agree with high-impact work just being there, but
| finding the time to do it other than "on your own time" is
| where the tension is between management and engineers who
| have a different belief of what the highest priority is.
| jwatzman wrote:
| > if this was your point, the example is terrible.
|
| Yeah, I addressed this in another comment here. I
| attempted in the last section to separate the ideas of "I
| heroically dived in" and "I took an opportunity in front
| of me" -- you shouldn't go _looking_ for heroics. This
| didn 't quite get across, and I totally understand why.
| Ah well.
| jwatzman wrote:
| Author here. For what it's worth, I (mostly) think you're
| right! IMO heroics were a huge problem at that company, and are
| a huge problem at the company I work at now too. The gruntwork
| is really really important.
|
| I tried to get at this in the last section, that it's _not_
| about going out trying to be a hero, and it 's unfortunate it
| didn't get across. That was my point that you shouldn't go look
| for company priorities to fix, to be a hero. Just look around
| you, see what's going on, and do good shit. Sometimes yeah that
| does involve heroics, sometimes it's just gruntwork.
|
| I don't want to go back and edit the essay now but I think it
| could be much clearer indeed!
|
| > Imagine the same story but somebody two months earlier had
| voiced a concern that Jabberwocky and ChaChing wouldn't play
| well together. Pushing for the APIs to be harmonized so that
| they could play together and integrate. [...]
|
| Disagree here, though it's maybe not obvious from the bits of
| the story I told why. ChaChing was one of... maybe three or
| four dozen experiments like it. All of the others failed. There
| was no reason to believe ChaChing would be any different, and
| if it _did_ work, even the best-case expectations were well
| below what it ended up doing. It really was a lightning strike.
| So it would have been a huge waste of time to do all of that
| work, delaying Jabberwocky, for a bunch of things which never
| ended up shipping.
| sleepymoose wrote:
| To give a different point of view, the message I got out of
| this was exactly what you were intending. I didn't come away
| with "go find opprotunities to be a hero". I found the
| opposite in fact, that you're saying ~don't~ go looking
| specifically for those heroic opprotunities. I feel that you
| reinforced this point in your closing statements quite well:
|
| >It isn't about creating bigger and bigger opportunities for
| yourself -- it's not about selfishly inventing self-serving
| projects. Rather, it's about getting better at recognizing
| and taking advantage of bigger opportunities which are
| already there and just making things happen.
| gopalv wrote:
| > you shouldn't go look for company priorities to fix, to be
| a hero
|
| The "to be a hero" is the problem, you can always feel good
| about solving hard problems and finding your boundaries, but
| "avoiding a crisis" is a thankless job & we should thank
| those people more than the "only I can fix it" people.
|
| The core problem for me is that the constructive "avoid
| crisis" people are baked out of these sort of events, rather
| than born risk averse in the first place.
|
| > Just look around you, see what's going on, and do good
| shit.
|
| One of the biggest side-effects of WFH was that this whole
| "who's around" mechanic has faded away without a proper
| replacement.
|
| Most of my satisfying work came from random interactions like
| this (not the most impactful or the most purposeful).
|
| I was a principal engineer and most of my 10 AM to 2 PM was
| taken up by meetings which mostly involved prevention of work
| duplication or to stop someone driving down to where I knew
| there was a dead-end (like, "I see you're using alpine docker
| images, why?" or more management boundary stuff like "the
| difference between these two ideas is ~140k over a year for
| all customers - can we agree that the expensive one is better
| for customers, because they don't see a 12$ month bill and it
| costs more than 140k$ to build the cheap one in NRE?").
|
| Most of that work is tiring and always unproductive - the
| actual work is done by smart people who answer my questions
| carefully, the dumber the questions, usually the better the
| results.
|
| But sometime around 3 pm, I'd be line to get a coffee and I
| would run into a PM or someone from support or dev who would
| have a question for me which sounds nonsensical at first -
| but having an hour & half to go down that rabbit hole looking
| over someone's shoulder usually ended up with mostly "hey, I
| know what's happening, I've seen this before" or the classic
| "hmm, that's strange, how could that even happen?".
|
| Six to ten weeks later, I would be in a meeting where I would
| get some sense of "deja vu" when someone describes their
| approach and reach back to that day & that problem, to ask
| "how would you detect X" (something like "machine reboots,
| comes back with same hostname, but different IP" - would you
| restablish connections to IP or hostname, what about krb5 ...
| yada yada).
|
| That part is the real value as an engineer, but entirely
| unappreciated from the management's point of view.
|
| With WFH, I would go get a coffee, go for a run/walk and as
| good as it was for my state of well being, I would eventually
| get pulled into these crisis situations Thursday at 9 pm
| instead of Tuesday at 3.
|
| I like WFH for the work I actually have to do, but this sort
| of temporal serendipity is not easy to recreate on Zoom.
| foobarian wrote:
| So... why are Alpine docker images a dead end? :)
| gopalv wrote:
| > why are Alpine docker images a dead end?
|
| That's the most controversial thing at work too.
|
| It comes up so often because it is a measurably good idea
| when you look at container sizes or startup times (gzip
| is terrible, terrible way to package container images).
|
| An engineer tries it out, it works great and it does
| because alpine is lean, fast and generally good.
|
| I work on performance, so any decision to make things
| slower gets routed through my desk - the perf buck stops
| at my desk. Now it's my thankless job to squash out her
| initiative and as gently as possible (there are lots of
| smart people in the industry, but being kind has a better
| ROI), so that it is not a "this is policy" form letter
| appeal to authority, but walk them through the entire
| list of tickets in my notes.
|
| Because we hit a bunch of SEGVs with the JVM and the JDK
| team basically closes them as WONTFIX when reported.
|
| And this sort of decision has a shelf life, I can be
| right for 18 months and wrong the week after - so this is
| not a hard line.
|
| This specific thing is definitely not a permanent thing,
| but a temporary headache - the JDK team is working on
| Portola and that'll fix any issues they had.
|
| https://openjdk.java.net/projects/portola/
|
| If you're going to install glibc as a workaround then
| you're basically giving up the space savings anyway.
|
| We still break things in production, but
| counterintuitively I'm the happier when the problem is
| bang in the middle of code I can commit to, rather than
| deep inside musl -> jdk interactions on what happens with
| a longjmp on a SIGSEGV.
|
| Also the turnaround for the CVE reported was much faster
| across teams if everyone picked the exact same base image
| - standardizing on Redhat Universal Base worked out (&
| also AquaSec scans etc is easier if 6 different products
| have the same image down there).
|
| Amazon Corretto does have alpine images, so this is sort
| of a 2020 advice with rapidly declining value (I'm
| funemployed for 2022, so my current "weakly held"
| opinions are entirely about rust syntax).
| not2b wrote:
| While this is true, it's also true that the people who are
| doing the less-than-glamorous grunt work are often unrecognized
| by management (which arguably is a management problem).
|
| I remember being at an all-hands company meeting years ago and
| listening carefully to what the people who were given the top
| awards got them for, and about three in a row were for diving
| in, working nights and weekends to come up with emergency fixes
| to save projects or products that had gone seriously wrong, and
| these were different projects. It was so blatant then even some
| senior managers finally noticed "Hey, we have a problem".
| high_byte wrote:
| I really enjoyed this article, the style of writing, the story
| itself and the other article linked. "Be the change you want to
| see in the world."
|
| Keep it up.
| jwatzman wrote:
| Author here -- thanks, glad you enjoyed it!
| CarVac wrote:
| There's a lot of low hanging fruit out there, you just have to
| recognize it or be lucky enough to stumble facefirst into it.
|
| I recently joined a budding open-source project (PhobGCC, a
| custom gamecube controller), saw that it was using a rather
| overbuilt full kalman filter, and within 2 days of work rewrote
| it into a less generalized filter that ran 7x faster (later
| improved even further) and performed even better at its filtering
| task.
|
| It was super basic work, despite it having been almost a decade
| since I last studied control systems, but it made a really big
| difference for the project.
| grue_some wrote:
| I have had very similar work out in my favor early on. New
| engineer, some hacky idea that saves the day doing something that
| was "impossible" and got a bigger raise and promotion than I have
| gotten in any way other than saying I was leaving a job since. It
| was nice recognition. I have had some repeat of this recently,
| but not quite the same bump. It comes to be that fixing some
| urgent problem is way more $$ rewarding than running a team and
| making sure your team doesn't run into that stuff and just
| delivers on time. It is odd, but in a way it makes sense. You get
| more recognition for putting out a fire than preventing one.
| Everyone is watching while something is burning and recognition
| from someone above your direct manager is the way to get a better
| than nominal raise.
___________________________________________________________________
(page generated 2022-05-24 23:01 UTC)