[HN Gopher] Take this on-call rotation and shove it
___________________________________________________________________
Take this on-call rotation and shove it
Author : mirawelner
Score : 74 points
Date : 2025-03-27 21:09 UTC (1 hours ago)
(HTM) web link (www.scottsmitelli.com)
(TXT) w3m dump (www.scottsmitelli.com)
| dylan604 wrote:
| " insure against every possible thing that could ever go wrong,
| they would have to build a second studio on a separate part of
| the city's electric grid, with redundant copies of all the
| equipment and broadcast content, along with a full crew of
| understudies ready to take over at a moment's notice."
|
| WTH?? I guess this person has never heard of backup generators?
| Every broadcast TV station has them.
| beepboopboop wrote:
| That just covers electricity. They seem to be implying coverage
| of a multi-failure scenario.
| cfraenkel wrote:
| To begin with, airplanes do fall out of they sky, sometimes
| right on your backup generator. (speaking from military
| experience where yes, there is an entire backup studio waiting
| to take over, just in case. Or rather, the 'studio' is
| geographically dispersed, with 100% redundancy, which is
| another way of saying the same thing.)
|
| But more importantly, *this* is what you noticed from that
| article?
| dylan604 wrote:
| No, but this was the point where the "let's make something
| up" got to be too much.
|
| I'm not talking about low budget UHF channels, but TV
| stations I've been in and around all have multiple studios.
| If the switcher in Studio-A goes down, the signals can be
| routed to the control room for Studio-B. Also, Alex the know-
| it-all is such a forced thing that is just ridiculous and eye
| roll inducing. Anybody that is a jack of that many trades is
| a master of none of them. The entire forced analogy just got
| to be too much and I lost interest before a point was ever
| made.
| boznz wrote:
| What they don't tell you about working for yourself is the fact
| you can be effectively on-call 24x7 every day. I am currently
| supporting four wineries that are processing thousands of tonnes
| of receivals 24x7. It happens for two months of the year and I am
| expected to be available from 06:00 to 22:00 during that time,
| there is no phoning in sick or having a lazy day, I work alone
| and only have one reputation. I don't want to be that contractor
| forever known for destroying a clients business.
|
| You can only do this for so long though, when two or three
| problems come in simultaneously it can cause issues as you drop
| something halfway through when something more important comes in.
| I once executed an SQL update query without a where clause under
| this kind of pressure, and ended up working until the next
| morning to recover, only to start again at 6AM. I have even had
| land-line calls at 2AM to bypass my mobile restrictions. The
| rewards are great, but don't let anyone tell you it is always
| easy.
|
| My current system is 16 years old now and I know all the ins and
| outs so it has been pretty easy to keep on top of things the last
| several years, however I am glad the replacement system is nearly
| written and it will be somebody else problem in 2026.
| netruk44 wrote:
| There's a big difference, though.
|
| In your case, you're the only employee of your business. And if
| you're not there the business will literally go under. And you
| also get directly rewarded _for_ being there. I would guess
| that being 'on call' in this manner is possibly less draining
| on a person's soul (depending on how well they tolerate the
| risk of owning a business).
|
| Contrast that with being 'on call' for your megacorporation,
| who isn't giving you anything extra for your on call time
| because they 'already pay you enough'. And where the only
| negative consequence for the company if you fail to immediately
| respond within 15 minutes is that some executive in the company
| is kept waiting longer than 15 minutes, or some ads aren't
| being shown for 15 minutes.
|
| But if you aren't there, your boss is going to get a phone call
| and that's definitely not going to look good on you. And
| there's no bonus for fixing the problem, that was already your
| job in the first place. Sucks that you had to do it outside of
| scheduled hours, oh well.
|
| I'm with the author of this article. Take your on-call rotation
| and shove it (if you're a large corporation). I'm fortunate
| enough to be able to take a firm stance on this point, and do
| so happily.
| Arainach wrote:
| Working for yourself is totally different.
|
| It's like demands from tech executives for long hours: "I
| worked long hours to make myself rich; why won't all of you
| work long hours to make me richer?"
| flerchin wrote:
| Jeez I guess what we do is industry standard best practice, and
| it sucks.
| darioush wrote:
| deferring to best practice instead of best judgement is a major
| plague of the software industry these days.
|
| best practices usually come from giant companies with tens of
| thousands of engineers like google (who doesn't seem to be
| keeping up with competition btw) and amazon (which is notorious
| for burning out people).
|
| what science or evidence drives the best practices?
| yodsanklai wrote:
| Excellent article. I can relate to a lot of it. The sad part is
| that we can't even control the quality of the systems we're
| oncall for. We're pushed by management for new features, not for
| robustness of the tools. Also some systems have no clear
| ownership, so nobody has an incentive to fix them. It'll be next
| oncall's business. Oncall is really the worst part of my job. I
| can stand long hours but this is something else.
| mortar wrote:
| I feel for you, I've also suffered through this a lot over the
| years, and am finally at the stage of career and wisdom to
| start pushing back on the quality that I can't control and
| ensuring that others are equally as accountable for their mess.
|
| For one particular occasion , once we took blame out of the
| equation (at least within the engineering team) and started
| doing Post Incident Reports, the incentives finally became
| clear for the business as we were able to compile a list of
| recurrent issues during every issue, calculate a financial loss
| and present it for inspection each and every time they either
| began a witch hunt for downtime or refused to allocate time to
| backlog. Small wins.
| Kwpolska wrote:
| That seems like a very long-winded way to say you hate on-call,
| which is a completely normal thing to do. That said, is on-call
| effectively mandatory or very popular in the US startup world?
| Because here, in the European established company world, I can't
| really recall seeing a job posting with on-call listed.
| spongebobstoes wrote:
| on-call is ubiquitous in the US tech industry. I've never had a
| job without it.
| Carrok wrote:
| I've been on-call for the past 10+ years.
| slt2021 wrote:
| being oncall forces the quality of software to improve.
|
| if you want fewer incidents: ensure better QA, monitoring,
| smaller rollouts
|
| usually developers start becoming more conservative after they do
| few oncall shifts and suddenly prioritize important reliability
| improvements, instead of shiny new features nobody will use
| geoffpado wrote:
| Being on-call forces the *desire* for the quality of software
| to improve. Shitty management can and will override that. We
| don't have time for QA or to waste an engineer adding
| monitoring, we gotta ship ship ship.
| darioush wrote:
| this doesn't always work. many things can go wrong in
| distributed systems and you cannot test for all of them. also
| you have no control of your dependencies like when AWS
| networking degrades or a 3rd party API provider changes their
| APIs without letting you know.
| Animats wrote:
| For "non-exempt" employees, that's paid "stand-by time"
| California.[1] Also see this case involving on-call coroners.[2]
|
| The way this works in most unionized jobs is that there's a
| stand-by rate paid for on-call hours, plus a minimum number of
| hours at full or overtime pay, usually four, when someone is
| called to duty. This is useful to management - if the call
| frequency is too high, it becomes cheaper to hire an additional
| person.
|
| [1] https://www.dir.ca.gov/dlse/CallBackAndStandbyTime.pdf
|
| [2] https://casetext.com/case/berry-v-county-of-sonoma
| mjcarden wrote:
| This article gave me unpleasant flashbacks to the first half of
| 2023. I resigned from planet.com in mid 2023 due to the stress
| caused by being on-call every second week. It took me six months
| to get my head into a healthy state again. Now I have a _much_
| better job, better paid and no possibility of on-call, ever.
| purplejacket wrote:
| Here's an idea: Compensate any on-call work received during off
| hours at 10X the normal hourly rate. E.g., if my salary is $150K
| per year, then my hourly pay rate is about $75 per hour, so
| compensate my on call work at a rate of $750 per hour. Thus if I
| get a call at 10pm, log in to my laptop and work for 30 minutes
| to resolve the issue to a satisfactory level, then I pocket $375.
| That puts a financial incentive on companies to structure their
| on call protocols so that only the most important calls are
| handled. And I can envision variations on this theme. Different
| sorts of on-call disasters could offer bids for how much they're
| worth to fix based on some automated rubrick, and anyone on the
| ENG team could pick these up on a first-come, first-serve basis.
| Or various combinations of the above for a guaranteed backup
| person. But the companies should offer enough incentive to make
| it worthwhile. And this is in the companies' own best interest.
| To maintain a workforce that can think clearly during the normal
| work, to have a good reputation in the industry, to get good
| reviews on Glassdoor, etc.
| mortar wrote:
| Good suggestion and I can see the benefit for honest people,
| but unfortunately it's as equally a system for financial abuse
| for others - sometimes enough to prevent people fixing things
| during their regular hours just to benefit at other times.
|
| A good counter balance to this might be to offer even more
| compensation for no incidents, or otherwise well handled
| incidents that go on to squash types of that incident now and
| into the future.
| spongebobstoes wrote:
| This makes it in the employee's interest to obfuscate and
| extend any remediation, to get paid more.
| ergl wrote:
| Having overtime pay that is a multiple of regular hourly rate
| is mandatory is many countries in Europe. Are you saying that
| European software tends to be more obfuscated? (answer: it is
| not).
| prawn wrote:
| Wouldn't that incentivise staff to take longer to fix issues?
| Once you've been interrupted, you might as well turn 30 minutes
| into 60 minutes, etc.
| dadkins wrote:
| I just want to point out that the answer is shift work. Here's an
| example of an SRE job at a national lab:
|
| https://lbl.referrals.selectminds.com/jobs/site-reliability-...
|
| "Work 5 shifts per week to monitor the NERSC HPC Facility, which
| includes 2 - 3 OWL (midnight - 8am) shifts. Some days may be
| onsite, some may be offsite. The schedule will be determined by
| staffing needs."
|
| 40 hours per week, full salary, full disclosure about the night
| shifts, but none of this 24x7 wake up in the middle of the night
| on top of your regular job bullshit that the tech industry
| insists on.
___________________________________________________________________
(page generated 2025-03-27 23:00 UTC)