[HN Gopher] Building an in-house on-call training program for yo...
___________________________________________________________________
Building an in-house on-call training program for your SRE team
Author : LenkaBo
Score : 53 points
Date : 2023-02-10 13:49 UTC (9 hours ago)
(HTM) web link (fiberplane.com)
(TXT) w3m dump (fiberplane.com)
| bennyp101 wrote:
| First step should be to ensure you have enough people that you
| can do an on-call schedule without burning out your handful of
| employees, second should be that you tailor your notifications so
| they don't get overwhelmed by "outages" that can actually be
| dealt with by other teams or left until the workday, thirdly make
| sure you pay people for doing so - so many places just assume
| that you will be about.
| JTbane wrote:
| Knowing corporate bean counters, step 1 and 3 will be
| unattainable due to "budget constraints".
| shahargl wrote:
| I'm trying to build OSS tool that will help fighting on-call
| fatigue so if you hate it let's talk and tell me how can I help
|
| https://calendly.com/d/zkv-4f8-jgj/keep-intro
| hummus_bae wrote:
| [dead]
| 0xbadcafebee wrote:
| > An on-call training program is essential to building and
| maintaining a successful incident response practice and culture.
|
| I think that employee training _in general_ is essential to the
| overall efficiency and quality of your business. Companies always
| skimp on it, and you can see how it hurts productivity and the
| bottom line. But it 's not necessarily going to impact your
| culture or day to day practices.
|
| The thing that's going to maintain a successful incident response
| practice is follow-through. Training just tells them what they're
| supposed to do. But if they're not _doing it_ , training doesn't
| matter.
|
| What's the most common problem with on-call? It's not waking
| people up in the middle of the night, or a lack of runbooks, or
| lack of postmortems. It's the follow-through. When you have an
| incident, and you "deal with it", and even have a postmortem,
| what's the last thing on the page? "Action Items" or "Next
| Steps". And are those action items actually getting done? Is a
| regular review of your practices getting done? Is the same alert
| popping up again and again? _That_ is the biggest problem with
| on-call, and training doesn 't solve that.
|
| To run a good incident response process you need people who are
| getting paid to do two things: 1) due diligence, 2) continuous
| improvement.
|
| 1) Give someone the primary responsibility of ensuring everyone
| is following the process. Do not let action items stagnate. Do
| not let weeks go by without writing the postmortem. Do not have
| alerts without resolutions, or runbooks for that matter. Somebody
| needs to get paid to do this, because it's boring as hell, not
| fun, but needs to be done.
|
| 2) Make sure the process is getting easier, and better. You don't
| want to have to check up on everything all the time, of course,
| and it doesn't scale well. So you want to improve your process so
| that it's more likely that the right things get done the first
| time.
|
| You can train your folks what to do. And you can "get by" with an
| incident response process that nobody actually follows and is
| never improved. But as incidents mount, it will drain your people
| and org more and more. By all means, if you're currently facing a
| deluge of incidents, do training! But the due diligence and
| continuous improvement is what's actually gonna push incident
| numbers down.
| ElevenLathe wrote:
| Can't agree more. I've worked in incident management for years
| and my superiors are always happy when we have an incident but
| one of our incident managers and a few SREs solve it relatively
| quickly. Yes, there is some talk of "how do we avoid something
| like this in the future?" but the answer (even when using "5
| Whys") is always a shallow one like adding even more gates and
| change review or, best-case, adding active-active failover. The
| answer is never "This team ships once every other month. We
| need to give them a mandate to move that up to multiple times a
| day."
|
| Still, it's nice to have SREs who understand the rhythms of an
| active incident instead of somebody whose last conversation
| with someone outside their immediate team was in 2016 and
| seemingly doesn't understand that they're on call or for what
| services.
| [deleted]
| netik wrote:
| Advertisement disguised as a useful post. Like a lot of
| marketing, it ends with "use our product"
| magicloop wrote:
| When I started out one of my first jobs came with an on-call
| pager. It was a puck shaped item that you carried on your belt.
| When sitting on the toilet it would slide out of its carrier and
| fall into the basin. Nearly happened twice to me (I caught it).
| Several others had it fall in. It had to be vigorously cleaned
| before hand over to the next engineer. That is what is called a
| "Clean Handover".
| cody_coconuts wrote:
| So this is what they were talking about when they said "shitty
| team culture"
| blue039 wrote:
| So many words and none of them include "pay people overtime for
| working on call".
| dopylitty wrote:
| Exactly this. On call is an abusive practice. If you think your
| service should be up 24/7 then pay people to be online to take
| care of it 24/7.
|
| Forcing people to sit by their phones/computers 24/7 unpaid
| waiting for a system to maybe have a problem should be a
| practice explicitly barred by the FLSA.
| olvy0 wrote:
| Completely agree. My team has on-call duty, but we don't have
| to do it 24/7, just during business hours. I hope we never
| get to the point where we're giving support 24/7.
|
| Slightly off topic: stdout's On-Call.
|
| https://www.youtube.com/watch?v=OiApm-OFfsY
| dangerwill wrote:
| Yeah I went from a job where being oncall meant a LOT of
| pages but the oncall rota was a follow the sun model and so
| the oncall shifts just ended up being normal work days where
| you were just 100% interrupted constantly. I thought this was
| hell but now that I work at a company with 72 hour long
| oncall shifts, even though the after-hours page rate is very
| low (once a month or so), the "always on" feeling is even
| worse.
|
| I am constantly thinking about how abusive this practice is
| makestuff wrote:
| Yeah IMO you either use a follow-the-sun rotation and have
| staff in multiple parts of the world, hire a dedicated
| nightshift, or pay OT.
|
| If the SRE is unable to solve the issue and needs to page the
| escalation engineer they should get OT for that if it outside
| business hours. The whole "oh you got woken up at 3am just
| log in late" is BS because I still have the same amount of
| work to do.
| MacsHeadroom wrote:
| At a unicorn I was paid 1.5x for every on-call hour, even while
| sleeping. That makes 40 hours in a week and 128 on-call hours
| worth 1.5x for a total of: 232 paid hours in one week with on-
| call.
|
| That's almost 6 weeks of pay in a single week. Yet we still had
| to pull teeth to get people to be on-call because of how much
| it sucks.
| tayo42 wrote:
| Yeah, There's diminishing returns on happiness derived from
| money. the extra few thousand when Im making 12k a month
| already doesn't offset it.
| jedberg wrote:
| I did on-call for work for 20 years. I just assumed my paycheck
| needed to cover that hassle. I usually got paid more than other
| people with similar experience because my job included on-call
| and theirs didn't.
|
| Overtime isn't necessary if expectations are set up front and
| the salary is boosted accordingly.
|
| But yeah, it makes people feel better if you pay them less
| overall and then give them overtime when they're on-call. It's
| also better for the company because then the raises are smaller
| since they're based on the base salary and not the overtime.
|
| Honestly I'd rather just get paid a higher salary with expected
| on-call hours up-front.
| blue039 wrote:
| It's not about "making them feel better". It's about wage
| theft from workers.
|
| If you pay me N extra hours, where N is less than the hours I
| worked on call then you stole from me. Plain and simple. This
| should be pursued by rabid labor lawyers and interest at the
| prevailing 30 year rate should accrue on those dollars hourly
| until they are paid back.
|
| The problem is, of course, the law dictating "exempt workers"
| was written in a time where the only people that would be
| exempt were executives with actual million dollars golden
| parachutes. Now, white collar labor is almost universally
| exempt allowing the state sanctioned theft of millions of
| dollars a year by corporations who lobby to keep the laws the
| same, and promoted by "grind culture" immoral bottom-barrel
| blood-suckers like PagerDuty.
|
| It's bullshit. You're suffering from Stockholm syndrome.
| You've somehow managed to say stealing from you is okay
| because they put some nominal amount of unrelated dollars in
| front of you.
| jedberg wrote:
| It's not stealing if they tell me up front "you will be on
| call and we expect you to work at least 10 hours a week
| outside of normal business hours". And then pay me for my
| time up front as part of my salary.
|
| Someone who is on-call _should_ be exempt from overtime if
| they 're not paid hourly _and it was clearly communicated
| up front_. I think that 's part that's missing. Most jobs
| don't make that clear up front.
| sambull wrote:
| Took my company being bought by a european one to get that to
| happen.
|
| On-call for many companies (most I've worked at in the past 23
| years) is a borderline abusive practice; and often times they
| already run the team on fumes.
| Ancalagon wrote:
| I think the position "Site Reliability Engineer" has become
| synonymous with people being perpetually on-call as their job,
| at least in the U.S. The job is basically just there to put out
| fires.
| LenkaBo wrote:
| You're right. And that's the most crucial point!
___________________________________________________________________
(page generated 2023-02-10 23:01 UTC)