[HN Gopher] GitHub availability report: October 2022
___________________________________________________________________
GitHub availability report: October 2022
Author : edmorley
Score : 41 points
Date : 2022-11-02 16:41 UTC (6 hours ago)
(HTM) web link (github.blog)
(TXT) w3m dump (github.blog)
| brink wrote:
| I'm not sure about the technical details behind their outages
| since they're a little vague on that, but it's funny how every
| Rails developer champions Github as a Ruby on Rails shop as why
| Rails should continue living on when their availability is some
| of the worst in the tech scene. Lazy evaluation is great until
| it's not.
| [deleted]
| pizza234 wrote:
| All the incidents seem to be platform-agnostic:
|
| - improper database validation
|
| - older component not tested against configuration change
|
| - uncontrolled automation DOS
|
| - incompletely distributed secrets
| tenderlove wrote:
| You're not sure about the technical details, but it's clearly
| the technology's fault?
| o_m wrote:
| Isn't it GitHub Actions that has been having the availability
| problems for a while now? It started having problems after
| Microsoft acquired GitHub and there has been speculation that
| it is because it was ported to .NET and Azure. Is codespaces
| created with Rails?
| erk__ wrote:
| Actions was not a thing until after GitHub was acquired, but
| idk if they used something else in the early days of actions.
| jabart wrote:
| It tends to have a lot of abuse where people try to run
| crypto miners on free github action accounts. Azure has been
| more stable the past few years and .NET wouldn't have any
| issues with stability at scale. Likely just a hard problem to
| solve at the scale they run at.
| c2h5oh wrote:
| IDK what is the reason, all I know it's been down so much
| we've blacklisted it for anything of importance and migrated
| projects already using it to something else.
| JoyrexJ9 wrote:
| Trust me, nothing has been ported to .NET, nothing would be
| gained from such a move. That's not how Microsoft works.
| Source: I work at Microsoft
| kgeist wrote:
| >Attempting to retry these failed jobs tied up our worker and it
| was unable to process new incoming events, resulting in a severe
| backlog in our queues.
|
| Interesting - we have this kind of thing quite often. Basically,
| an event is stuck in the queue due to a logic error or a prior
| race condition, and it's endlessly retried blocking the rest of
| the events from being processed. We can't just automatically
| remove such an event from the queue because events must be
| processed in order or client data can get corrupted. It requires
| manual intervention (we have alerts in place), and every time
| it's a new event so we have to be creative and think quickly -
| how to unblock the queue without corrupting client data by
| skipping events. After an event is unstuck, there's a huge queue
| of unprocessed events which can take up to a few hours to be
| emptied in worst cases. Fortunately we have some sharding in
| place so there can be several independent workers processing the
| same global queue - with workers' shard affinity we can process
| shard data in order AND in parallel, so SRE can temporarily
| increase the number of workers when the queue gets too large, to
| speed it up. I still don't know how to solve this kind of problem
| once and for all (i.e. to have zero manual intervention). Is it
| even solvable?
| simonpantzare wrote:
| Sounds like a DAG based task orchestrator could be a good fit.
| Where tasks state their dependencies and are allowed to run
| only when they have all completed.
| chris24680 wrote:
| It's amazing how many people here are able to precisely diagnose
| what GitHub should 'just' do, especially without access to their
| code base or experience with working at their scale.
| everfrustrated wrote:
| Of all the many SaaS vendors I use, GitHub has the worst
| availability by far.
|
| There isn't a month that goes by without our devs being impacted.
|
| GitHub - please just work on fixing this. Your product is great
| but your availability is your biggest problem. It's beyond a joke
| at this point.
| rozenmd wrote:
| Thankfully the web app was relatively stable last month
| compared to September: https://github.onlineornot.com/
| lol768 wrote:
| > Of all the many SaaS vendors I use, GitHub has the worst
| availability by far.
|
| Have you ever tried using GitLab?
| handsclean wrote:
| GitHub's reputation for reliability really did a 180 after the
| Microsoft acquisition, as some predicted. It's strange to me
| that despite this, you still get vociferous argument every step
| of the way to blaming Microsoft. People don't remember GitHub's
| past reputation for excellent reliability, then they accuse you
| of rose tinted glasses, then they say we're just noticing it
| more now, then they say GitHub's complexity significantly
| changed at a time that just happened to coincide with the
| acquisition, then they say better reliability is impossible.
| No, man, Microsoft acquired it, and when they got around to
| transitioning it to their infra, reliability plummeted.
| group_love wrote:
| > There isn't a month that goes by without our devs being
| impacted.
|
| +1 -- same experience here for a medium size (60) eng team.
| rvz wrote:
| I know. They have been very unreliable for years as I have
| predicted in here [0] and you can see all the times it went
| down or had intermittent issues [1]. I'm not really surprised
| to see GitHub become less reliable than someone self-hosting a
| typical Git server.
|
| This is why it makes no sense going 'all in' on GitHub
| services.
|
| [0] https://news.ycombinator.com/item?id=22867803
|
| [1] https://news.ycombinator.com/item?id=32752965
| pdimitar wrote:
| Maybe start moving away from Ruby on Rails. Good web stacks in
| Golang and Rust do exist and at that scale they're likely the
| only sensible choices.
|
| How far must the sunk cost fallacy go before something is done?
| speedgoose wrote:
| Which Golang or rust web stacks are as good as rails?
| pdimitar wrote:
| I'm not saying they are _as good_ -- I 'm saying that they
| are good.
|
| Meaning that at one point extra programmer difficulty is
| worth it if your everyday web stack can't keep up.
| speedgoose wrote:
| Oh I see. It's difficult to justify replacing rails, that
| has been successfully used for many years, by something not
| as good (worse?).
| pdimitar wrote:
| Rails is convenient and intuitive, I don't think anyone
| reasonable is arguing that.
|
| My point is that if the stack regularly falls over then
| the programmer convenience has to be sacrificed in favor
| of stable and mega-fast alternative that requires more
| programmer energy.
|
| I love working with dynamic languages. I can prototype
| almost anything that I want to do, in hours. But I also
| recognized the need for a hardcore stack for a previous
| contract and went the long and painful route with Rust.
|
| Result: the project is running for 7 months now, has only
| been restarted 4 times for updating it (re-deployment),
| never crashed once, handles 5000+ network connections and
| streams data from them 24/7.
|
| Peak CPU usage on a 4-core VPS: 27%.
|
| Peak memory usage: 180MB. Normal average memory usage:
| 80MB.
|
| Right tool for the job.
| eddsh1994 wrote:
| Why do you assume the outages are language related and
| not due to the complex product having bugs? How does Rust
| prevent bad schema changes or missing data in the DB?
| pdimitar wrote:
| Because I worked with Rails for 6.5 years. Outages beyond
| smaller scales were at best a weekly occurrence.
|
| Obviously I can't know for sure but it's not an
| uninformed assumption.
| byroot wrote:
| You know you can just click on the post title, that will
| open the posted link in which you can read the detailed
| cause of all the outages they had that month.
|
| If you do this, you will realize that none are close to
| what you describe.
|
| Also have you considered that if you had weekly outage
| when billion dollars companies continued to stick with
| Rails, maybe you were the problem?
| pdimitar wrote:
| I did read the article. One of the incidents was about
| their webhook worker(s) being swamped -- plus had errors
| due to deleted DB workloads that were necessary for the
| event to be processed. So I'd count that one as a slow
| endpoint attributed to Ruby on Rails (and it's famous for
| that).
|
| And even if zero of their incidents alluded to
| performance problems with Rails I still worked a lot with
| it and I know for a fact that it's a factor.
|
| Your snark doesn't change reality but you are free to
| pretend otherwise, fine with me.
|
| > _Also have you considered that if you had weekly outage
| when billion dollars companies continued to stick with
| Rails, maybe you were the problem?_
|
| Indeed, a programmer not having executive powers to
| influence change of deployment tech and server (was Puma
| at the time) is indeed me being a problem, surely.
| Especially after he made a study demonstrating the
| problems and calculated how much programmer time is
| wasted on these matters every week and he still got
| ignored. Perhaps I am the problem indeed!
| byroot wrote:
| > their webhook worker(s) being swamped
|
| That's a capacity problem caused by a logic bug. Nothing
| stack specific. If you throw more work at a system than
| it is designed to handle, you'll hit a bottleneck.
|
| > Your snark doesn't change reality
|
| What reality? You are just barking your uneducated
| opinion. No one who ever worked on a service anywhere
| close to the scale of GitHub (regardless of the stack)
| would make such statements.
| [deleted]
| eddsh1994 wrote:
| > However, many of these events caused exceptions in our
| webhook delivery worker because data needed to generate
| their webhook payloads had been deleted from the
| database. Attempting to retry these failed jobs tied up
| our worker and it was unable to process new incoming
| events, resulting in a severe backlog in our queues.
|
| I bet you I could cause this bug on a Rust product if you
| let me near the code ;)
| pdimitar wrote:
| Oh, absolutely. It can happen everywhere -- in theory.
|
| In practice however, I found people working with certain
| languages and stacks to be more thorough. Still largely
| depends on the person in the important position though,
| that much is always true.
| darksaints wrote:
| Rails is a huge problem, but the most mature
| libraries/frameworks in Go/Rust are all micro-frameworks, which
| isn't much of a replacement. Maybe some .NET frameworks would
| be a better choice.
| pdimitar wrote:
| Well I mean if stability plus performance are the main
| requirements then I'd disqualify everything except Rust.
|
| Though I'd personally do it in Elixir but again, speed.
| GitHub is huge and should rise up to the challenge.
___________________________________________________________________
(page generated 2022-11-02 23:02 UTC)