[HN Gopher] GitHub Incident
___________________________________________________________________
GitHub Incident
Author : aggrrrh
Score : 106 points
Date : 2026-01-15 17:00 UTC (6 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| nullfish wrote:
| I suspect the migration to Azure is continuing to go well
| rvz wrote:
| Yes indeed. 6 years of non-stop outages across the platform
| every month.
|
| Even self-hosting would have been more stable than sitting on
| GitHub as predicted more than half a decade ago. [0]
|
| Now there is no 'CEO of GitHub' to contact this time (Satya
| does not care).
|
| [0] https://news.ycombinator.com/item?id=22867803
| someguyiguess wrote:
| I did not come to hacker news expecting comedy gold but you
| have done it my friend!
| ascendantlogic wrote:
| This feels more like Copilot-as-platform-engineer to me
| DeepYogurt wrote:
| Github's been running on vibe code for a while now and it's
| starting to show
| nottimbo wrote:
| Microsoft, it's time to hire some SREs.
| arm32 wrote:
| We did hire some, boss! Soshie, Vizzy and Dexter. They're AI,
| but they're supposed to be way better than a human SRE. At
| least that's what the Sintra salesguy told us.
| rvz wrote:
| So that's what the Tay, and Zoe AI bots were doing all this
| time after they were cancelled and banned off of Twitter.
|
| Working on the GitHub Azure migration and for years it's gone
| so well so far.
| lenerdenator wrote:
| Why hire anyone to fix a problem when you can make an AI agent
| to "fix" it, tell investors about it to pump the price, and not
| fix anything knowing that you have a monopoly?
| ferguess_k wrote:
| Yes we did hire SREs, unfortunately they are in another
| continent and they only know how to pull others into the chat.
| We also have some AI too, do you want to try them? They are
| pretty good SREs, one of them wrote 100K lines of code in a
| week while another one reviews every line along the way. It was
| fantastic! Fantastic!! FANTASTIC!!!
|
| OK I have no idea about MSFT SREs, just to be /s.
| aruggirello wrote:
| Clippy to the rescue! :-)
| VirusNewbie wrote:
| Microsoft doesn't pay well enough to attract good SRE talent.
| postexitus wrote:
| I believe it is an Azure outage or some type of MS service -
| everything on Azure is down.
| zxcvasd wrote:
| having no issues on azure here, seeing no azure incidents on
| the status page or any of my admin panels
| verst wrote:
| I second this. Not experiencing any Azure issues at this
| time.
| deathanatos wrote:
| > _seeing no azure incidents on the status page_
|
| ... in all seriousness, that is hardly proof that Azure isn't
| having an outage.
| zxcvasd wrote:
| if i thought it alone was proof enough, i wouldnt have also
| included the bit about how i was actively using azure.
|
| its one signal, among others. and in any case, i wasn't
| trying to _prove_ the parent commenter wrong. i was
| offering my own signal to the crowd.
| ctxc wrote:
| My az services seem to be up.
| MadameMinty wrote:
| Angry unicorns seem to be over.
| tapoxi wrote:
| helm repo add gitlab https://charts.gitlab.io/ && helm upgrade
| --install gitlab gitlab/gitlab
|
| I did this in 2019, it avoided so many headaches. CI is better
| too since there's a nice clean mapping of build -> pod for
| everything and I can just exec in if something's borked.
| odie5533 wrote:
| Things would have to get really bad before I considered
| managing my own repositories. Trading someone else's headaches
| for my own.
| 0xbadcafebee wrote:
| ^ this. the last thing i want is to add to my workload. take
| my money and make my life easier, even if it means that for
| one hour every couple months i can't do anything
| NewJazz wrote:
| Have you ever actually hosted gitlab?
| tapoxi wrote:
| It's not as bad as you think, I run the helm upgrade when
| patches come out, the backing store is S3 or managed SQL, it
| runs a nightly k8s cron called gitlab-backup which tarballs
| the whole thing into an s3 bucket with a single command
| restore should disaster strike. (This is part of the product,
| not a thing I wrote.)
|
| I probably only babysit it for 30 minutes per year, including
| all the upgrades.
| nine_k wrote:
| It depends how high you value your headaches, and how high,
| your org's downtime. Github not working accrues over the
| hourly rate of every developer affected, which is likely
| $70-$100 a hour. 10 hours of outage in a year affecting a
| team of 10 would cost north of $70k, enough to hire a part-
| time SRE dedicated just to tend to your Gitlab installation.
| zxcvasd wrote:
| > _10 hours of outage in a year affecting a team of 10
| would cost north of $70k_
|
| 10 hours x 10 developers x $70 per hour = $7000, not
| $70000.
| nine_k wrote:
| Thank you for the correction! This indeed completely
| changes the picture :-\
| corvad wrote:
| Github's recent reliability has honestly been abysmal. Not
| surprised.
| ferguess_k wrote:
| Unless some major customers are moving away, I don't think they
| are going to seriously care about it.
| corvad wrote:
| I suspect some companies may already be considering it.
| Especially with the wealth of alternatives today.
| nine_k wrote:
| What kind of alternatives do you see as viable for
| large(ish) commercial users?
| toephu2 wrote:
| GitHub on-prem. Officially called GitHub Enterprise
| Server. You can have GitHub, but hosted on your own
| servers.
| NewJazz wrote:
| So you still pay them, you do the hosting work, and you
| get a product with worse features than gitlab?
| zxcvasd wrote:
| but you can be smug when theres a github incident, and
| thats hard to put a price on
| NewJazz wrote:
| You can do that with gitlab.
| g947o wrote:
| Companies are already using on-premise GitHub server, if
| they are using GitHub in the first place. There are many
| other self hosted solutions which are quite common in
| enterprise environment.
| supriyo-biswas wrote:
| In my experience companies are _moving into_ GitHub for
| Copilot and GHA.
| appplication wrote:
| GHA maybe, but copilot is just another mid tier player in
| a congested space.
| NewJazz wrote:
| Doesn't stop folks from wanting to buy the MS brand.
| Execs are really out of touch these days.
| pxc wrote:
| What is the quality-first, high uptime alternative to GitHub?
| My employer uses both GitHub and GitLab, and while I think
| GitLab is better, its quality also frankly sucks. It's
| riddled with bugs that have just been marinating on the issue
| tracker for years, and the most common "fix" for gnarly bugs
| in the CI platform is "revise the documentation to reflect
| the existing (broken) behavior".
| stefan_ wrote:
| It's amazing, before we even had ChatGPT, GitLab was
| building so much endless slop halfbaked crap in their
| pursuit of ever more "enterprise checkboxes". Now they have
| slowed right down, no doubt collapsing under the escalating
| maintenance weight of all the nonsense that was created,
| like the canaries in the vibe coding mines telling us of
| impending doom.
|
| Now you go to their blog, theres a banner at the top
| announcing "GitLab Agentic AI whatever is GA (GENERAL
| AVAILABILITY)" and you try to click it its literally a
| fucking 404 not found. That's the level of their stability
| and quality. Try it for yourself:
|
| https://about.gitlab.com/blog/
| fhd2 wrote:
| Maybe it's GU already.
| ferguess_k wrote:
| *Stupid question*: What is so hard about self hosting one's
| own repo? I get it must be difficult for a mega
| corporation, but for companies like us, who have hundreds
| of repos but only 20 of them are regularly used, and
| concurrent read/write is relatively light -- considering
| our largest team is less than 20 persons, so even if all of
| them are reading/writing from the repo, it doesn't seem to
| be a huge issue.
|
| Even for a bigger company, say 5x developers (we have about
| 100+ SWEs and maybe 10-20 other titles who use GitHub), is
| it really a big thing to self host their own repos?
| External applications are definitely on another level
| because you could have hundreds of concurrent visits
| easily.
|
| What did I miss?
| pxc wrote:
| [delayed]
| bakje wrote:
| Perhaps the gemini-cli bot arguing with itself is taking its toll
|
| https://github.com/google-gemini/gemini-cli/issues/16750
| johnisgood wrote:
| Wonderful, lmao.
| lol768 wrote:
| Jeez, what a mess. Some of those issues have over 5000 events
| on them.
|
| I really hope that didn't send emails out to people.
| embedding-shape wrote:
| Haha, reminds me off bringing down office mail servers by
| accidentally creating loops of emails back in the day... What
| is old is new again, but this time with probabilities :)
| pdimitar wrote:
| I could not resist to put my sarcastic comment about RAM price
| increases serving a good cause in there.
| dgxyz wrote:
| Having just had to buy 4TB of RAM, I appreciate this.
| MisterTea wrote:
| That's like 100,000 USD. I keep thinking about making a rap
| video wearing a 10 TB gold chain surrounded by big booty
| girls with their naughty bits covered in m.2 SSD's while
| dissing the AI industry. Though I cant afford the RAM :-/
| zxcvasd wrote:
| like most rap videos do with cars/jets/mansions, just
| rent the ram sticks for a few hours!
| dgxyz wrote:
| Yep that much. 64Gb DDR5 ECC sticks (128Gb don't exist at
| the moment apparently). They declined the PO 6 months
| ago. That'll teach 'em.
|
| I was pissed that there weren't any sticks heading to the
| recycling out of the nodes otherwise I would make myself
| that chain :)
| TheJoeMan wrote:
| It's sad that I can't interpret if you mean to actually
| shoot your rap video on film, or have an AI generate it
| lol. Either way you're going to need RAM.
| kps wrote:
| Shooting on film doesn't need any RAM. Unfortunately the
| price of silver is also through the roof.
| dpacmittal wrote:
| Use Veo
| MattIPv4 wrote:
| https://github.com/google-gemini/gemini-cli/issues/16723 is
| even worse, GitHub shows `5195 remaining items` in the
| collapsed timeline.
| unaut wrote:
| Wow that's whole a lot of yapping
| DeepYogurt wrote:
| Wow. If you look at all the issues this seems pretty common
|
| https://github.com/google-gemini/gemini-
| cli/issues?q=is%3Ais...
| omoikane wrote:
| Maybe the bots need rule of ko.
|
| https://en.wikipedia.org/wiki/Rules_of_Go#Ko
| jbverschoor wrote:
| Good thing git is a distributed system
| dgxyz wrote:
| Virtually no one knows how to do anything with it outside of
| github.
| Joe_Cool wrote:
| That's a them problem.
| nine_k wrote:
| Your favorite search engine or LLM will show you in a second,
| it's really easy.
|
| The problem is that it's not enough. The fact that Github
| uses Git specifically is a technical detail; it could use
| mercurial equally easily, as Bitbucket used to. Github
| Actions, OWNERS files, PRs and review tools, issue tracker,
| wiki are all not Git features.
| dgxyz wrote:
| Not a chance. I think you need to spend some time in low
| ball corporate IT. It's just monkeys throwing faeces at the
| wall. We only just levered them off subversion...
|
| (I use Fossil 100% offline for personal projects for ref)
| tonymet wrote:
| i still find insightful ways to use git every day. amazing
| tool. it's a shame for those who only see it as "how to sync
| my repo with my coworkers"
| TZubiri wrote:
| You might be surprised, but that's not true at all.
|
| I once read someone commenting "Nobody writes code by hand
| without looking syntax up".
|
| Man, you are just outing yourself as a complete beginner, the
| field is way deeper than you imagine and it's not even close.
| dgxyz wrote:
| Not really. I've been around a while. Git for about 15
| years. Subversion before that. Perforce before that. rcs
| before that (back down to sun3 machines). Mostly Fossil now
| for personal things.
|
| What I am saying is that people learn as much as they need
| to. They generally don't need to know any more git than is
| required to interact with github. If anything problematic
| comes up, they go in with a wrecking ball because they
| don't truly understand what they are doing. And git has a
| lot of wrecking balls available.
|
| If you threw them at raw git and asked them to collaborate
| with someone they'd be up shit creek. They have no idea how
| SSH or email works for example.
| nine_k wrote:
| Git is!
|
| PRs and code review are not. CI/CD is not.
|
| I mean, there are solutions, but none of them seems to have a
| large enough mindshare and efficiency. (Even though Github's
| code review tools are pretty spartan.)
| globular-toast wrote:
| > PRs and code review are not. CI/CD is not.
|
| They can be. A PR can be made and code review conducted by
| submitting a patch to a mailing list. That's how the kernel
| and, I think, git itself is developed.
|
| CI/CD is really a methodology. It just means
| integrating/deploying stuff as soon as its ready. So you just
| need maintainers to be able to run the test suite and deploy,
| which seems like a really basic thing.
| TZubiri wrote:
| True, workers can still commit to their local git.
|
| I've been looking into having a separate git server that we can
| commit to and add plain ole git hooks to, and just having it be
| synced with github as a clone.
| sirmoveon wrote:
| Check out Gitea. Its kind of a clone of github but you can
| self host.
| phtrivier wrote:
| Fixed in about 30m to an hour.
|
| Definitely annoying, but I'll try the hot take that, contrary to
| popular belief, GH is not critical infrastructure - or so I hope.
|
| Please tell me no part of the Ukrainian air defense system
| depends on a gh action hook.
| eddd-ddde wrote:
| You've heard of infrastructure as code, now presenting air
| strikes as code!
|
| Need a new secret offensive operation? Create a new JSON file
| with the coordinates, make a merge request and get Commander
| approval, merge it, and our new proprietary GitHub action
| runner will deploy a drone in seconds!
| philipallstar wrote:
| This is far too simple. The correct way is to generate an NFT
| that's a screenshot from Google Maps of where you'd like to
| hit, and a blockchain-watching AI will spot it, figure out
| where you probably mean and send the coordinates to the fire
| control system.
| ares623 wrote:
| When millions of man-hours are lost waiting for your service to
| be back up, I think that deserves a bit of resiliency.
| vaylian wrote:
| It's not critical, but there's still a lot of reliance on it.
|
| It's also the only reason why I still need IPv4.
| NewJazz wrote:
| The status page says things are still not fixed.
| toephu2 wrote:
| This is why companies should host their own source code on-prem.
| howToTestFE wrote:
| If GH has an issue, it seems to always be around 4pm or 5pm GMT.
| I'm starting to think that i should avoid any planned production
| releases around this time.
| andrewinardeer wrote:
| Days since last GitHub incident: 0.2
| imglorp wrote:
| 14 incidents this month. So far.
___________________________________________________________________
(page generated 2026-01-15 23:01 UTC)