[HN Gopher] GitHub Partial Outage
___________________________________________________________________
GitHub Partial Outage
Author : danfritz
Score : 170 points
Date : 2025-11-13 15:04 UTC (7 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| danfritz wrote:
| Related to the recent announcement they are moving to Azure?
| the_af wrote:
| Wow. It wasn't already running on Azure? What was it (or is it)
| running on?
| noir_lord wrote:
| iirc (it's been a while) they where on rackspace when
| Microsoft bought them out - there was an article a few months
| ago saying they where moving to Azure and freezing new
| features while they do the move[1].
|
| [1] https://thenewstack.io/github-will-prioritize-migrating-
| to-a...
|
| Honestly I don't know half the features they have added
| because the surface is huge at this point everyone seems to
| be using a (different) subset of them anyway.
|
| So a feature freeze isn't likely to have much impact on me.
|
| EDIT: went and checked - https://github.blog/news-
| insights/github-is-moving-to-racksp... not sure if they moved
| again before the MS acquisition though.
| nixgeek wrote:
| A team of us moved it off Rackspace in 2013, it's been
| mostly in a set of GitHub operated colo since then. Used to
| be there was some workloads on AWS and a bit of
| DirectConnect. Now it's some workloads on Azure.
|
| To the best of my knowledge there's been no Rackspace in
| the picture since about 2013, the details behind that are
| fuzzy as it's been 10+ years since I worked on
| infrastructure at GitHub.
| antn wrote:
| yeah, we did not have anything in Rackspace for many
| years before the Microsoft acquisition. I remember having
| to migrate some tiny internal things off of Heroku,
| though!
| le_stoph wrote:
| In the Pragmatic Engineer podcast episode with the former CEO
| of Github, the latter mentioned that they had their own infra
| for everything. If I remember correctly, this was due to the
| fact that Github is quite old and at the time when Github
| Actions became a thing, cloud providers were not really
| offering the kind of infra that was necessary to support the
| feature.
| Kwpolska wrote:
| GitHub is old, but GitHub Actions are not. Indeed, GitHub
| Actions launched two months after the Microsoft acquisition
| was announced [0], and it is a half-assed clone of Azure
| Pipelines.
|
| [0] https://en.wikipedia.org/wiki/Timeline_of_GitHub
| saghm wrote:
| I can't read the entirety of this article[1] because it's
| paywalled, but it looks like they ran their own servers:
|
| > GitHub is currently hosted on the company's own hardware,
| centrally located in Virginia
|
| I imagine this predates their acquisition from Microsoft.
| Honestly, given how often Github seems to be down compared to
| the level of dependency people have on it, this might be one
| of the few cases where I might have understood if Microsoft
| embraced and extended a bit harder.
|
| [1]: https://www.theverge.com/tech/796119/microsoft-github-
| azure-...
| loloquwowndueo wrote:
| Well... https://www.reuters.com/technology/microsoft-azure-
| down-thou...
| saghm wrote:
| Fair enough, my Azure experience is minimal enough that
| maybe I shouldn't make assumptions about whether this
| would improve things. That being said, I do think there's
| merit in the idea that if Microsoft is going to be able
| to solve this problem, they probably should try to solve
| it just once, and in a general way, rather than just for
| Github?
| balamatom wrote:
| >Microsoft
|
| >solve it just once, and in a general way
| drcongo wrote:
| Oh no. I look forward to watching my browser redirect 40 times
| on every attempted page load.
| bob1029 wrote:
| https://news.ycombinator.com/item?id=45517173
| stackskipton wrote:
| Doubt it. I'm Ops person on Azure, while they just had terrible
| outage recently, they tend to be as stable as any other cloud
| provider and I haven't had many issues with Azure itself
| compared to whatever slop the devs are chucking into
| production.
| whoknowsidont wrote:
| >they tend to be as stable as any other cloud provider
|
| Absolutely not.
| Fokamul wrote:
| Not Sharepoint? What a bummer.
| arccy wrote:
| Your weekly reminder to take a break
| hoherd wrote:
| I still can't pull new branches even though the incident says
| it's resolved. I don't think my boss would be happy with me
| taking a break this long... but what else can I do when our
| business uses GH?
| theletterf wrote:
| I was getting crazy thinking that there was something wrong with
| my SSH keys all of a sudden. Thanks $DEITY it's just GitHub.
| no_wizard wrote:
| Same. I reflex replaced mine thinking it needed to be. Glad its
| working now though
| rvz wrote:
| Looking forward to the postmortem.
|
| Are they using AI agents this time to resolve the outage?
| Probably not.
|
| But this time, there is no CEO of GitHub to contact and good luck
| contacting Satya to solve the outage.
| stuffn wrote:
| The postmortem will be simple since Github goes down so
| consistently every week you can almost use it as an alternative
| timekeeping system.
| ares623 wrote:
| The pulsar of web services
| gunalx wrote:
| Yep. Was using github for oauth on a petproject of mine. Got the
| unicorn, and was considering takingthe break, or just etting up
| something else. Seems to be running again for me now though.
| nkzd wrote:
| I thought my SSH keys were revoked, whew.
| coffeebeqn wrote:
| Just started to replace mine when I saw someone post a message
| about GitHub
| JLCarveth wrote:
| This sure does seem to happen a lot
| fishgoesblub wrote:
| Must be a day ending in Y.
| numbsafari wrote:
| Anyone using GitLab have any insight on how well their operations
| are running these days?
|
| We originally left GitLab for GitHub after being bit by a major
| outage that resulted in data loss. Our code was saved, but we
| lost everything else.
|
| But that was almost 10 years ago at this point.
| colesantiago wrote:
| No issues on GitLab.
|
| Haven't seen any outage from GitLab in like, ever.
| philipwhiuk wrote:
| https://status.gitlab.com/pages/history/5b36dc6502d06804c083.
| ..
| colesantiago wrote:
| Never had any problems really.
|
| GitHub on the other hand has outages more frequently.
| tux3 wrote:
| That has definitely not been my experience. I like Gitlab,
| but they've had regular incidents all along. If a git push
| failed I wouldn't question it, it's almost never my network.
| I'd just open Gitlab's Gitlab and find the current active
| issue.
|
| To Gitlab's credit their observability seems to be good, and
| they do a good job communicating and resolving incidents
| quickly.
|
| Some companies that shall not be named have status pages that
| always show green and might as well be a static picture. Some
| use words like "some customers may have experienced partial
| service degradation" to mean "complete downtime". Gitlab also
| has incidents, but they're a lot more trustworthy. You can
| just open the issue tracker and there's the full incident
| complete with diagnosis.
| boilerupnc wrote:
| Not sure what specific operational services are of interest -
| but here's a link to their historical service status [0]
|
| [0]
| https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
| kaishiro wrote:
| We use GitLab on the daily. Roughly 200 repos pushing to ~20 on
| any given day. There have been a few small, unpublished outages
| that we determined were server side since we have a geo-
| distributed team, but as a platform seems far more stable than
| 5-6 years ago.
|
| My only real current complaint is that the webhooks that are
| supposed to fire in repo activity have been a little flaky for
| us over the past 6-8 months. We have a pretty robust chatops
| system in play, so these things are highly noticeable to our
| team. It's generally consistent, but we've had hooks fail to
| post to our systems on a few different occasions which forced
| us to chase up threads until we determined our operator
| ingestion service never even received the hooks.
|
| That aside, we're relatively happy customers.
| gen220 wrote:
| FWIW, GitHub is also unreliable with webhooks. Many recent GH
| outages have affected webhooks.
|
| They are pretty good, in my experience, at *eventually*
| delivering all updates. The outages take the form of a
| "pause" in delivery, every so often... maybe once every 5
| weeks?
|
| Usually the outages are pretty brief but sometimes it can be
| up to a few hours. Basically I'm unaware of any provider
| whose webhooks are as reliable as their primary API. If
| you're obsessive about maintaining SLAs around timely state,
| you can't really get around maintaining some sort of fall-
| back poll.
| kaishiro wrote:
| Completely agree on all points. We've had dual remotes
| running on a few high traffic repos pushing to both GitLab
| and GitHub simultaneously as a debug mechanism and our
| experiences mirror yours.
| cmckn wrote:
| > you can't really get around maintaining some sort of
| fall-back poll.
|
| This has been my experience with GitHub Actions as well,
| which I imagine rely on the same underlying event system as
| webhooks.
|
| Every so often, an Action will not be triggered or
| otherwise go into the void. So for Actions that trigger on
| push, I usually just add a cron schedule to them as well.
| ctkhn wrote:
| My org hosts it on prem, and while I don't like the way pages
| are organized for projects, I only really interact with the PR
| page and that is laid out well. Most of my interaction with git
| is happening from my terminal anyway so -\\_ (tsu)_/-
| geoffbp wrote:
| We're using gitlab, loads of issues and outages, we want to go
| to github
| contravariant wrote:
| Ah that was why. Oh well, I just needed to get the code to the
| server, so I didn't really need Github anyway.
| ecshafer wrote:
| Github is owned by Microsoft, so this is a pretty small time
| indie operation, you need to give them a break.
| cube00 wrote:
| Not replacing the CEO suggests they aren't focusing on it as
| much as they were.
| lysace wrote:
| Just your casual $3.8T company.
|
| There were so many severe Github Actions outages (10+ ?) in the
| past year. Cause: Migration to the disaster zone also known as
| Azure, I assume. Most of them happened during (morning) CET
| working hours, as to not inconvenience the americans and/or
| make headlines.
|
| Money doesn't buy competency. It's a long-term culture thing.
| You can never let go on maintaining competency in your
| organization. It rots if you do. I guess Microsoft did let go.
| conception wrote:
| "guess Microsoft did let go" - are we thinking of the same
| Microsoft here?
| lysace wrote:
| I am thinking of the atrophying one. Not MikeRoweSoft.
| mook wrote:
| I thought GitHub Actions (in particular; not the rest of
| GitHub) was always Azure, because it was initially a fork of
| Azure Pipelines?
|
| GitHub as a whole, including the previously non-Azure bits,
| does seem flakier than a few years ago though, for sure.
| lysace wrote:
| You seem to be correct. Not that much visible from the
| outside, but yes it seems like they always ran on Azure,
| from the 2018 launch. (Apologies for the disinfo, although
| I qualified it with the "I assume".)
| jedahan wrote:
| Pre-launch I seem to recall using an entirely different
| product with the same name, that supported CUE or HCL and
| had a better gui editor. I think post acquisition they
| scrapped it for the current (and IMO) worse reskin.
| degamad wrote:
| It's possible that, even though the Actions part was always
| on Azure, migrating the other parts to Azure broke some
| connectivity between the pieces....
| isodev wrote:
| I bet Microsoft is sad not because people can't push, but
| because the training data for Copilot has slowed down.
|
| PS: None of our 40+ engineers felt anything, our self hosted
| Forgejo is as snappy as ever.
| no_wizard wrote:
| until your hardware fails! Or your VPS provider goes down!
|
| Or whatever else, software services going down is going to
| happen in some capacity, eventually. Real question is what is
| acceptable
| isodev wrote:
| When you self host you also learn how to backup. It's not
| complicated actually, you should look into it.
| arnvald wrote:
| I'm old enough to remember when GitHub was on main page due to a
| cool feature they added, now they just end up here when it stops
| working
| wavemode wrote:
| It's possible that Microsoft buying GitHub was a large-scale
| psyop intended to reduce the productivity of the competition.
|
| Any time their startup competitors are making too much progress
| they can just push the "GitHub incident" button and slow everyone
| down.
| grepfru_it wrote:
| We used to obsessively care about 500s. Like I would make a
| change that caused a 0.1% spike in 500s and I would silently
| say I'm sorry to the folks who got the unicorn page.
|
| I'm not sure the new school cares nearly as much. But then
| again this is how companies change as they mature. I saw this
| with StubHub as well.. The people who care the most are the
| initial employees, employee #7291 usually dgaf
| 0x1ch wrote:
| I fall into the new school gen z category, and I think you're
| right. We don't care. We don't care about the problems
| started before us, and we owe nothing to no one (but our
| employers, must increase value for shareholders of course).
|
| I simply want to survive. I'll kiss ass where I have to, but
| not to people I don't work on behalf of.
| ares623 wrote:
| Hell yeah
| kataklasm wrote:
| Can't say that's entirely true for me ('02). If my [
| employer, supervisor, ... ] provides me with logical,
| traceable tasks with their context properly laid out, I can
| totally put a ton of effort into providing meticulous, well
| thought out solutions, that are as good as it gets under
| the provided constraints. It's the non-sensical (be it
| actually non-sensical or just not understood enough because
| of unprovided context) tasks that make me not care.
| Arch485 wrote:
| I'll throw in my $0.02, as a fellow zoomer. I care about
| the things that are mine (as in, my code, my decisions,
| etc. etc.). But if management fucks up and tells me to fix
| it, there is no amount of money that will make me care.
| Especially if I advised management _not_ to do that in the
| first place.
| dustfinger wrote:
| Why does the main page show all green when there is an ongoing
| incident? All green here -> https://www.githubstatus.com/
| gkoberger wrote:
| It's marked as resolved for some reason
| blibble wrote:
| because then some mid-level manager gets a telling off
|
| and/or has to pay the SLA out of their budget
| dustfinger wrote:
| ahh, you are right. I am blind.
| zamalek wrote:
| This is normal for Microsoft. It's as though status is owned
| and controlled by either marketing or accounting, not
| engineering.
| pfyra wrote:
| Coincidentally, Azure Devops was also missing the ssh keys
| earlier today, both in the web ui and for ssh login.
| spockz wrote:
| Well, github is moving to Azure and they are consolidating
| systems. No surprise there.
| carlyai wrote:
| thought i was going crazy
| prymitive wrote:
| Speaking of GitHub issues if you go to Insight->Traffic in your
| repo you'll most likely see this banner:
|
| " Referring sites and popular content are temporarily unavailable
| or may not display accurately. We're actively working to resolve
| the issue."
|
| It's been like that for months now with no sign of anyone working
| on it. They just don't care about user experience anymore.
|
| https://github.com/orgs/community/discussions/173494
| embedding-shape wrote:
| Speaking of "temporarily unavailable but it's actually
| forever", I've been wanting to get into Fallout and Starfield
| modding, so been waiting for their official wiki to come out of
| maintenance mode. I think I first tried to access it when
| Starfield launched (September 2023), and still today it is
| "currently down for backend maintenance".
| https://wiki.bethesda.net/
| whoknowsidont wrote:
| Another outage brought to you by Azure.
___________________________________________________________________
(page generated 2025-11-13 23:00 UTC)