[HN Gopher] GitHub: Git operation failures
___________________________________________________________________
GitHub: Git operation failures
Author : wilhelmklopp
Score : 273 points
Date : 2025-11-18 20:40 UTC (2 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| shooker435 wrote:
| https://www.githubstatus.com/incidents/5q7nmlxz30sk
|
| it's up now (the incident, not the outage)
| bhouston wrote:
| I cannot push/pull to any repos. Scared me for a second, but of
| course I then checked here.
| dogman123 wrote:
| hell yea brother
| shooker435 wrote:
| The internet is having one heck of a day! we focus on ecommerce
| technology and I can't help but think our customers will be
| getting nervous pre-BFCM.
| MattGaiser wrote:
| https://www.githubstatus.com/incidents/5q7nmlxz30sk
| fidotron wrote:
| It used to be having GitHub in the critical path for deployment
| wasn't so bad, but these days you'd have to be utterly
| irresponsible to work that way.
|
| They need to get a grip on this.
| MattGaiser wrote:
| Eh, the lesson from us-east-1 outage is that you should cling
| to the big ones instead. You get the convenience + nobody gets
| mad at you over the failure.
| bhouston wrote:
| Everything will have periods of unreliability. The only
| solution is to be multi-everything (multi-provider for most
| things), but the costs for that are quite high and hard to
| see the value in that.
| dylan604 wrote:
| yes, but if you are going to provide assurances like SLAs,
| you need to be aware of your own allow for them. if you're
| customers require working with known problem areas, you
| should add a clause exempting those areas when they are the
| cause.
| thinkindie wrote:
| I really can't believe this. I had issues with CircleCI too
| earlier, soon after the incident with Cloudflare resolved.
| arbol wrote:
| I'm also getting this. Cannot pull or push but can authenticate
| with SSH myrepo git:(fix/context-types-
| settings) gp ERROR: user:1234567:user fatal:
| Could not read from remote repository. myrepo
| git:(fix/context-types-settings) ssh -o ProxyCommand=none
| git@github.com PTY allocation request failed on channel 0
| Hi user! You've successfully authenticated, but GitHub does not
| provide shell access. Connection to github.com closed.
| smashah wrote:
| same
| JLCarveth wrote:
| The last outage was a whole 5 days ago
| https://news.ycombinator.com/item?id=45915731
| jmclnx wrote:
| Didn't I hear github is moving to Microsoft Azure ? I wonder if
| these outages are related to the move.
|
| Remember hotmail :)
| bhouston wrote:
| Huh? What were they on before? The acquisition by MSFT is 7
| years ago, they maintained their own infrastructure for that
| long?
| JLCarveth wrote:
| The Github CEO did step down a few months ago, they never
| named a successor. Could have something to do with the
| recent issues.
| https://news.ycombinator.com/item?id=44865560
| silverwind wrote:
| Yes they are/were on their own hardware. The outages will
| only get worse with this move.
| kennysmoothx wrote:
| What a day...
| clbrmbr wrote:
| Same issue here for me. Downdetector [1] agrees, and github
| status page was just updated now.
| cjonas wrote:
| I didn't really want to work today anyways. First cloudflare, now
| this... Seems like a sign to get some fresh air
| dlahoda wrote:
| we depend too much on usa centralized tech.
|
| we need more soverenity and decentralization.
| worldsavior wrote:
| How is this related to them being located in the USA?
| CivBase wrote:
| The sad part is both the web and git were developed as
| decentralized technologies, both of which we foolishly
| centralized later.
|
| The underlying tech is still decentralized, but what good
| does that do when we've made everything that uses it
| dependent on a few centralized services?
| lorenzleutgeb wrote:
| Please check out radicle.dev, helping hands always welcome!
| letrix wrote:
| > Repositories are replicated across peers in a
| decentralized manner
|
| You lost me there
| hungariantoast wrote:
| "Replicated across peers in a decentralized manner" could
| just as easily be written about regular Git. Radicle just
| seems to add a peer-to-peer protocol on top that makes it
| less annoying to distribute a repository.
|
| So I don't get why the project has "lost you", but I also
| suspect you're the kind of person any project could
| readily afford to lose as a user.
| smashah wrote:
| Spooky day today on the internet. Huge CF outage, Gemini 3
| launches now I can't push anything to my repos.
| sgreene570 wrote:
| github has had a few of these as of late, starting to get old
| baq wrote:
| Remember talking about the exact same thing with very similar
| wording sometime pre-COVID
| cluckindan wrote:
| MSFT intentionally degrading operations to get everyone to move
| onto Azure... oh, wait, they just moved GitHub there, carry on
| my wayward son!
| blasphemers wrote:
| GitHub hasn't been moved onto azure yet, they just announced
| it's their goal to move over in 2026
| kennysmoothx wrote:
| FYI in an emergency you can edit files directly on Github without
| the need to use git.
|
| Edit: ugh... if you rely on GH Actions for workflows though
| actions/checkout@v4 is also currently experiencing the git
| issues, so no dice if you depend on that.
| ruuda wrote:
| FYI in an emergency you can `git push` to and `git pull` from
| any SSH-capable host without the need to use GitHub.
| cluckindan wrote:
| FYI in an emergency you can SSH to your server and edit files
| and the DB directly.
|
| Where is your god now, proponents of immutable filesystems?!
| BadBadJellyBean wrote:
| I love when people do that because they always say "I will
| push the fix to git later". They never do and when we
| deploy a version from git things break. Good times.
|
| I started packing things into docker containers because of
| that. Makes it a bit more of a hassle to change things in
| production.
| noir_lord wrote:
| Depends on the org, the big ones I've worked for regular
| Devs even seniors don't have anything like the level of
| access to be able to pull a stunt like that.
|
| At the largest place I _did_ have prod creds for
| everything because sometimes they are necessary and I had
| the seniority (sometimes you do need them in a "oh crap"
| scenario).
|
| They where all setup on a second account in my work Mac
| which had a danger will Robinson wallpaper because I
| _know myself_ , far far too easy to mentally fat finger
| when you have two sets of creds.
| egeozcan wrote:
| FYI in an emergency, you can buy a plane ticket and send
| someone to access the server directly.
|
| I actually had the privilege of being sent to the server.
| noir_lord wrote:
| Had a coworker have to drive across the country once to
| hit a power button (many years ago).
|
| Because my suggestion they have a spare ADSL connection
| for out of channel stuff was an unnecessary expense...
| Til he broke the firewall knocked a bunch of folks
| offline across a huge physical site and locked himself
| out of everything.
|
| The spare line got fitted the next month.
| lenerdenator wrote:
| I'm actually getting "ERROR: no healthy upstream" on `git
| pull`.
|
| They done borked it good.
| avree wrote:
| If your remote is set to a git@github.com remote, it won't
| work. They're just pointing out that you could use git to
| set origin/your remote to a different ssh capable server,
| and push/pull through that.
| rco8786 wrote:
| Yup, we were just trying to hotfix prod and ran into this. What
| is happening to the internet lately.
| vielite1310 wrote:
| True that, and this time Github AI actually have a useful
| answer to check for githubstatus.com
| lopatin wrote:
| Can you create a branch through GitHub UI?
| hobofan wrote:
| Yes. Just start editing a file and when you hit the "commit
| changes" button it will ask you what name to use for the
| branch.
| shrikant wrote:
| We're not using Github Actions, but CircleCI is also failing
| git operations on Github (it doesn't recognise our SSH keys).
| zackify wrote:
| this is actually the 5-6th time this month. actions have been
| degraded constantly now push and pull breaks back to back
| etchalon wrote:
| What is today and who do I blame for it
| baq wrote:
| Computers are great at solving problems that wouldn't have
| existed without computers
| stronglikedan wrote:
| Computers and alcohol.
| saydus wrote:
| Cherry on top will be another aws outage
| linsomniac wrote:
| Funny you should say that, I'm here looking because our
| monitoring server is seeing 80-90% packet loss on our wireguard
| from our data center to EC2 Oregon...
| linsomniac wrote:
| FYI: Not AWS. Been doing some more investigation, it looks
| like it's either at our data center, or something on the path
| to AWS, because if I fail over to our secondary firewall it
| takes a slightly different path both internally and
| externally, but the packet loss goes away.
| SimoncelloCT wrote:
| Same issue, and I need to complete my work :(
| personjerry wrote:
| Looks like Gemini 3 figured out the best way to save costs on its
| compute time was to shut down github!
| ssawchenko wrote:
| Same.
|
| ERROR: no healthy upstream fatal: Could not read from remote
| repository.
| whynotmaybe wrote:
| We gonna need xkcd "compiling" but with
| "cloudflare||github||chatgpt||spotify down".
|
| https://xkcd.com/303/
| sre2025 wrote:
| Why are there outages everywhere all the time now? AWS, Azure,
| GitHub, Cloudflare, etc. Is this the result of "vibe coding"?
| Because before "vibe coding", I don't remember having this many
| outages around the clock. Just saying.
| elicash wrote:
| I think it has more to do with layoffs.
|
| "Why do we need so many people to keep things running!?! We
| never have downtime!!"
| Refreeze5224 wrote:
| Which the true reason for AI, reducing payroll costs.
| themafia wrote:
| The reason I detest those who push AI as a technological
| solution. I think AI as a field is interesting but highly
| immature, but it's been over hyped to the point of
| absurdity, and now it is having real negative pressure on
| wages. That pressure has carry over effects and I agree
| that we're starting to observe those.
| brovonov wrote:
| Has to be a mix of both.
| noosphr wrote:
| They fired a ton of employees with no rhyme or reason to cut
| costs, this was the predictable outcome. It will get worse if
| it ever gets better.
|
| The funny thing is that the over hiring during the pandemic
| also had the predictable result of mass lay-offs.
|
| Whoever manages HR should be the ones fired after two back to
| back disasters like this.
| baq wrote:
| And yet we keep paying the company
| nawgz wrote:
| AI use being pushed, team sizes being reduced, continued lack
| of care towards quality... enshittification marches on, gaining
| speed every day
| harshalizee wrote:
| Could also be the hack and slash layoffs are starting to show
| its results. Removing crucial personnel, teams spread thin,
| combined with low morale industrywide and you've got the
| perfect recipe for disaster.
| _pdp_ wrote:
| Is it just me or it seems that there is an increased frequency of
| these types of incidents as of late.
| RGamma wrote:
| ICE keeps finding immigrants in the server cabinets.
| 0dayman wrote:
| Github is down a lot...
| laurentiurad wrote:
| A lot of failures lately during the aI ReVoLuTiOn.
| SOLAR_FIELDS wrote:
| GitHub has a long history of garbage reliability that long
| predates AI
| matkv wrote:
| Just as I was wondering why my `git push` wasn't working all of a
| sudden :D
| chrsstrm wrote:
| I thought I was going crazy when I couldn't push changes but now
| it seems it's time to just call it for the day. Back at it
| tomorrow.
| peciulevicius wrote:
| same, i've started pulling my hair out, was about to nuke my
| setup and set it up all from scratch
| keepamovin wrote:
| lol same. Hilarious when this shit goes down that we all rely
| on like running water. I'm assuming GitHub was hacked by the
| NSA because someone uploaded "the UFO files" or sth.
| Mossly wrote:
| Seeing auth succeed but push fail was an exercise in hair
| pulling.
| curioussquirrel wrote:
| Same, even started adding new ssh keys to no avail... (I was
| getting some nondescript user error first, then unhealthy
| upstream)
| chrsstrm wrote:
| Would love to see a global counter for the number of times
| 'ssh -T git@github.com' was invoked.
| mepage wrote:
| Seeing "ERROR: no healthy upstream" in push/pull operations
| _jab wrote:
| GitHub is pretty easily the most unreliable service I've used in
| the past five years. Is GitLab better in this regard? At this
| point my trust in GitHub is essentially zero - they don't deserve
| my money any longer.
| noosphr wrote:
| You can make it as reliable as you want by hosting it on prem.
| themafia wrote:
| Flashbacks to me pushing hard for GitLab self hosting a few
| months ago. The rest of the team did not feel the lift was
| worth it.
|
| I utterly hate being at the mercy of a third party with an
| after thought of a "status page" to stare at.
| jakub_g wrote:
| > as reliable as you want
|
| We self-host GitLab but the team owning it is having hard
| time scaling it. From my understanding talking to them, the
| design of gitaly makes it very hard to scale it beyond
| certain repo size and # of pushes per day (for reference: our
| repos are GBs in size, ~1M commits, hundreds of merges per
| day)
| loloquwowndueo wrote:
| Forgejo, my dudes.
| esafak wrote:
| Do we know its uptime statistics?
| ecshafer wrote:
| We self host gitlab, so its very stable. But Gitlab also kind
| of is enterprise software. It hits every feature checkbox, but
| they aren't well integrated, and they are kind of half way
| done. I don't think its as smooth of an experience as Github
| personally, or as feature rich. But Gitlab can self host your
| project repos, cicd, issues, wikis, etc. and it does it at
| least _okay_.
| input_sh wrote:
| I would argue GitLab CI/CD is _miles ahead_ of the dumpster
| fire that is GitHub Actions. Also the homepage is actually
| useful, unlike GitHub 's.
| yoyohello13 wrote:
| We've been self hosting GitLab for 5 years and it's the most
| reliable service in our organization. We haven't had a single
| outage. We use Gitlab CI and security scanning extensively.
| markbnj wrote:
| Ditto, self-hosted for over eight years at my last job. SCM
| server and 2-4 runners depending on what we needed. Very
| impressive stability and when we had to upgrade their
| "upgrade path" tooling was a huge help.
| JonChesterfield wrote:
| Couldn't log into it this morning when cloudflare was down so
| there's that.
| tapoxi wrote:
| Another GitLab self-hosting user here, we've run it on
| Kubernetes for 6 years. It's never gone down for us, maybe an
| hour of downtime yearly as we upgrade Postgres to a new
| version.
| jakub_g wrote:
| My company self-hosts GitLab. Gitaly (the git server) is a
| weekly source of incidents, it doesn't scale well (CPU/memory
| spikes which end up taking down the web interface and API).
| However we have pretty big monorepos with hundreds of daily
| committers, probably not very representative.
| cactusfrog wrote:
| There's this Gitlab incident
| https://www.youtube.com/watch?v=tLdRBsuvVKc
| tottenhm wrote:
| Frequently use both `github.com` and self-hosted Gitlab. IMHO,
| it's just... different.
|
| Self-hosted Gitlab periodically blocks access for auto-
| upgrades. Github.com upgrades are usually invisible.
|
| Github.com is periodically hit with the broad/systemic cloud-
| outage. Self-hosted Gitlab is more decentralized infra, so you
| don't have the systemic outages.
|
| With self-hosted Gitlab, you likely to have to deal with rude
| bots on your own. Github.com has an ops team that deals with
| the rude bots.
|
| I'm sure the list goes on. ( _shrug_ )
| mepage wrote:
| Seeing "ERROR: no healthy upstream" in push/pull.
| dadof4 wrote:
| Same for me, fatal: unable to access
| 'https://github.com/repository_example.git/': The requested URL
| returned error: 500
| projproj wrote:
| Obviously just speculation, but maybe don't let AI write your
| code...
|
| Microsoft CEO says up to 30% of the company's code was written by
| AI https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-
| to-3...
| dollylambda wrote:
| Maybe AI is the tech support too
| Aloisius wrote:
| Sweet. 30% of Microsoft's code isn't protected by copyright.
|
| Time to leak that.
| tauchunfall wrote:
| It's degraded availability of Git operations.
|
| The enterprise cloud in EU, US, and Australia has no issues.
|
| If you look at the incident history disruptions happen often in
| the public cloud for years already. Before AI wrote code for
| them.
| TimTheTinker wrote:
| The enterprise cloud runs on older stable versions of
| GitHub's backend/frontend code.
| smsm42 wrote:
| That sounds very bad, but I guess it depends also on which code
| it is. And whether Nadella actually knows what he's talking
| about, too.
| angrydev wrote:
| What a ridiculous comment, as if these outages didn't happen
| before LLMs became more commonplace.
| malfist wrote:
| What a ridiculous comment, as if these outages haven't been
| increasing in quantity since LLMs became more commonplace
| case0x wrote:
| I wish I could say something smart such as "People/Organisations
| should host their own git servers", but as someone who had the
| misfortune of doing that in the past I rather have a non-
| functional GitHub.
| Mossly wrote:
| I've found Gitea to be pretty rock solid, at least for a small
| team.
| gelbphoenix wrote:
| Would even recommend Forgejo (the same project Codeberg also
| uses as the base for their service)
| mkreis wrote:
| I'm curious to learn from your mistakes, can you please
| elaborate what went wrong?
| lol768 wrote:
| > We are seeing failures for some git http operations and are
| investigating
|
| It's not just HTTPS, I can't push via SSH either.
|
| I'm not convinced it's just "some" operations either; every
| single one I've tried fails.
| olivia-banks wrote:
| A friend of mine was able to get through a few minutes ago,
| apparently. Everyone else I know is still fatal'ing.
| silverwind wrote:
| It's not only http, also ssh.
| imdsm wrote:
| Cloudflare, GitHub...
| alexskr wrote:
| Mandatory break time has officially been declared. Please step
| away from your keyboard, hydrate, and pretend you were productive
| today.
| brovonov wrote:
| Good thing I already moved away from gh to a selfhosted Forgejo
| instance.
| SteveNuts wrote:
| I have a serious question, not trying to start a flame war.
|
| A. Are these major issues with cloud/SaaS tools becoming more
| common, or is it just that they get a lot more coverage now? It
| seems like we see major issues across AWS, GCP, Azure, Github,
| etc. at least monthly now and I don't remember that being the
| case in the past.
|
| B. If it's becoming more common, what are the reasons? I can
| think of a few, but I don't know the answer, so if anyone in-the-
| know has insight I'd appreciate it.
|
| Operations budget cuts/layoffs? Replacing critical
| components/workflows with AI? Just overall growing pains, where a
| service has outgrown what it was engineered for?
|
| Thanks
| kkarpkkarp wrote:
| > If it's becoming more common, what are the reasons?
|
| Someone answered this morning, while Cloudflare outage, it's AI
| vibe coding and I tend to think there is something true in
| this. At some point there might be some tiny grain of AI
| engaged which starts the avalanche ending like this.
| junon wrote:
| Been on GitHub for a long time. It _feels_ like they 're more
| often. It used to be yearly if at all that GitHub was noticably
| impacted. Now it's monthly, and recently, seemingly weekly.
| cmrdporcupine wrote:
| In the early days of GitHub (like before 2010) outages were
| extremely common.
| junon wrote:
| Not from my recollection. Not like this. BitBucket on the
| other hand had a several day outage at one point. That one
| I do recall.
| sampullman wrote:
| I remember periods of time when GitHub was down every few
| weeks, my impression is that it's become more stable over
| the years.
| bovermyer wrote:
| I agree, for what that's worth.
|
| However, this is an unexpected bell curve. I wonder if
| GitHub is seeing more frequent adversarial action lately.
| Alternatively, perhaps there is a premature reliance on new
| technology at play.
| cmrdporcupine wrote:
| I pulled my project off github and onto codeberg a couple
| months ago but this outage still screws me over because I
| have a Cargo.toml w/ git dependency into github.
|
| I was trying to do a 1.0 release today. Codeberg went
| down for "10 minutes maintenance" multiple times while I
| was running my CI actions.
|
| And then github went down.
|
| Cursed.
| netghost wrote:
| I think it was generally news when there were upages and
| the site was up. Similar with twitter for that matter.
| chadac wrote:
| I suspect that the Azure migration is influencing this one.
| Just a bunch of legacy stuff being moved around along with
| Azure not really being the most reliable on top... I can't
| imagine it's easy.
| zackify wrote:
| there has been 5 between actions and push pull issues just
| this month. it is more often
| 0x457 wrote:
| Definitely not how I remember. First, I remember seeing
| unicorn page multiple times a day some weeks. There were also
| time when webhook delivery didn't work, so circle ci users
| couldn't kick off any builds.
|
| What change is how many services GitHub can be having issues.
| xmprt wrote:
| One possibility is increased monitoring. In the past, issues
| that happened weren't reported because they went under the
| radar. Whereas now, those same issues which only impact a small
| percentage of users would still result in a status update and
| postmortem. But take this with a grain of salt because it's
| just a theory and doesn't reflect any actual data.
|
| A lot of people are pointing to AI vibe coding as the cause,
| but I think more often than not, incidents happen due to poor
| maintenance of legacy code. But I guess this may be changing
| soon as AI written code starts to become "legacy" faster than
| regular code.
| Kostic wrote:
| At least with GitHub it's hard to hide when you get "no
| healthy upstream" on a git push.
| wnevets wrote:
| > A. Are these major issues with cloud/SaaS tools becoming more
| common, or is it just that they get a lot more coverage now? It
| seems like we see major issues across AWS, GCP, Azure, Github,
| etc. at least monthly now and I don't remember that being the
| case in the past.
|
| FWIW Microsoft is convinced moving Github to Azure will fix
| these outages
| bovermyer wrote:
| Microsoft is also convinced that its works are a net benefit
| for humanity, so I would take that with a grain of salt.
| andrewstuart2 wrote:
| I think it would be pretty hard to argue against that point
| of view, at least thus far. If DOS/Windows hadn't become
| the dominant OS someone would have, and a whole generation
| of engineers cut their teeth on their parents' windows PCs.
| bovermyer wrote:
| I'm not convinced of your first point. Just because
| something seems difficult to avoid given the current
| context does not mean it was the only path available.
|
| Your second point is a little disingenuous. Yes,
| Microsoft and Windows have been wildly successful from a
| cultural adoption standpoint. But that's not the point I
| was trying to argue.
| goda90 wrote:
| What if that alternate someone had been better than
| DOS/Windows and then engineers cut their teeth on that
| instead?
| cdaringe wrote:
| There are some pretty zany alternative realities in the
| Multiverses I've visited. Xerox Parc never went under and
| developed computing as a much more accessible commodity.
| Another, Bell labs invented a whole category of analog
| computers that's supplanted our universe's digital
| computing era. There's one where IBM goes directly to
| super computers in the 80s. While undoubtedly Microsoft
| did deliver for many of us, I am a hesitant to say that
| that was the only path. Hell, Steve Jobs existed in the
| background for a long while there!
| noir_lord wrote:
| AT&T sold Unix machines (actually a rebadged Olivetti for
| the hardware) and Microsoft has Xenix when windows wasn't
| a thing.
|
| So many weird paths we could have gone down it's almost
| strange Microsoft won.
| bilegeek wrote:
| I wish things had gone differently too, but a couple of
| nitpicks:
|
| 1.) It's already a miracle Xerox PARC escaped their
| parent company's management for as long as they did.
|
| 3.) IBM was playing catch-up on the supercomputer front
| since the CDC 6400 in 1964. Arguably, they did finally
| catch up in the mid-late 80's with the 3090.
| krabizzwainch wrote:
| What's funny is that we were some bad timing away from
| IBM giving the DOS money to Gary Kildall and we'd all be
| working with CP/M derivatives!
|
| Gary was on a flight when IBM called up the Digital
| Research looking for an OS for the IBM-PC. Gary's wife,
| Dorothy, wouldn't sign an NDA without it going through
| Gary, and supposedly they never got negotiations back on
| track.
| switchbak wrote:
| DOS and Windows kept computing behind for a VERY long
| time, not sure what you're trying to argue here?
| Lammy wrote:
| Everything old is new again.
|
| https://www.zdnet.com/article/ms-moving-hotmail-to-
| win2000-s...
|
| https://jimbojones.livejournal.com/23143.html
| codethief wrote:
| From the second link:
|
| > In 2002, the amusement continued when a network security
| outfit discovered an internal document server wide open to
| the public internet in Microsoft's supposedly "private"
| network, and found, among other things, a whitepaper[0]
| written by the hotmail migration team explaining why unix
| is superior to windows.
|
| Hahaha, that whitepaper is pure gold!
|
| [0]: https://web.archive.org/web/20040401182755/http://www.
| securi...
| einsteinx2 wrote:
| The same Azure that just had a major outage this month?
| pm90 wrote:
| Github isn't in the same reliability class as the hyperscalars
| or cloudflare; its comically bad now, to the point that at a
| previous job we invested in building a readonly cache layer
| specifically to prevent github outages from bringing our system
| down.
| __MatrixMan__ wrote:
| I think it's cancer, and it's getting worse.
| smsm42 wrote:
| It certainly feels that way, though it may be an instance of
| availability bias. Not sure what's causing it - maybe extra
| load from AI bots (certainly a lot of smaller sites complain
| about it, maybe major providers feel the pain too), maybe some
| kind of general quality erosion... It's certainly something
| that is waiting for a serious research.
| myth_drannon wrote:
| Looking around, I noticed that many senior, experienced
| individuals were laid off, sometimes replaced by
| juniors/contractors without institutional knowledge or
| experience. That's especially evident in ops/support, where the
| management believes those departments should have a smaller
| budget.
| AIorNot wrote:
| well layoffs across tech probably havent helped
|
| https://techrights.org/n/2025/08/12/Microsoft_Can_Now_Stop_R...
|
| ever since Musk greenlighted firing people again.. CEOs can't
| wait to pull the trigger
| Wowfunhappy wrote:
| I'm more interested in how this and the Cloudflare outage
| occurred on the same day. Is it really just a coincidence?
| sunshine-o wrote:
| 1/ Most of the big corporations moved to big cloud providers in
| the last 5 years. Most of them started 10 years ago but it
| really accelerated in the last 5 years. So there is for sure
| more weight and complexity on cloud providers, and more impact
| when something goes wrong.
|
| 2/ Then we cannot expect big tech to stay as sharp as in the
| 2000s and 2010s.
|
| There was a time banks had all the smart people, then the telco
| had them, etc. But people get older, too comfortable, layers of
| bad incentive and politics accumulate and you just become a
| dysfunctional big mess.
| averageRoyalty wrote:
| I suspect there is more tech out there. 20 years ago we didn't
| have smartphones. 10 years ago, 20mbit on mobile was a good
| connection. Gigabit is common now, infrastructure no longer has
| the hurdles it used to, AI makes coding and design much easier,
| phones are ubiquitous and usage of them at all times (in the
| movies, out and dinner, driving) has become super normalised.
|
| I suspect (although have not researched) that global traffic is
| up, by throughput but also by session count.
|
| This contributes to a lot more awareness. Slack being down
| wasn't impactful when most tech companies didn't use Slack. An
| AWS outage was less relevant when the 10 apps (used to be
| websites) you use most didn't rely on a single AZ in AWS or you
| were on your phone less.
|
| I think as a society it just has more impact than it used to.
| grayhatter wrote:
| End of year, pre-holiday break, code/project completion for
| perf review rush.
|
| Be good to your Stability reliability engineers for the next
| few months... it's downtime season!
| tingletech wrote:
| Years ago on hackernews I saw a link about probability
| describing a statistical technique that one could use to answer
| a question about if a specific type of event was becoming more
| common or not. Maybe related to the birthday paradox? The gist
| that I remember is that sometimes a rare event will seem to be
| happening more often, when in reality there is some cognitive
| bias that makes it non-intuitive to make that decision without
| running the numbers. I think it was a blog post that went
| through a few different examples, and maybe only one of them
| was actually happening more often.
| ambicapter wrote:
| If the events are independent, you could use a binomial
| distribution. Not sure if you can consider these kinds of
| events to be independent, though.
| swed420 wrote:
| > B. If it's becoming more common, what are the reasons?
|
| Among other mentioned factors like AI and layoffs: mass brain
| damage caused by never-ending COVID re-infections.
|
| Since vaccines don't prevent transmission, and each re-
| infection increases the chances of long COVID complications,
| the only real protection right now is wearing a proper
| respirator everywhere you go, and basically nobody is doing
| that anymore.
|
| There are tons of studies to back this line of reasoning.
| dlenski wrote:
| > Are these major issues with cloud/SaaS tools becoming more
| common, or is it just that they get a lot more coverage now?
|
| I think that "more coverage" is part of it, but also "more
| centralization." More and more of the web is centralized around
| a tiny number of cloud providers, because it's just extremely
| time-intensive and cost-prohibitive for all but the largest and
| most specialized companies to run their own datacenters and
| servers.
|
| Three specific examples: Netflix and Dropbox do run their own
| datacenters and servers; Strava runs on AWS.
|
| > If it's becoming more common, what are the reasons? I can
| think of a few, but I don't know the answer, so if anyone in-
| the-know has insight I'd appreciate it.
|
| I worked at AWS from 2020-2024, and saw several of these
| outages so I _guess_ I 'm "in the know."
|
| My somewhat-cynical take is that a lot of these services have
| grown enormously in complexity, far outstripping the ability of
| their staff to understand them or maintain them:
|
| - The OG developers of most of these cloud services have moved
| on. Knowledge transfer within AWS is generally very poor,
| because it's not incentivized, and has gotten worse due to
| remote work and geographic dispersion of service teams.
|
| - Managers at AWS are heavily incentivized to develop "new
| features" and not to improve the reliability, or even security,
| of their existing offerings. (I discovered numerous security
| vulnerabilities in the very-well-known service that I worked
| for, and was regularly punished-rather-than-rewarded for trying
| to get attention and resources on this. It was a big part of
| what drove me to leave Amazon. I'm still sitting on a big pile
| of zero-day vulnerabilities in ______ and ______.)
|
| - Cloud services in most of the world are basically a 3-way
| oligopoly between AWS, Microsoft/Azure, and Google. The costs
| of switching from one provider to another are often ENORMOUS
| due to a zillion fiddly little differences and behavior quirks
| ("bugs"). It's not apparent to laypeople -- or even to me --
| that any of these providers are much more or less reliable than
| the others.
| 85392_school wrote:
| Seeing "404: Not Found" for all raw files
| keepamovin wrote:
| It's weird to think all of our data lives on physical servers
| (not "in the cloud") that are falliable and made and maintained
| by falliable humans, and could fail at any moment. So long to all
| the data! Good ol' byzantine backups.
| mandus wrote:
| Good thing git was designed as a decentralized revision control
| system, so you don't really need GitHub. It's just a nice
| convenience
| ElijahLynn wrote:
| You just lose the "hub" of connecting others and providing a
| way to collaborate with others with rich discussions.
| parliament32 wrote:
| All of those sound achievable by email, which, coincidently,
| is also decentralized.
| Aurornis wrote:
| Some of my open source work is done on mailing lists
| through e-mail
|
| It's more work and slower. I'm convinced half of the reason
| they keep it that way is because the barrier to entry is
| higher and it scares contributors away.
| awesome_dude wrote:
| Wait, email is decentralised?
|
| You mean, assuming everyone in the conversation is using
| different email providers. (ie. Not the company wide one,
| and not gmail... I think that covers 90% of all email
| accounts in the company...)
| ramon156 wrote:
| SSH also down
| blueflow wrote:
| SSH is as decentralized as git - just push to your own
| server? No problem.
| jimbokun wrote:
| Well sure but you can't get any collaborators commits that
| were only pushed to GitHub before it went down.
|
| Well you can with some effort. But there's certainly some
| inconvenience.
| gertlex wrote:
| My pushing was failing for reasons I hadn't seen before. I
| then tried my sanity check of `ssh git@github.com` (I think
| I'm supposed to throw a -t flag there, but never care to),
| and that worked.
|
| But yes ssh pushing was down, was my first clue.
|
| My work laptop had just been rebooted (it froze...) and the
| CPU was pegged by security software doing a scan (insert
| :clown: emoji), so I just wandered over to HN and learned of
| the outage at that point :)
| kragen wrote:
| SSH works fine for me. I'm using it right now. Just not to
| GitHub!
| jimbokun wrote:
| As long as you didn't go all in on GitHub Actions. Like my
| company has.
| esafak wrote:
| Then your CI host is your weak point. How many companies have
| multi-cloud or multi-region CI?
| IshKebab wrote:
| Do you think you'd get better uptime with your own solution?
| I doubt it. It would just be at a different time.
| tcoff91 wrote:
| Compared to 2025 github yeah I do think most self-hosted CI
| systems would be more available. Github goes down weekly
| lately.
| Aperocky wrote:
| Aren't they halting all work to migrate to azure? Does
| not sound like an easy thing to do and feels quite easy
| to cause unexpected problems.
| davidsainez wrote:
| Doesn't have to be an in house system, just basic
| redundancy is fine. eg a simple hook that pushes to both
| GitHub and gitlab
| nightski wrote:
| I mean yes. We've hosted internal apps that have four nines
| reliability for over a decade without much trouble. It
| depends on your scale of course, but for a small team it's
| pretty easy. I'd argue it is easier than it has ever been
| because now you have open source software that is
| containerized and trivial to spin up/maintain.
|
| The downtime we do have each year is typically also on our
| terms, not in the middle of a work day or at a critical
| moment.
| wavemode wrote:
| Uptime is much, much easier at low scale than at high
| scale.
|
| The reason for buying centralized cloud solutions is not
| uptime, it's to safe the headache of developing and
| maintaining the thing.
| tyre wrote:
| My reason for centralized cloud solutions is also uptime.
|
| Multi-AZ RDS is 100% higher availability than me managing
| something.
| wavemode wrote:
| Well, just a few weeks ago we weren't able to connect to
| RDS for several hours. That's way more downtime than we
| ever had at the company I worked for 10 years ago, where
| the DB was just running on a computer in the basement.
|
| Anecdotal, but -\\_(tsu)_/-
| jakewins wrote:
| "Your own solution" should be that CI isn't doing anything
| you can't do on developer machines. CI is a convenience
| that runs your Make or Bazel or Just or whatever you prefer
| builds, that your production systems work fine without.
|
| I've seen that work first hand to keep critical stuff
| deployable through several CI outages, and also has the
| upside of making it trivial to debug "CI issues", since
| it's trivial to run the same target locally
| prescriptivist wrote:
| It's fairly straightforward to build resilient, affordable
| and scalable pipelines with DAG orchestrators like tekton
| running in kubernetes. Tekton in particular has the benefit
| of being low level enough that it can just be plugged into
| the CI tool above it (jenkins, argo, github actions,
| whatever) and is relatively portable.
| stevage wrote:
| Curious whether you actually think this, or was it sarcasm?
| 0x457 wrote:
| It was sarcasm, but git itself is Decentralized VCS.
| Technically speaking, every git checkout is a repo of itself.
| GitHub doesn't stop me from having the entire repo history up
| to last pull, and I still can push either to the company
| backup server or my coworker directly.
|
| However, since we use github.com fore more than just a git
| hosting it is SPOF in most cases, and we treat it as a snow
| day.
| lopatin wrote:
| The issue is that GitHub is down, not that git is down.
| Conscat wrote:
| I'm on HackerNews because I can't do my job right now.
| y42 wrote:
| I work in the wrong time zone. Good night.
| __MatrixMan__ wrote:
| This escalator is temporarily stairs, sorry for the
| convenience.
| Akronymus wrote:
| Tbh, I personally don't trust a stopped escalator. Some of
| the videos of brake failures on them scared me off of ever
| going on them.
| collingreen wrote:
| You've ruined something for me. My adult side is grateful
| but the rest of me is throwing a tantrum right now. I hope
| you're happy with what you've done.
| rvnx wrote:
| I read a book about elevators accidents; don't.
| yjftsjthsd-h wrote:
| elevators accidents or escalator accidents?
| keybored wrote:
| I don't use GitHub that much. I think the thing about "oh no
| you have centralized on GitHub" point is a bit exaggerated.[1]
| But generally, thinking beyond just pushing blobs to the
| Internet, "decentralization" as in software that lets you do
| everything that is Not Internet Related locally is just a great
| thing. So I can never understand people who scoff at Git being
| decentralized just because "um, actually you end up pushing to
| the same repository".
|
| It would be great to also have the continuous build and test
| and whatever else you "need" to keep the project going as local
| alternatives as well. Of course.
|
| [1] Or maybe there is just that much downtime on GitHub now
| that it can't be shrugged off
| mysteria wrote:
| Cloudflare this morning, and now this. A bunch of work isn't
| getting done today.
|
| Maybe this will push more places towards self-hosting?
| mendyberger wrote:
| After restarting my computer, reinstalling git, almost ready to
| reinstall my os, I find out it's not even my fault
| OptionOfT wrote:
| It is insane how many failures we've been getting lately,
| especially related to actions. * jobs not being
| picked up * jobs not being able to be cancelled *
| jobs running but showing up as failed * jobs showing up
| as failed but not running * jobs showing containers as
| pushed successfully to GitHub's registry, but then we get errors
| while pulling them * ID token failures (E_FAIL) and
| timeouts.
|
| I don't know if this is related to GitHub moving to Azure, or
| because they're allowing more AI generated code to pass through
| without proper reviews, or something else, but as a paying
| customer I am not happy.
| veighnsche wrote:
| Probably because AI-generated reviews has made qa way worse.
| manbitesdog wrote:
| Same! The current self-hosted runner gets hung every so often
| lherron wrote:
| Gemini 3 = Skynet ?
| broosted wrote:
| can't do git pull or push 503 and 500 errors
| spapas82 wrote:
| Having a self hosted gitea server is a godsend in times like
| this!
| broosted wrote:
| can't do git pull or git push 503 and 500 errors
| consumer451 wrote:
| We live in a house of cards. I hope that eventually people in
| power realize this. However, their incentive structures do not
| seem to be a forcing function for that eventuality.
|
| I have been thinking about this a lot lately. What would be a
| tweak that might improve this situation?
| mistercheph wrote:
| Using p2p or self hosted, and accepting the temporary tradeoffs
| of no network effects.
| sznio wrote:
| Not exactly for this situation, but I've been thinking about
| distributed caching of web content.
|
| Even if a website is down, someone somewhere most likely has it
| cached. Why can't I read it from their cache? If I'm trying to
| reach a static image file, why do I have to get it from the
| source?
|
| I guess I want torrent DHT for the web.
| stuffn wrote:
| Centralized internet continues to show it's wonderful benefits.
|
| At least microsoft decided we all deserve a couple hour break
| from work.
| netsharc wrote:
| I remember a colleague setting up a CI/CD system (on an aaS
| obviously) depending on Docker, npm, and who knows what else... I
| thought "I wonder what % of time all those systems are actually
| up at the same time"
| chazeon wrote:
| Seem images on GitHub web also not showing
| theoldgreybeard wrote:
| can the internet work for 5 minutes, please?
| Argonaut998 wrote:
| Mercury is in retrograde
| WesolyKubeczek wrote:
| So that's how the Azure migration is going.
| pyenvmanger wrote:
| Git pull and push not working
| pyenvmanger wrote:
| Git push and pull not working. Getting a 500 response.
| bstsb wrote:
| side effect that isn't immediately obvious: all
| raw.githubusercontent.com content responds with a "404: Not
| Found" response.
|
| this has broken a few pipeline jobs for me, seems like they're
| underplaying this incident
| pm90 wrote:
| yeah something major is borked and they're unwilling to admit
| it. The status page initially claimed "https git operations are
| affected" when it was clear that ssh were too (its updated to
| reflect that now).
| treeroots wrote:
| what else is out there like github?
| kragen wrote:
| Gitlab, Forgejo, Gitea, Gogs, ... or you can just push to your
| own VPS over SSH, with or without an HTTP server. We had a good
| discussion of this last option here three weeks ago:
| https://news.ycombinator.com/item?id=45710721
| robowo wrote:
| Give https://codeberg.org/ a go
| parliament32 wrote:
| GitLab is probably the next-largest competitor. Unlike GitHub
| it's actually open source, so you can use their managed
| offering or self-host.
| olivia-banks wrote:
| This is incredibly annoying. I've been trying to fix a deployment
| action on GitHub for a the past bit, so my entire workflow for
| today has been push, wait, check... push, wait, check... et
| cetera.
| luca616 wrote:
| You should really check out (pun intended) `act`
| https://github.com/nektos/act
| kevinlajoye wrote:
| Pain
|
| My guess is that it has to do with the Cloudflare outage this
| morning.
| lenerdenator wrote:
| It would be nice if this was actually broken down bit-by-bit
| after it happened, if only for paying customers of these cloud
| services.
|
| These companies are supposed to have the top people on site
| reliability. That these things keep happening and no one really
| knows why makes me doubt them.
|
| Alternatively,
|
| The takeaway for today: clearly, Man was not meant to have
| networked, distributed computing resources.
|
| We thought we could gather our knowledge and become omniscient,
| to be as the Almighty in our faculties.
|
| The folly.
|
| The hubris.
|
| The arrogance.
| swedishuser wrote:
| Almost one hour down now. What differs this from recent AWS and
| Cloudflare issues is that this appears to be a global issue?
| angrydev wrote:
| Ton of people in the comments here wanting to blame AI for these
| outages. Either you are very new to the industry or have
| forgotten how frequently they happen. Github in particular was a
| repeat offender before the MS acquisition. us-east-1 went down
| many times before LLMs came about. Why act like this is a new
| thing?
| mrguyorama wrote:
| I'm going to awkwardly bring up that we have avoided all github
| downtime and bugs and issues by simply not using github.
|
| Our git server is hosted by Atlassian. I think we've had one
| outage in several years?
|
| Our self hosted Jenkins setup is similarly robust, we've had a
| handful of hours of "Can't build" in again, several years.
|
| We are not a company made up of rockstars. We are not especially
| competent at infrastructure. None of the dev teams have ever had
| to care about our infrastructure (occasionally we read a wiki or
| ask someone a question).
|
| You don't have to live in this broken world. It's pretty easy not
| to. We had self hosted Mercurial and jenkins before we were
| bought by the megacorp, and the megacorp's version was even
| better and more reliable.
|
| Self host. Stop pretending that ignoring complexity is somehow
| better.
| futurestef wrote:
| I wonder how much of this stuff has been caused by AI agents
| running on the infra? Claude Code is amazing for devops, until it
| kubectl deletes your ArgoCD root app
| Aeroi wrote:
| just realized my world stops, when github does.
| whinvik wrote:
| Haha I don't know if its a good test or not but I could not
| figure out why git pull was failing and Claude just went crazy
| trying so many random things.
|
| Gemini 3 Pro after 3 random things announced Github was the
| issue.
| randall wrote:
| can't go down is better than won't go down.
|
| the problem isn't with centralized internet services, the problem
| is a fundamental flaw with http and our centralized client server
| model. the solution doesn't exist. i'll build it in a few years
| if nobody else does.
| usui wrote:
| It's working again now.
| arbol wrote:
| Its back
| theideaofcoffee wrote:
| Man, I sound like a broken record, but... Love that for them.
|
| How many more outages until people start to see that farming out
| every aspect of their operations maybe, might, could have a big
| effect on their overall business? What's the breaking point?
|
| Then again, the skills to run this stuff properly are getting
| more and more rare so we'll probably see more and more big
| incidents popping up more frequently like this as time goes on.
| ashishb wrote:
| I have said this before, and I will say this again: GitHub
| stars[1] are the real lock-in for GitHub. That's why all open-
| core startups are always requesting you to "star them on GitHub".
|
| The VCs look at stars before deciding which open-core startup to
| invest in.
|
| The 4 or 5 9s of reliability simply do not matter as much.
|
| 1 - https://news.ycombinator.com/item?id=36151140
| aeldidi wrote:
| I'm becoming concerned with the rate at which major software
| systems seem to be failing as of late. For context, last year I
| only logged four outages that actually disrupted my work; this
| quarter alone I'm already on my fourth, all within the past few
| weeks. This is, of course, just an anecdote and not evidence of
| any wider trend (not to mention that I might not have even logged
| everything last year), but it was enough to nudge me into writing
| this today (helped by the fact that I suddenly had some
| downtime). Keep in mind, this isn't necessarily specific to this
| outage, just something that's been on my mind enough to warrant
| writing about it.
|
| It feels like resiliency is becoming a bit of a lost art in
| networked software. I've spent a good chunk of this year chasing
| down intermittent failures at work, and I really underestimated
| how much work goes into shrinking the "blast radius", so to
| speak, of any bug or outage. Even though we mostly run a
| monolith, we still depend on a bunch of external pieces like
| daemons, databases, Redis, S3, monitoring, and third-party
| integrations, and we generally assume that these things are
| present and working in most places. My response was to better
| document the failure conditions, and once I did, realize that
| there was many more than I initially thought. Since then we've
| done things like: move things to a VPS instead of cloud services,
| automate deployment more than we already had, greatly improve the
| test suite and docs to include these newly considered failure
| conditions, and generally cut down on moving parts. It was a ton
| of effort, but the payoff has finally shown up: fewer surprises,
| fewer distractions, a much calmer system overall. Without that
| unglamorous work, things would've only grown more fragile as
| complexity crept in. And I worry that, more broadly, we're slowly
| un-learning how to build systems that stay up even when the
| inevitable bug or failure shows up.
|
| For completeness, here are the outages that prompted this: the
| AWS us-east-1 outage in October (took down the Lightspeed R
| series API), the Azure Front Door outage (prevented Playwright
| from downloading browsers for tests), today's Cloudflare outage
| (took down Lightspeed's website, which some of our clients rely
| on), and the Github outage affecting basically everyone who uses
| it as their git host.
| HardwareLust wrote:
| It's money, of course. No one wants to pay for
| resilience/redundancy. I've launched over a dozen projects
| going back to 2008, clients simply refuse to pay for it, and
| you can't force them. They'd rather pinch their pennies, roll
| the dice and pray.
| suddenlybananas wrote:
| To be deliberately provocative, LLMs are being more and more
| widely used.
___________________________________________________________________
(page generated 2025-11-18 23:00 UTC)