[HN Gopher] Days since last GitHub incident
___________________________________________________________________
Days since last GitHub incident
Author : AquiGorka
Score : 187 points
Date : 2025-12-11 16:52 UTC (6 hours ago)
(HTM) web link (github-incidents.pages.dev)
(TXT) w3m dump (github-incidents.pages.dev)
| _def wrote:
| woah this time i even caught it before the status page reported
| something - i thought they were rate-limiting me.
| laurmaedje wrote:
| It seems to have started slowly. For me, Github releases have
| failed to serve requests for hours already.
| rienbdj wrote:
| If GitHub actions break I now assume it's them and not me.
| GitHub needs to work on stability ahead of AI features.
| udev4096 wrote:
| Friendly reminder to use https://radicle.xyz!
| GaryBluto wrote:
| _Friendly reminder_ to stop saying _friendly reminder_ when
| what you 're saying isn't a reminder.
| echelon wrote:
| For any ESL folks here -
|
| "Friendly reminder" is typically used for reminding people of
| common knowledge. Especially for beneficial but inconvenient
| things that some or most people neglect to do, either because
| they're annoying, inconvenient, or time consuming. Things for
| which busy people might need a "wink wink, nudge nudge".
|
| Friendly reminder to floss. Friendly reminder to have your
| cancer screening. Friendly reminder to check your tires.
| Friendly reminder to file your taxes early. Friendly reminder
| to drink more water, eat fiber, etc.
| loloquwowndueo wrote:
| You're assuming that no native speaker is unfamiliar with
| the idiom and also that no ESL speaker is familiar with it.
| ZeroConcerns wrote:
| This is a bit... low-effort, isn't it? I'd at least expect a
| video of an exasperated Github user walking up to the '# days
| since the last GitHub incident' board, sliding out the '1' or '2'
| card, and replacing it with a '0'.
|
| I mean, that joke is as old as the universe (heck, in the brief
| period that I worked in an office, decades ago, I had a "# days
| since the last person asked a stupid question" sign to enact the
| _exact same gag_ )...
| rob74 wrote:
| Or an octocat standing in front of the board, holding cards
| from 0 to 7 in its tentacles (with the rest lying on the
| ground) and looking at them quizzically?
| kermatt wrote:
| An octocat with zero tentacles?
| queuebert wrote:
| Base 8 numbering system?
| old_bayes wrote:
| How can we add AI to this perfectly functional product?
| venturecruelty wrote:
| Tie it to quarterly performance results!
| llbbdd wrote:
| I don't use Github Pages so I might be wrong but IMO I think at
| least part of the joke is that its URL betrays that it's a
| completely static site.
| Tade0 wrote:
| I was half-expecting the "days since last" meme - the one with
| a person smiling awkwardly while clapping with a large four-
| digit counter in the background showing only zeroes.
| venturecruelty wrote:
| I used to have a magic 8-ball that people could use when they
| wanted me to debug their code for them. I think it was broken,
| though; it kept saying "Outlook good". Must've been a Microsoft
| magic 8-ball.
| asplake wrote:
| Very low effort. Couldn't read the text on my iPhone without
| zooming in. I nearly mistook it for a blank page!
| geophph wrote:
| Is GitHub deployed using GitHub Actions?
| Oakwhisper wrote:
| I was talking with some GH sales/marketing engineers last month
| and they said it deploys with actions, but they have a custom
| deploy queue
| 0cf8612b2e1e wrote:
| It is a fun bootstrapping problem. How do you firewall enough
| dedicated resources to stand up your infrastructure if you
| dogfood your own product. Probably insidiously easy to have a
| dependency on the production service.
| shakna wrote:
| An Azure outage took out Office365 the day before
| CrowdStrike happened. I would not trust Microsoft to get
| this balance right.
| Lapalux wrote:
| Oh nice - I'm literally playing around with a site to detect
| outages for major provides (AWS/cloudflare/github) based on
| social media/HN posts
| Lapalux wrote:
| hehe - thank you for helping me Github -
| https://imgur.com/a/0KqmKpU
| llbbdd wrote:
| I've gotten accustomed lately to spending a lot of time in the
| Github Copilot / agent management page. In particular I've been
| having a lot of fun using agents to browse some of my decade-old
| throwaway projects; telling it to "setup playwright, write some
| tests, record screenshots/videos and commit them to the repo"
| works every time and it's a great way to browse memory lane
| without spending my own time getting some of these projects
| building and running again.
|
| However this means I'm now using the Github website and services
| 1000x more than I was previously, and they're trending towards
| having coin-flip uptime stats.
|
| If Github sold a $5000 box I could plug into a corner in my house
| and just use that entire experience locally I'd seriously
| consider it. I'm guessing maybe I could get partway there by
| spending twice that on a Mac Pro but I have no idea what the
| software stack would look like today.
|
| Is there a fully local equivalent out-of-the-box experience that
| anyone can vouch for? I've used local agents primarily through
| VSCode, but AFAIK that's limited to running a single active agent
| over your repo, and obviously limited by the constraints of
| running on a single M1 laptop I currently use. I know at least
| some people are managing local fleets of agents in some manner,
| but I _really_ like how immensely easy Github has made it.
| bastardoperator wrote:
| They do, it's called GHES.
|
| https://docs.github.com/en/enterprise-server@3.19/admin/over...
|
| "GitHub Enterprise Server is a self-hosted version of the
| GitHub platform"
| ModernMech wrote:
| I've tried getting this set up at my University, it was hell
| dealing with them. We ended up going with Gitlab.
| verst wrote:
| That does not include the Copilot related APIs though.
| AceJohnny2 wrote:
| you're not getting copilot on the self-hosted version, which
| is what the parent was focusing on.
| Aurornis wrote:
| None of the open weights models you can run locally will
| perform at the same level as the hosted frontier models. Some
| of them are becoming better, but the step-down in output
| quality is very noticeable for me.
|
| > If Github sold a $5000 box I could plug into a corner in my
| house and just use that entire experience locally I'd seriously
| consider it. I'm guessing maybe I could get partway there by
| spending twice that on a Mac Pro but I have no idea what the
| software stack would look like today.
|
| Right now, the only reasons to host LLMs locally are if you
| want to do it as a hobby or you are sensitive about data
| leaving your local network. If you only want a substitute for
| Copilot when GitHub is down, any of the hosted LLMs will work
| right away with no up front investment and lower overall cost.
| Most IDEs and text editors have built-in support for connecting
| to other hosted models or installing plugins for it.
|
| > I know at least some people are managing local fleets of
| agents in some manner,
|
| If your goal is to run fleets of agents in parallel, local LLM
| hosting is going to be a bottleneck. Familiarize yourself with
| some of the different tool options out their (Claude Code,
| Cline, even the new Mistral Vibe) and sign up for their cloud
| API. You can also check OpenRouter for some more options. The
| cloud hosted LLMs will absorb parallel requests without
| problem.
| llbbdd wrote:
| Thank you, a bit sad to hear that local inference isn't
| really at this level of performance yet. I was previously
| using the VSCode agent chat and playing with both OpenAI and
| Github hosted models but I switched to using the Github web
| UI directly a lot since my workflow became a lot more
| issue/PR-focused. Sounds like I should probably tighten up
| the more generic IDE-centric workflow and make it a keyboard
| shortcut to switch around when a given provider is down. I
| haven't actually used Claude directly yet but I think Github
| agents often use it under the hood anyway.
| colechristensen wrote:
| An NVIDIA DGX Spark is $4000, pair that with a relatively cheap
| second box to run GitLab in the corner and you would have
| pretty good local AI inference setup. (you'd probably have to
| write a nontrivial amount of software to get your setup where
| you want)
|
| The local models are just right on the edge of being really
| useful, there's a tipping point to where accuracy is high
| enough so that getting things done is easy vs models getting
| continuously stuck. We're in the neighborhood.
|
| Alternatively, just have local GitLab and use one of the many
| APIs, those are much more stable than github. Honestly just get
| yourself a Claude subscription.
| llbbdd wrote:
| I can't say I'm not tempted looking at the Spark, I could
| probably save some cash on heating my house with that thing.
| Though yeah unless there's some good software already built
| around a similar LLM workflow I could use it'd probably be
| wasted on me, or spend its time desperately trying to pay for
| itself with crypto mining.
|
| Adding Claude to my rotation is starting to look like the
| option with the least amount of building the universe from
| scratch. I have to imagine it can be used in a similar or
| identical workflow to the Copilot one where it can create PRs
| and make adjustments in response to feedback etc.
| colechristensen wrote:
| >Though yeah unless there's some good software already
| built around a similar LLM workflow I could use it'd
| probably be wasted on me, or spend its time desperately
| trying to pay for itself with crypto mining.
|
| A big part of my success using LLMs to build software is
| building the tools to use LLMs and the LLMs making that
| tool building easy (and _possible_ ).
| llbbdd wrote:
| I tried this for a little while and couldn't really get
| passionate about it; I have too many other backlogged
| projects that I was eager to tear into with LLMs and I
| got impatient. That was a while ago though and the ROI
| for building my own tools has probably gotten a lot more
| attractive.
| smcleod wrote:
| The DGX Spark is not good for inference though it's very
| bandwidth limited - around the same as a lower end MacBook
| Pro. You're much better off with a Apple silicon for
| performance and memory size at the moment but I'd recommend
| holding off until the M5 Max comes out early in the early as
| the M5 has vastly superior performance to any other Apple
| silicon chip thanks to its matmul instruction set.
| llbbdd wrote:
| Oof, I was already considering an upgrade from the M1 but
| was hoping I couldn't be convinced to go for the top of the
| line. Is the performance jump from the M# -> M# Max chips
| that substantial?
| doublerabbit wrote:
| I've not been able to browse any repo sources without the Unicorn
| for the past few hours.
| tonymet wrote:
| This is pretty dishonest because some trivial service no one
| cares about will reset this counter .
| guywithabike wrote:
| Your definition of "trivial" is not everyone's definition of
| trivial.
| bdcravens wrote:
| True, but the point remains that defining the whole as "down"
| when a subset is dilutes the value.
| tonymet wrote:
| github haters (who still use the platform, for free) are the
| worst
| toastal wrote:
| This is why I keep encouraging folks to a) have a mirror & b)
| make sure their tools automatically pick up the mirrors.
|
| I recently got mirror support upstreamed into Nixpkgs for
| fetchdarcs & fetchpijul which actually work on my just-alpha-
| released pinning tool, Nixtamal
| <https://darcs.toastal.in.th/nixtamal/trunk/README.rst>, for just
| this sort of thing.
| barbazoo wrote:
| I envy you. Most of us struggle to get the resources to make
| our actual customer facing applications resilient, let alone
| our build pipeline.
| toastal wrote:
| Building your software usually involves getting dependencies,
| & those dependencies are, hopefully, in more than one
| location--which includes a cronjob to a bare repo, or Alice's
| fork on another repo that at least has the latest tags. It
| should be trivial to point to these as mirrors for the cases
| where any forge/repository, even the ones held by
| megacorporations, inevitably go down. Even Nixpkgs itself,
| while not maintaining their own official mirrors, are
| mirrored by TUNA. Backups are an important strategy, & the
| source code should also be a part of that.
| maccard wrote:
| That's great for the repository, but what about if you're using
| ghcr, actions, issues, or copilot?
| toastal wrote:
| These are different concerns. There are a lot of use cases,
| where folks are just getting dependencies & not interacting
| with bug tracker or continuous integration use which are less
| critical & can be accessed later or ran locally.
| ferguess_k wrote:
| I guess none of us really needs those 9s, and even two 9s are
| just good enough. I even doubt whether *SOME* of the banking
| transactions really really really need those 9s too -- like, I
| don't really mind if 1 out of 100 credit payment doesn't go
| through so I have to do it again -- it does happen once for a
| while and I just swiped it again.
| 0xdeafbeef wrote:
| Retry is fine, but imagine being unable to pay for something
| within 10 minutes in month. And 10m in 1M is 99.98% sla. So it
| depends
| dec0dedab0de wrote:
| I had an ATM glitch out on me a few months ago, I tried again
| and it confiscated my card. I called, and they explained that
| it is the failure mode to prevent people modifying them while
| they're offline.
| tormeh wrote:
| GitHub has a container registry. That going down can cause pod
| start failure. I agree the source code probably doesn't need
| infinite nines, but the container registry is different.
| mystifyingpoi wrote:
| Which should not even be that hard, because read-only
| replicas of artifact repos are trivial to create and easy to
| loadbalance.
| cedws wrote:
| I found out when Actions started failing again for the Nth time
| this month.
|
| The internal conversation about moving away from Actions or
| possibly GitHub has been triggered. I didn't like Zig's post
| about leaving GitHub because it felt immature, but they weren't
| wrong. It's decaying.
| DetroitThrow wrote:
| Combined with security concerns, this made us reconsider even
| our self-hosted GH Actions last month.
|
| GH Packages is something we're extricating ourselves from after
| today too. One more outage in the next year and maybe we get
| the ammunition to move away from GH entirely.
|
| It's still hard to believe that they couldn't even keep the
| lights on on this thing.
| IgorPartola wrote:
| The Zig post has since been updated and the objectionable parts
| have been removed. I think we can put that part to rest.
| landr0id wrote:
| I have no problem with their opinions but I don't think it
| should have been said in a Foundation post.
|
| It may have been updated, but nobody is reading the update.
| YetAnotherNick wrote:
| Idk, if being bad is the reason for leaving Github Actions, I
| think people would have left it ages ago. It stuck not because
| it is better than competitors but because it is included in the
| Github plans. It's decaying implies that it has somehow became
| worse, in fact it was one of the worst implementation to start
| with.
| hinkley wrote:
| If you consider that an American maintainer was cheesed off
| enough to move an entire project off GitHub two days before
| Thanksgiving then the tone of the original post was completely
| in line with the energy involved.
|
| Anger is a communication tool. It should absolutely be used
| when boundaries are being violated. Otherwise you'll get walked
| all over.
| bilkow wrote:
| I mostly agree, but a generalized attack at the remaining
| GitHub workers by calling them "losers" and then "rookies" is
| unwarranted and leaves a bad taste IMO.
|
| See the edit history here:
| https://news.ycombinator.com/item?id=46133179
|
| Edit: 1. just to be clear, it's very good that they have
| accepted the feedback and removed that part, but there's no
| apology (as far as I know) and it still makes you wonder
| about the culture. On the other side, people make mistakes
| under stress. 2. /s/not warranted/unwarranted/
| stefan_ wrote:
| GitHub has seem to come under the same management as VSCode,
| everything has to be made AI and that is the only priority.
| It's like the Google+ of old but stupider.
| chrisandchris wrote:
| Hopefully with that much AI they can finally make the Explore
| page more useful than "most stars" and "most recent updated".
| There seems to be no way to discover stuff on GitHub except
| knowing where it is (hence not discovering but knowing).
| samcheng wrote:
| Missed a chance to put this in meme format e.g.
| https://imgflip.com/memetemplate/439302803/Days-without-acci...
| lawlessone wrote:
| The amazing part about this is the page even works when i'm
| offline.
| queuebert wrote:
| At this point, is there any downside to switching to GitLab?
| loloquwowndueo wrote:
| What's gitlab?
|
| (Snarky way of saying: GitHub still has huge mindshare and
| networking effects, dealing with another forge is probably too
| much friction for a lot of projects)
|
| Not that GitHub doesn't suck...
| burningChrome wrote:
| When GitHub was bought by Microsoft, Gitlab made moving your
| repos to them super easy. Apparently not enough people have
| moved and it would seem even with sustained attacks from all
| kinds of different vectors, it would seem people continue to
| stick with them.
|
| I use both Gitlab and Github and have yet to experience any
| downtime on any of my stuff. I do however, work at a large
| corporation and the latest NPM bug that hit Github caused
| enough of a stir where it basically shut down development in
| all of our lower environments for about two weeks so there's
| that.
|
| But I do agree, and it seems like their market share
| increased after the Microsoft acquisition which is contrary
| to what I heard in all my dev circles because of how uncool
| MSFT is to many of my friends.
| richardwhiuk wrote:
| Is the uptime any better?
| burningChrome wrote:
| Not really:
|
| GitHub - Historically, GitHub reports uptime around 99.95% or
| higher, which translates to roughly 20-25 minutes of downtime
| per month. They have a large infrastructure and redundancy,
| so outages are rare but can happen during major incidents.
|
| GitLab - GitLab also targets 99.95% uptime for its SaaS
| offering (GitLab.com). However, GitLab has had slightly more
| frequent service disruptions compared to GitHub in the past,
| especially during scaling events or major upgrades. For self-
| hosted GitLab instances, uptime depends heavily on your own
| infrastructure.
| bdcravens wrote:
| If escaping downtime is your goal, then you should aim for a
| service with less downtime than Github. (they're roughly the
| same, with Gitlab having a slightly higher percentage of
| "major" outages)
| JackSlateur wrote:
| Is it any better ?
|
| We had that last year, with the full premium stuff ("pay as
| much as we can" mindset)
|
| Please see this: a basic feature, much needed by lots of people
| (those who are stuck on azure ..): https://gitlab.com/gitlab-
| org/gitlab/-/issues/360592
|
| Please read the entire thread with a particular attention to
| the timeline
| matheusmoreira wrote:
| I've been getting some weird cryptocurrency spam notifications on
| GitHub and they can't be cleared for some reason. Blue dot is
| gonna be there forever apparently. Some users made an issue out
| of it but nobody cared to fix it.
| fastball wrote:
| Same happened to me. You can clear it via the CLI, hilariously.
|
| ```
|
| gh api notifications\?all=true | jq -r 'map(select(.unread) |
| .id)[]' | xargs -L1 sh -c 'gh api -X PATCH
| notifications/threads/$0'
|
| ```
| bdcravens wrote:
| HN doesn't support markdown, but you can "code" format it
| with 2+ spaces.
|
| https://news.ycombinator.com/formatdoc
| dennis-tra wrote:
| Had the same issue that the blue dot won't disappear. I was
| able to clear the dot with:
|
| gh api notifications -X PUT -F
| last_read_at=2025-10-06T00:00:00Z
|
| Just change the date to today. I also got that line from a gh
| issue somewhere - maybe it was the same issue that you're
| referring to.
| cassidoo wrote:
| This should have been fixed here:
| https://github.blog/changelog/2025-12-04-notifications-trigg...
|
| Are you still seeing it, would you mind checking? Our team will
| get on it if so.
| shakna wrote:
| I had the same. The dot is cleared on the mobile site. It is
| present on the desktop site.
| matheusmoreira wrote:
| Somewhat similar situation here. Cleared cache and logged
| back in on mobile, dot was fixed. Haven't tested on my
| laptop yet.
| cassidoo wrote:
| Thank you all, sharing internally now to get that fixed!
| Super appreciate the feedback.
| matheusmoreira wrote:
| Thanks for fixing it!
| ashton314 wrote:
| Here's how you can clear it:
|
| https://github.com/orgs/community/discussions/174310#discuss...
|
| I had the same issue too, and this was the only thing that
| fixed it for me.
| Aperocky wrote:
| Once GitHub fully migrates to Azure, it should be known as GitHub
| 11.
| ragall wrote:
| Github 3.11 for Workgroups.
| aranw wrote:
| GitHub Vista Cloud Edition
| TuxPowered wrote:
| It should always be at 0, because GitHub is unreachable over
| IPv6, which in 2025 should be considered an incident.
| tonymet wrote:
| mobile adoption is high, desktop (residential and corporate) is
| still quite low.
|
| I'm a big advocate for github to add ipv6 support , but let's
| not pretend it's critical for their business.
| fragmede wrote:
| "yeah but when I turn on ipv6 everything breaks"
| locusofself wrote:
| The Primagen video about the bash scripts underpinning github
| actions runner was crazy. I'm a half-assed programmer at best and
| I don't even think I would make some of those mistakes.
___________________________________________________________________
(page generated 2025-12-11 23:01 UTC)