[HN Gopher] GitHub Packages Is Down
___________________________________________________________________
GitHub Packages Is Down
Author : stonekyx
Score : 118 points
Date : 2023-03-01 13:07 UTC (9 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| lopkeny12ko wrote:
| What I find funny and unexplainable is that this class of problem
| was solved decades ago with distribution mirrors. It's not really
| clear to me why, within the last decade or so, we collectively
| decided to centralize hosting on one specific cloud service whose
| downtime now affects builds across nearly every company.
|
| What's perhaps even more surprising to me is that, after a
| repeated track history of severe and frequent Microsoft-Github
| outages in the last three years, it is _still_ a hard dependency
| for so much of the modern software stack.
| electroly wrote:
| IMO we all realized that it doesn't actually matter that much,
| most of the time. Here we are, indeed, after three years of
| severe and frequent outages! But everything is... basically
| fine? Life is full of tradeoffs.
| zmgsabst wrote:
| People who really care make their own dependency/build
| caches, eg, we had Docker containers we could fall back to.
| If you _really_ needed to patch, build on top of an existing
| artifact image -- and then rebuild when vendor service comes
| back. In practice, I just waited a few hours.
|
| Problem solved(ish).
| taeric wrote:
| Mostly agreed, but I'd hazard a guess that the scale of github
| is far larger than distribution mirrors of old.
| schainks wrote:
| There are many distribution mirrors that are financed by
| universities that are on the the Internet backbone in the US
|
| Heck, even in Asia I did not have trouble with finding a good
| mirror.
| taeric wrote:
| I suspect this is the kind of advice that works for anyone,
| but would fail for everyone. That is, for most, it is a
| valid cost/benefit tradeoff to use the central option.
| Specifically, not just for them, but for everyone. If
| everyone was following this advice, it would likely start
| hitting scale/cost problems that would make running the
| mirrors of dubious value.
| dralley wrote:
| It's a bit harder to "mirror" an active web app like a
| container registry than it is a directory of RPM or Debian
| packages + distro metadata.
| solatic wrote:
| It's not even close to the same thing. Universities hosting
| mirrors piggybacked off academic networks; not just the
| computer kind but the social kind, where professors would
| regularly meet professors from other institutions at academic
| conferences. It was in the collective interests of the
| universities to set up mirrors to solve the pre-eminent issue
| of slow WAN networks.
|
| Today most companies need private package registries. Legacy
| networks are a resource drain. Nobody else uses your private
| packages nor do you want anybody else to host a mirror and
| authentication is required anyway.
|
| Plus the idea that GitHub is hosting everything in a single
| datacenter is laughable on its face.
| dijit wrote:
| > the idea that GitHub is hosting everything in a single
| datacenter is laughable on its face.
|
| Personally I find the notion that GitHub is somehow magically
| superior to the rest of the entire internet a _bit_ silly.
|
| I've worked on distributed systems my entire career, I have
| yet to find a single one that is completely immune to a
| datacenter outage, there is always _some_ single point of
| failure not considered- often it is even known, everyone has
| the "special" datacenter.
|
| Its also true that "market forces" push for better cost
| optimisation, which can, in cases, lead to being not
| sufficiently sized to cope with an outage of a whole DC- made
| worse are people who think cloud will solve this; because
| every customer will be doing the same thing as you during a
| zonal outage.
|
| Regardless of that; you are basically suggesting that github,
| as a centralised system, is better equipped to deal with the
| distribution of packages than a literal distribution of
| package repositories?
|
| That's odd to me, maybe not "laughable", but certainly odd.
| yamtaddle wrote:
| Those take more time and money to set up, host, maintain, and
| use, than a single provider. Maybe more than it's worth to
| avoid occasional downtime.
| dpkirchner wrote:
| I think it's exactly this. We often see posts lamenting the
| lack of financial support for open source projects. How much
| more or less likely would it be for a mirror of a for-profit
| corporation's servers to receive financial support? How would
| they even reach out to potential sponsors without annoying
| users (ala donation requests in npm install output)?
| chatgptfan wrote:
| [flagged]
| dijit wrote:
| The idea of the instability of a Microsoft platform moving you
| back to.. Microsoft, is funny.
|
| Notwithstanding that services on third party systems are a
| weird reason to buck a first party platform.
| rvz wrote:
| Oh dear, Last time this happened was 11 days ago when more than
| just packages went down. [0] Perhaps relying and going all in on
| GitHub doesn't seem to be good in the long run. Especially GitHub
| Actions.
|
| This is where OpenAI is now feeling the effects of instability
| [1] on Azure since their recent outage. I expect them to also
| have issues like GitHub has every month.
|
| [0] https://news.ycombinator.com/item?id=34843748
|
| [1] https://news.ycombinator.com/item?id=34958375
| jbergknoff wrote:
| Packages had an incident two days ago, also:
| https://www.githubstatus.com/incidents/sn4m3hkqr4vz. I noticed
| it when a Terraform provider download was failing, citing a 404
| from objects.githubusercontent.com.
| VyseofArcadia wrote:
| Microsoft, historically, has not had much competitive pressure
| to be reliable. For decades, they've had a lock on the PC OS
| market, and there basically is no larg2 business that doesn't
| rely on MS. Even world governments rely on MS.
|
| Now even in the few areas where MS actually does have this
| pressure (e.g. Azure) they're struggling to make it part of the
| culture.
| aninteger wrote:
| The "is" seems to be used confusingly here. This reads better as
| either:
|
| * GitHub's Packages Service is down
|
| * GitHub Packages are down
| jasonlotito wrote:
| You are getting down voted. I'll explain.
|
| The name is GitHub Packages. It's singular. The use of "is" is
| correct here. GitHub uses "is" in similar circumstances as
| well.
| mrweasel wrote:
| I'm disappointed that this is an issue for some package
| management systems. 20 years ago I helped run a mirroring
| service, it's still running today. Distributions such as Debian
| have hundreds of mirrors. This is a solved problems, but we just
| decided to but everything in the hands of one for-profit company.
| naikrovek wrote:
| just yesterday I was stuck for an hour because a debian package
| mirror went down. took a long time to talk the user through
| changing their sources.list so that another mirror was chosen,
| and the mirror chosen out of that pool was down also. finally I
| had to manually check for a good mirror and give them the URLs.
|
| the user's take was "why don't they use GitHub packages?"
|
| "still running today" doesn't mean 100.0% uptime.
| ddtaylor wrote:
| This is not how this works anymore. The system that is
| behaving this way must be relatively old at this point since
| almost all modern Debian based distros use the "mirror://"
| URI syntax now that automatically falls back to another
| mirror if one fails.
| pxc wrote:
| I don't think a clean Debian stable install uses that
| today.
|
| But even so, at least the mirrorlist.txt file that appears
| in the mirror:// URI must be available for it to work,
| right?
| Karellen wrote:
| If only there were some way to make git distributed!
|
| /s
| crest wrote:
| You mean something like a git annex enabled branch tracking
| mirror locations of each release artifact like HTTP URLs,
| (webseeded) torrents, maybe even something content addressed
| like IPFS? _sigh_
| booi wrote:
| uhhh wait can you explain what you mean?
| Xylakant wrote:
| I can't speak for homebrew, but cargo/crate.io is fundamentally
| different from debian. The rate of churn is significantly
| higher, packages are published constantly and people expect
| them to be available. You can't really do that with a system of
| mirrors like debian does. You can do edge caching and crates
| does that, but you want some central authority of which package
| versions are available. And every cargo run queries that index.
|
| It's acceptable for debian mirrors lag a few minutes or hours
| behind. The same thing is much harder to accept when the rate
| of change is much higher. Different requirements, different
| tradeoffs.
| mrweasel wrote:
| I guess I don't really understand the need for something like
| Cargo to be up to date to the seconds, or even minutes. My
| assumption is that you build your code with certain package
| versions in mind and release that to testing. Unless it's a
| security update, if won't matter if your 24 hours behind.
|
| Say that version 1.2.1 of a library is release right as you
| do your build, that won't go into production within that 24
| hour window anyway. If it is a security fix, then, like
| Debian, you pull that from another repository, which is under
| tighter control.
| kibwen wrote:
| This thread is glossing over some important details. A
| package repo has two distinct storage concerns: the index
| (a list of which versions of packages have ever been
| released), and the actual packages themselves. It's
| convenient to have the index centralized, for maximum
| consistency. But the packages themselves can be stored
| however you want, and if you try to access a stale mirror
| without the most recent version of a package, then the
| client should have the option of using a different mirror
| or else accepting the old version.
|
| For crates.io specifically, the packages are stored in S3,
| whereas the index is currently stored as a bog-standard
| Github repo (not as a Github Package), and in the near
| future the crates.io index will also move to crates.io
| itself (https://blog.rust-lang.org/inside-
| rust/2023/01/30/cargo-spar...).
| mrweasel wrote:
| Thanks, that wasn't clear to me. Why not just dump the
| index on the same storage as the packages? If text files
| are insufficient then do an SQLite database.
| kibwen wrote:
| There are sound technical reasons to give the index
| special treatment.
|
| First, the index is very large, and it only ever gets
| larger over time. I just cloned and compressed the
| crates.io index (https://github.com/rust-lang/crates.io-
| index), which resulted in a 58 MB archive (note that I
| did remember to delete the .git directory).
|
| Second, the index changes very often. Every time anyone
| ever publishes a new version of a package, that changes
| the index. For crates.io, this happens hundreds or
| thousands of times per day.
|
| Third, the index is append-only.
|
| Fourth, the index is extremely frequently requested. Any
| time the user manually asks for an update, or any time
| the user adds a new dependency, the local copy of the
| index needs to be updated.
|
| Putting it all together, since the index is constantly
| changing and since users will constantly be asking for
| the latest version, this means that it would be very
| inefficient to serve the whole thing each time. Instead,
| a fine-grained solution is more efficient. In the early
| days of crates.io, this problem was solved by just
| storing the index in a git repo and letting git take care
| of fetching new diffs to the index (and the problem of
| "who pays for hosting" was solved by using Github). Now
| that the crates.io index is outgrowing this solution,
| it's moving to a more involved protocol where clients
| will not have local copies of the full index, but instead
| will only lazily fetch individual index entries as
| necessary, which is much faster (especially for fresh
| installs (including every CI run!)).
| kibwen wrote:
| _> cargo /crate.io is fundamentally different from debian_
|
| Given the OP, note that packages on crates.io don't (and
| can't) reference Github. Crates.io has its own storage, and
| the only way to upload a crate to crates.io is if 100% of its
| dependencies are also on crates.io.
| Animats wrote:
| Right. Although crates.io links to Github repositories, it
| doesn't get the code from them. They can be out of sync,
| which caused me some trouble yesterday.
| kibwen wrote:
| Indeed, anyone can list whatever URL they want as the
| "repository" on the crates.io page for any page they
| link. There's not much of an alternative, given that
| crates.io is designed to be immutable, and the internet
| in general is not. (At best, crates.io could provide a
| link to a browser-rendered directory tree of the code
| that crates.io has on hand for any given version.)
| BonoboIO wrote:
| Move fast and break things ... /s
| alexellisuk wrote:
| I'm also having issues pulling images from ghcr.io
| [deleted]
| bin_bash wrote:
| FYI: this breaks homebrew
| taywrobel wrote:
| Man, this disappoints me. I was a tech lead on the packages
| project about 3 years ago, specifically on the redesign for OCI
| container support and making anonymous downloads of public
| packages as reliable as possible was a top priority.
|
| If that flow was broken there's only a handful of things it
| could be; specifically azure blob store or azure MySQL, but
| both of those should have layers of redundancy. Public
| anonymous download bypasses most everything else; no auth
| services, rails monolith, or metered billing. It does emit some
| events to the message bus for metrics, but that'd effect much
| more than packages if there was an issue with it.
|
| As far as I'm aware this is the first time anonymous public
| package download broke since I left a few years back.
| hummus_bae wrote:
| [dead]
| giancarlostoro wrote:
| Does it break all of Homebrew or just some packages? I never
| knew homebrew started using GitHub Packages.
| mfer wrote:
| Homebrew uses it to store bottles (the built assets).
| jacobsenscott wrote:
| So this is why hb started asking for access to my keychain
| all the time? (macos) Does it need to log in to gh to
| install open source software now?
| naikrovek wrote:
| you don't have to, but hb will ask you to if you have a
| saved credential because your quota on GitHub for
| downloading packages is much higher if you are logged in.
|
| anonymous stuff on GitHub is usually limited to 60
| requests _per hour_ per ip address. if you 're
| authenticated, it's several hundred if not several
| thousand.
| cozzyd wrote:
| That sounds... problematic for people on big NATs? (e.g.
| universities?)
| oasisbob wrote:
| NATs are problematic already.
|
| Every office I've ever worked at has that one guy who is
| really good at tickling Google with scripts until it puts
| you all behind a CAPTCHA.
| 0xbadcafebee wrote:
| Fun fact: Linux distributions (and some older open source
| programming language package managers) use hundreds of
| mirrors distributed around the world to distribute their
| assets. If any mirror goes down, you just pick a different
| one. Even when they could just use SourceForge as a mirror
| (formerly the largest repository for open source software),
| they still used hundreds more mirrors. Distribution was
| made easy with rsync, and mirrors could choose what files
| they mirrored (just the latest release, or all releases, or
| just binaries and not source code)
| naikrovek wrote:
| GitHub uses pools of mirrors, too, they're just
| transparent.
|
| twice this year I've had to spend an hour or more with a
| user because a mirror was down. that's one more time than
| I've had to deal with a GitHub packages outage this year.
|
| the latest was yesterday. they chose another pool of
| mirrors and the mirror continually chosen from that pool
| was down as well. finally I manually checked a mirror,
| made sure it was up and that signatures matched, then
| gave them that specific hostname.
|
| the Linux package distribution system is not better. it's
| just different.
| bin_bash wrote:
| anything you don't install via `--build-from-source`
| ducktective wrote:
| Nix too I presume
| mananaysiempre wrote:
| You presume incorrectly. GitHub Packages, the package
| registry and binary hosting service, does not even support
| Nix. As for GitHub in general--
|
| The binary hosting at https://cache.nixos.org/ is independent
| of GitHub, and so are the old-style channels at
| https://channels.nixos.org/. The new-style flake registry
| used to be fetched from GitHub but has now been moved to
| https://channels.nixos.org/flake-registry.json. Admittedly in
| a new-style situation you're likely to be using unlocked
| flake references that refer to GitHub (e.g. Nixpkgs), but
| it's on you to lock them and pull them into your Nix store in
| that case.
|
| Of course, you also get GitHub references for upstreams that
| host their code there, but that applies to almost any
| distro('s build system) except the oldest of the old-timers
| which host the source for the whole distro on their own
| infrastructure, like Debian. (I happen think the old-timers
| are right here, but that's beside the point.)
| lom wrote:
| Only if you pull from master
| pxc wrote:
| Nope.
|
| This is actually the second (maybe third; I didn't even know
| about GitHub's outage a week and a half ago so idk how Nix
| was or was not affected) time in three months or less that a
| partial GitHub outage or GitHub change has taken down
| Homebrew while leaving Nix unaffected.
| josephd79 wrote:
| how so? It's working for me.
| orwellg1984 wrote:
| affirmative
| VWWHFSfQ wrote:
| I think it will break any of the open source package managers
| that rely on GitHub's proprietary hosting and distribution.
| Cargo, etc.
| kreco wrote:
| I did not know package managers relied on Github, this is the
| most unwise thing to do from a package manager perspective.
|
| Anyone could just change username/organization and break
| thousands/millions of build.
| crabbone wrote:
| Hello and welcome to PEP-508!
|
| In Python, we don't say "we don't host packages on a
| proprietary platform", we say "we have absolutely no clue
| where they are hosted and nobody audits them anyways, and
| we don't enforce package signing, and we'll just build from
| source with no build isolation what so ever, unless you
| remember to specify an obscure command-line option when
| installing... and have a nice day!"
| deckard1 wrote:
| it's amazing how we never learn. Things that Perl and
| CPAN and Linux distros figured out decades ago are
| constant issues today. It wasn't that long ago that NPM
| didn't even have checksums. CPAN runs unit tests on
| install. I can't imagine how slow that would be with npm.
|
| Package signing is, well... I suppose that's another
| lesson from the '90s people will learn about soon enough.
| With a web of trust as broad as python or npm you'll just
| have everyone running around with signing keys and
| "trusting" any key they come across because none of it is
| built on personal relationships. When Archlinux asks me
| to confirm adding package keys, what am I going to do?
| Say no? I don't know these people, but I want my shit to
| work.
| crabbone wrote:
| When it comes to my personal laptop, I also, typically,
| blindly trust the keys coming from developers because I
| don't have time for that. Not so much if I have to deploy
| a system into environment that several orders of
| magnitude more expensive than my laptop...
|
| With systems like Python, I'd imagine that a solution to
| web of trust would be that some group of developers would
| organize a curated set of packages. So, for the cases
| where you need better security assurances, you'd use
| that. I mean, of course there's no guaranteed solution
| for the web of trust, but, in practical terms, something
| like that would be good enough for regulators.
|
| There's already stuff like NumFOCUS. They don't
| particularly focus on the technical side of things, or
| endorsing more secure practices, but, in principle, they
| could. Maybe there will also be others once we have been
| bitten more times by some security breaches.
| oynqr wrote:
| The entire python package management situation can be
| summed up with "we have no clue".
| chisquared wrote:
| > Anyone could just change username/organization and break
| thousands/millions of build.
|
| GitHub redirects you to the new name in the event of a
| rename and you look up the old one.
| JamesonNetworks wrote:
| For now, hopefully
| naikrovek wrote:
| yeah. if GitHub is even _real_.
| JamesonNetworks wrote:
| Well, we know GitHub is real. But we should also remember
| that they can change their API at anytime and basing a
| package manager on their priorities is not the best
| situation for the long term success of that package
| manager, unless they are owned by Microsoft
| adders wrote:
| until someone claims the old name as a new org/repo
| naikrovek wrote:
| GitHub redirects when this happens. if you move/rename
| within GitHub, anyway. I don't know how long those
| redirects last but they probably last until the name is
| used again.
|
| so it's not quite as bad as you're imagining but still not
| great.
|
| fortunately GitHub is starting to require 2FA for very
| popular projects (starting with NPM) because of supply
| chain attacks like what you describe.
| bin_bash wrote:
| no, this is GitHub _packages_ , not GitHub repositories.
| Cargo doesn't use this. It doesn't use GitHub repositories
| either, they store the crates internally.
|
| EDIT: I was wrong, the crates _index_ does use a GitHub repo
| VWWHFSfQ wrote:
| Cargo is hard-coded to use GitHub for the crates.io index
| [1]
|
| [1] https://github.com/rust-
| lang/cargo/blob/master/src/cargo/sou...
| snacktaster wrote:
| That's a big yikes...
| inejge wrote:
| That's being addressed with a new index protocol,
| specified a while ago[1], available for testing since the
| middle of the last year[2], and slated for release in a
| week's time[3].
|
| [1] https://rust-lang.github.io/rfcs/2789-sparse-
| index.html
|
| [2] https://blog.rust-lang.org/2022/06/22/sparse-
| registry-testin...
|
| [3] https://blog.rust-lang.org/inside-
| rust/2023/01/30/cargo-spar...
| bin_bash wrote:
| I stand corrected--I was thinking of the crates
| themselves.
| ameliaquining wrote:
| The crates.io index actually _is_ a GitHub repository, so I
| think a GitHub outage that affected repositories (not just
| packages) could break Cargo. Only metadata is stored there,
| though, not the actual crates. I 'm not 100% sure why it
| works like this; there seems to be a plan to change things
| soon so that Cargo running on users' machines doesn't talk
| directly to GitHub by default (https://blog.rust-
| lang.org/inside-rust/2023/01/30/cargo-spar...), though the
| GitHub repo would still be the source of truth.
|
| crates.io also uses GitHub as an OAuth provider (and it's
| currently the only one offered), so if that broke then
| people wouldn't be able to publish crates, though
| downloading existing ones would presumably still work since
| you don't have to log in to do that.
| fulafel wrote:
| Did this include the container registry service?
| asdf123wtf wrote:
| It's never been up in the first place, for those of us who are
| waiting for python package support.
|
| Poke, poke!
___________________________________________________________________
(page generated 2023-03-01 23:02 UTC)