[HN Gopher] Git archive generation meets Hyrum's law
___________________________________________________________________
Git archive generation meets Hyrum's law
Author : JamesCoyne
Score : 47 points
Date : 2023-02-02 18:47 UTC (4 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| avgcorrection wrote:
| Mob engineering: you don't have to read the documentation if a
| million other people also do not.
| ajross wrote:
| I think that's uncharitable. Almost no one realized these
| things were being generated. We all assumed that links to
| github's "releases" were just links to files because they
| _look_ like links to files! Here 's one to Zephyr 3.2.0:
| https://github.com/zephyrproject-rtos/zephyr/archive/refs/ta...
|
| You pull that and get a tarball that is presented to the world
| as an "official release". Looks like a file. Acts like a file.
| It's a file.
|
| So now your package manager or reproducible build engine or
| whatever needs a reference to the "official source code
| release", and what do you point it to? That file, obviously.
| It's right there on the "release" page for the download. And of
| course you checksum it for security, because duh.
|
| Then last week all of a sudden that file changed! Sure, it has
| the same contents. But the checksum that you computed in good
| faith based on the official release tarball doesn't match!
|
| If there's a misunderstanding here, it's on github and not the
| users. They can't be providing official release tarballs if
| they won't guarantee consistency. "As documented", this feature
| was a huge footgun. That's bad.
| jmclnx wrote:
| > more easily support compression across operating systems
|
| I cannot help but wonder if this change was forced upon github by
| Microsoft because gzip is GPL 3, maybe this other version is a
| clean room clone. We all know corporations hate GPLv3, including
| the large corporation I work for.
|
| https://www.gnu.org/software/gzip/
| fukawi2 wrote:
| It was git that implemented the change, then github upgraded to
| the affected version. AFAIK, MS has no influence over upstream
| git.
|
| As much as I distrust Microsoft, I don't think there were any
| ulterior motives here.
| eklitzke wrote:
| First of all the change was made upstream in git, which is not
| controlled by GitHub (even though GitHub does have some
| developers who work on git). And the stated reason (not relying
| on third party tools/libraries) is compatible with many other
| changes made to git over its history, e.g. the conversion of
| many git commands from Perl to C.
|
| Furthermore, gzip isn't even necessarily the best tool to
| produce gzip data. If you want multi-core parallelism there's
| pigz, and if you're willing to trade higher CPU usage to get a
| better compression ratio you can use zopfli. I don't know the
| details of the implementation in git and whether it tries to
| leverage multi-threading or zopfli-like techniques, but the
| point stands that gzip isn't the final word on producing gzip
| data.
| vore wrote:
| If this were true, this would have been a problem a long time
| ago. Why would Microsoft wait such a long time to change this
| when under your assumption it would have been a continuous
| legal liability?
| travisgriggs wrote:
| Had to follow the links to figure out what Hyrum's Law was (I
| like laws). The best link from that law is the obligatory xkcd at
| the very bottom. Reshared here:
|
| https://xkcd.com/1172/
| [deleted]
| ElijahLynn wrote:
| I found a good video about this too
| https://lwn.net/SubscriberLink/921787/949cf79f2599f734/
| (Original Post) --> https://www.hyrumslaw.com/ -->
| https://twitter.com/hyrumwright -->
| https://twitter.com/dret/status/1573897062785032192 -->
| https://www.youtube.com/watch?v=5Wdgjw6IGDM (Hyrum's Law: Hyrum
| Wright on Programming over Time - Interview of by Erik Wilde)
| pcj-github wrote:
| If it can't be made stable, `git archive` should specifically add
| random content (under a feature flag to be removed after a year
| or two) to so as to make the generated checksum completely
| unreliable and force users to adopt different workflows.
| syntheticnature wrote:
| 2018 Gentoo-dev called, wants to let you know this is old news:
| https://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg...
| bentley wrote:
| Indeed. The proper thing (also read as: the friendliest way for
| distro packagers) is for software projects to generate and
| publish a tarball themselves as part of their tag+release
| process.
|
| That provides multiple advantages. Unlike GitHub's unreliable
| automatically generated files, a fixed file can be hashed or
| cryptographically signed by the project (with SSH signatures,
| Signify, PGP, etc.), and later verified without having to
| extract the files first or check out the underlying repo.
|
| Another thing many projects aren't aware of: if your project
| uses Git submodules, anyone using GitHub's autogenerated
| tarballs will be unable to build your software, because those
| don't contain submodules.
| elesiuta wrote:
| > Indeed. The proper thing (also read as: the friendliest way
| for distro packagers) is for software projects to generate
| and publish a tarball themselves as part of their tag+release
| process.
|
| And this is easy enough to do automatically with GitHub
| actions, I have a workflow [1] which runs on each release to
| create a stable archive of the source and attaches it to the
| release.
|
| [1] https://github.com/elesiuta/picosnitch/blob/master/.githu
| b/w...
| GauntletWizard wrote:
| The proper thing is for the software build processes that
| rely on tarballs from GitHub to switch to using git directly;
| either by shallow clone or storing a full repo and checking
| out worktrees as appropriate. Tarballs at a tagged revision
| are fine as release artifacts if your upstream is publishing
| them as release artifacts, but the whole point of this is
| that they aren't.
| [deleted]
| skywal_l wrote:
| Everybody who had to maintain an API knows this.
|
| 1. You can't just rely on documentation ("we never said we would
| guarantee this or that") to push back on your users' claims that
| you introduced a breaking change. If you care more about your
| documentation than your users, they will turn their back on you.
|
| 2. However if you start guaranteeing too much stability,
| innovation and change will become too costly or even impossible.
| In this instance, if the git team has to guarantee their hashes
| (which seems impossible anyway because it depends on the external
| gzip program) then they can never improve on compression.
|
| Tough situation to be in.
| ablob wrote:
| Someone once stated that every observable behaviour will be
| depended upon by someone sooner or later.
|
| I can only imagine someone going to great lengths to avoid such
| "a stable order of operations was never guaranteed" discussion
| by just randomizing the order of execution or something similar
| (I bet someone will then use that as a seed for prng).
|
| edit: skipping the first paragraph lead to repeating hyrums
| law.
| avgcorrection wrote:
| > Someone once stated that every observable behaviour will be
| depended upon by someone sooner or later.
|
| ......... Hyrum's law?
| ablob wrote:
| Yeah, I only noticed when it was too late. I was drawn to
| the first quote, rather than the block of text next to the
| author.
___________________________________________________________________
(page generated 2023-02-02 23:01 UTC)