[HN Gopher] Could the XZ backdoor been detected with better Git/...
___________________________________________________________________
Could the XZ backdoor been detected with better Git/Deb packaging
practices?
Author : ottoke
Score : 51 points
Date : 2025-10-19 17:38 UTC (5 hours ago)
(HTM) web link (optimizedbyotto.com)
(TXT) w3m dump (optimizedbyotto.com)
| ottoke wrote:
| How did the changes in the binary test files
| tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-
| large_compressed.lzma, and the makefile change in m4/build-to-
| host.m4) manifest to the Debian maintainer? Was there a chance of
| noticing something odd?
| Groxx wrote:
| _mostly_ no, from my reading - it was a multi-stage chain of
| relatively normal looking things that added up to an exploit.
| helped by the tests involved using compressed data that wasn 't
| human-readable.
|
| you can of course come up with ways it could have been caught,
| but the code doesn't stand out as abnormal in context. that's
| all that really matters, unless your build system is already
| rigid enough to prevent it, and has no exploitable flaws you
| don't know about.
|
| finding a technical overview is annoyingly tricky, given all
| the non-technical blogspam after it, but e.g.
| https://securelist.com/xz-backdoor-story-part-1/112354/ looks
| pretty good from a skim.
| XorNot wrote:
| Compression algorithms are deterministic over fixed data
| though (possibly with some effort).
|
| There's no good reason to have opaque, non generated data in
| the repository and it should certainly be a red flag going
| forwards.
| Groxx wrote:
| committed files with carefully crafted bad data is
| extremely common for testing how your code handles invalid
| data, especially with regression tests. and lzma
| _absolutely_ needs to test itself against bad, possibly-
| malicious data.
| sanjams wrote:
| I agree, but perhaps OP is suggesting that the hand-
| crafted data can be generated in a more transparent way.
| For example, via a script/tool that itself can be
| reviewed.
| Groxx wrote:
| _could_ have, absolutely.
|
| _should not have been in any commit_ , which is
| basically necessary to prevent this case, almost
| definitely not. it's normal, and requiring all data to be
| generated just means extremely complicated generators for
| precise trigger conditions... where you can still hide
| malicious data. you just have to obfuscate it further.
| which does raise the difficulty, which is a good thing,
| but does not make it impossible.
|
| I completely agree that it's a good/best practice, but
| _hard-requiring everywhere_ it has significant costs for
| all the (overwhelmingly more common) legitimate cases.
| crote wrote:
| Yes, but the carefully crafted bad data should be
| explainable.
|
| Instead of committing blobs, why not commit documented
| code which _generates_ those blobs? For example, have a
| script compress a bunch of bytes of well-known data, then
| have it manually corrupt the bytes belonging to file size
| in the archive header.
| secondcoming wrote:
| There are tons of reasons to have hand-crafted data in a
| repository.
| sanjams wrote:
| The article references a technical write-up:
| https://research.swtch.com/xz-script
| Groxx wrote:
| ah, yes, this is one I remember seeing early on! thank you!
| I couldn't find much past the blogspam this time :/
| flerchin wrote:
| > As of today only 93% of all Debian source packages are tracked
| in git on Debian's GitLab instance at salsa.debian.org. Some key
| packages such as Coreutils and Bash are not using version control
| at all
|
| This bends my brain a little. I get that they were written before
| git, but not before the advent of version control.
| simoncion wrote:
| > I get that they were written before git, but not before the
| advent of version control. git clone
| https://git.savannah.gnu.org/git/bash.git git clone
| https://git.savannah.gnu.org/git/coreutils.git
|
| Plug the repo name into
| https://savannah.gnu.org/git/?group=<REPO_NAME> to get a link
| to browse the repo.
| kryptiskt wrote:
| Look at the commit log in the bash repo. What good does it do
| if it notionally is version controlled if the commits look
| like this: 2025-07-03 Bash-5.3 distribution
| sources and documentation bash-5.3 Chet Ramey 896
| -103357/+174007
| simoncion wrote:
| That looks to be the headline for the public release
| commit. If you'd bothered to look around for a full sixty
| seconds, you'd have found that the commits tagged with
| bash-5.3 and bash-5.2 follow that format.
|
| Here are the headlines for a couple of fix commits:
| Bash-5.2 patch 12: fixes for compat mode leaving extglob
| enabled after command substitution Bash-5.2 patch 1:
| fix crash with unset arrays in arithmetic contexts
|
| It looks like discussion of the patches happens on the
| mailing list, which is easy to access from the page that
| brought you to the repo browser.
| oivey wrote:
| Ahh yes, if only the commit message was better. That would
| have stopped the xz attack.
| rcxdude wrote:
| It's not just the commit message, but the fact that it's
| a single commit with >100k lines changed (though, if
| that's just a merge commit, it might be not super unusual
| for that kind of workflow. Though big merge commits are a
| good place to hide things in git, given that they can
| introduce their own changes)
| simoncion wrote:
| > It's not just the commit message, but the fact that
| it's a single commit with >100k lines changed
|
| When you diff the `bash-5.3` tag against the
| `bash-5.3-rc2` tag, the set of changes is reduced by a
| _ton_. It 's the same story with previous release commits
| (at least for as far as I care to go back)... there's a
| "next version" branch that gets tagged with alpha, beta,
| and rc releases, and then there's a release commit that's
| made on master with the changes from the "next version"
| branch, plus some additional changes.
|
| Why do they do things this way? I have no idea, but it
| clearly seems to work for them.
| ottoke wrote:
| This is the upstream source control. The article talks about
| the Debian packaging source not being in git (on e.g.
| salsa.debian.org).
| simoncion wrote:
| Eh, I didn't bother to read TFA. So, it was ambiguous as to
| whether OP was talking about the projects or Debian's
| packages of the same. I figured it was more likely that OP
| was talking about the projects and proceeded accordingly.
|
| If that quote's about keeping Debian packaging in source
| control, I don't really see much benefit for packages like
| coreutils and bash that generally Just Work(TM) because
| they're high-quality and well-tested. Sign what you package
| up so you can detect tampering, but I don't see you really
| _needing_ anything else.
| NewJazz wrote:
| Specifically the _packaging_ is not in version control. The
| actual software is, but the Debian maintainer for whatever
| reason doesn 't use source control for their packaging.
| goodpoint wrote:
| The author is incorrect. Keeping the packaging files under git
| is done out of convenience but it does not help for security
| and reproducibility.
|
| The packages uploaded in Debian are what matters and they are
| versioned.
| crote wrote:
| And how are you supposed to verify that the _right_ packages
| have been uploaded?
|
| The easiest way to verify that is by using a reproducible
| automated pipeline, as that moves the problem to "were the
| packaging files tampered with".
|
| How do you verify the packaging files? By making them
| auditable by putting them in a git repository, and for
| example having the packager sign each commit. If a suspicious
| commit slips in, it'll be _immediately_ obvious to anyone
| looking at the logs.
| charcircuit wrote:
| It shouldn't have happened in the first place. OpenSSH should
| control their exact dependencies and Debian shouldn't be meddling
| with them and swapping them out, loading random code into
| OpenSSH's process.
|
| >we can only trust open source software. There is no way to audit
| closed source software
|
| The ability to audit software is not sufficient, nor neccessary
| for it to be trustworthy.
|
| >systems of a closed source vendor was compromised, like
| Crowdstrike some weeks ago, we can't audit anything
|
| You can't audit open source vendors either.
| IshKebab wrote:
| That's really incidental. There are a gazillion vectors for
| exploitation once you control a package like xz. You can't fix
| this issue by plugging them one by one.
| 1718627440 wrote:
| > Debian shouldn't be meddling with them
|
| Debian is the OS, and the OS vendor should decide and modify
| the components it uses as a foundation to create the OS as he
| desires. That's what I am choosing Debian for and not some
| other OS.
|
| > You can't audit open source vendors either.
|
| What defines open source, is that you can request the sources
| for audit and modification, so I think this statement is just
| untrue.
| charcircuit wrote:
| If Debian wants to improve or modify OpenSSH and put their
| own code is, they should rename it and stop using the name of
| the project. Debian's actions created reputational damage by
| introducing a backdoor into someone else's product without
| clearly informing the consumer that they did so.
|
| >you can request the sources
|
| Organizarions that open source software can have closed
| source infrastructure that you can't request.
| 1718627440 wrote:
| Debian is famous for modifying all programs it ships, it is
| more the rule than the exception. That's the deal I get
| when choosing Debian. SSH is more of a protocol, than a
| trademarked program.
|
| > Organizarions that open source software can have closed
| source infrastructure that you can't request.
|
| Which can't be a source for the program binaries, so you
| can still audit them, you just can't rely on e.g. their
| proprietary test suite.
| toast0 wrote:
| > It shouldn't have happened in the first place. OpenSSH should
| control their exact dependencies and Debian shouldn't be
| meddling with them and swapping them out, loading random code
| into OpenSSH's process.
|
| IIRC, this dependency isn't in upstream OpenSSH.
|
| However, OpenSSH is open source with a non-restrictive license
| and as such, distributors (including Linux distributions) can
| modify it and distribute modified copies.
|
| Debian frequently modifies software it packages, often for
| better overall integration; ocassionally with negative security
| consequences. Adding something to OpenSSH to work better with
| systemd is in both categories, I guess.
| ape4 wrote:
| Wouldn't the next malware use a different way to embed itself
| xmodem wrote:
| Why would they bother if we don't act on any of the learnings
| from this one?
| jart wrote:
| Folks have been ringing the alarm bell for a decade.
| https://www.nongnu.org/lzip/xz_inadequate.html xz is insane
| because it appears to be one of the most legitimately dangerous
| compression formats with the potential to gigafry your data but
| is exclusively used by literal turbonormies who unironically want
| to like "shave off a few kilobytes" and basically get oneshotted
| by it.
| Delk wrote:
| The question of whether the xz format is a good choice for
| long-term archival is entirely unrelated to backdoors or open
| source supply chain security.
| 1970-01-01 wrote:
| >Can we trust open source software? Yes -- and I would argue that
| we can only trust open source software.
|
| But should we trust it? No!! _That 's why we're here!_
|
| I'm not satisfied with the author's double-standard-conclusion.
| Trust, but verify does not have some kind of hall pass for OSS
| "because open-source is clearly better."
|
| Trust, but verify is independent of the license the coders
| choose.
| rcxdude wrote:
| Yes, I would say that being able to view the source code and
| build it yourself is a necessary but not sufficient condition
| of properly trusting the software. (which is not quite the same
| thing as it being open source, but it's relatively rare outside
| of being a very big customer that you can do this for non-open-
| source code).
| 1718627440 wrote:
| When you get the source code as a big costumer, that is open
| source. It might even be free software.
| johnny22 wrote:
| many folks make a distinction between source available and
| open source.
| 1718627440 wrote:
| The latter meaning: accepts patches, or what?
| sega_sai wrote:
| From reading this, it seems that one thing one can do is to be
| force separation of the build from testing, so the build never
| has access to binary code that can be injected.
| acka wrote:
| I believe the XZ compromise partly stemmed from including binary
| files in what should have remained a source-only project. From
| what I remember, well-run projects such as those of the GNU
| project have always required that all binaries--whether
| executables or embedded data such as test files--be built
| directly from source, compiling a purpose-built DSL if necessary.
| This ensures transparency and reproducibility, both of which
| might have helped catch the issue earlier.
| dijit wrote:
| thats not the issue, there will always be prebuilt binaries
| (hell, deb/rpm _are_ prebuilt binaries).
|
| The issue for xz was that the build system was not hermetic
| (and sufficiently audited).
|
| Hermitic build environments that can't fetch random assets are
| a pain to maintain in this era, but are pretty crucial in
| stopping an attack of this kind. The other way is reproducible
| binaries, which is also very difficult.
|
| EDIT: Well either I responded to the wrong comment or this
| comment was entirely changed. I was replying to a comment that
| said. "The issue was that people used pre-built binaries" which
| is materially different to what the parent now says, though
| they rhyme.
| mananaysiempre wrote:
| The XZ project's build system is and was hermetic. The
| exploit was right there in the source tarball. It was just
| hidden away inside a checked-in binary file that masqueraded
| as a test for handling of invalid compressed files.
|
| (The ostensibly autotools-built files in the tarball did not
| correspond to the source repository, admittedly, but that's
| another question, and I'm of two minds about that one. I know
| that's not a popular take, but I believe Autotools has a
| point with its approach to source distributions.)
| dijit wrote:
| I thought that the exploit was not injected into the Git
| repository on GitHub at all, but only in the release
| tarballs. And that due to how Autoconf & co. work, it is
| common for tarballs of Autoconf projects to include extra
| files not in the Git repository (like the configure
| script). I thought the attacker exploited the fact that
| differences between the release tarball and the repository
| were not considered particularly suspicious by downstream
| redistributors in order to make the attack less
| discoverable.
___________________________________________________________________
(page generated 2025-10-19 23:00 UTC)