[HN Gopher] NixOS and reproducible builds could have detected th...
___________________________________________________________________
NixOS and reproducible builds could have detected the xz backdoor
Author : birdculture
Score : 79 points
Date : 2025-03-22 19:39 UTC (3 hours ago)
(HTM) web link (luj.fr)
(TXT) w3m dump (luj.fr)
| IshKebab wrote:
| Yeah it certainly would have made hiding the backdoor more
| difficult. But far from impossible. You can always hide backdoors
| in source code if you want, it just takes more effort to make a
| plausible bug, and probably has a higher chance of detection.
| ltbarcly3 wrote:
| Yes, if you use a trusted framework then you are safe from things
| until that framework is attacked. The xz backdoor might have been
| detected, but the xz backdoor wasn't crafted with the goal of
| working against the Nix ecosystem. When a nix core developer ends
| up being a spy or whatever then there will end up being an attack
| against the nix ecosystem. Don't reply to this with some claim
| that Nix is inherently secure unless you want me to track you
| down and make you admit you were wrong when Nix ends up getting
| successfully exploited in a year or two.
| vlovich123 wrote:
| The standard never has been and never will be absolute
| security. That's an impossible threshold nothing would ever
| meet even though it's objectively true that software today is
| generally more secure than software 30 years ago. The strongman
| claim being made is "Nix is harder and more expensive to
| exploit than traditional build systems". So sure, if you find a
| cheap way to exploit Nix, track me down. But until then, it
| remains at least plausible & in practice very likely that Nix
| is harder to exploit than alternate systems on a technical
| level.
| xvector wrote:
| > But until then, it remains at least plausible & in practice
| very likely that Nix is harder to exploit than alternate
| systems on a technical level.
|
| Do people even read package derivations? Feels like it'd be
| easy to check in a derivation with an exploit.
| cocoto wrote:
| The backdoor was not targeting Nix but it had to not raise any
| suspicion during a build in Nix to not be exposed.
| dzaima wrote:
| The backdoor build script specifically checked for things
| indicating that it's being built for debian, and if not, not
| inserting the backdoor; so it only ever was non-reproducible
| in situations where reproducibility wasn't expected. Not hard
| to make sure a backdoor with control over the build
| environment doesn't raise suspicions in non-targeted places.
| lolinder wrote:
| Note that NixOS and reproducible builds _did not_ detect the xz
| backdoor, and in fact NixOS shipped the malicious builds of xz
| (though they didn 't do anything because the malware didn't
| target NixOS):
|
| > I am a NixOS developer and I was surprised when the backdoor
| was revealed to see that the malicious version of xz had ended up
| being distributed to our users.
|
| As always theory and reality are different, and the thing that
| made xz possible was never a technical vulnerability with a
| technical solution--xz was possible because of a _meatspace_
| exploit. We as a community are very very bad at recognizing that
| you can 't always just patch meatspace with better software.
| jjmarr wrote:
| I still like the blogpost, because NixOS bills itself as a
| technical solution to prevent build artifacts that are
| decoupled from the source code (i.e. not reproducible), and the
| xz backdoor was hidden in build artifacts.
| lolinder wrote:
| Yeah, it's a good blog post in part _because_ it gets into
| the details of how it was possible that this vulnerability
| made it into NixOS, which purports to solve the problem.
|
| Also, I'm not a NixOS critic either: I'm writing this from
| NixOS! I just don't think there's such a thing as a security
| cure-all as long as humans are in the loop anywhere.
| crote wrote:
| Sure, but you could achieve the same thing by requiring that
| the build artifacts are generated by a Github Actions runner.
| tomjakubowski wrote:
| That would only make the xz attack harder, not make it
| impossible. Just add some step to the action run that
| fetches blobs from an internet resource you control, then
| swap out the blobs for malicious ones.
| nextos wrote:
| > NixOS and reproducible builds did not detect the xz backdoor
|
| Nix declarativeness is quite useful to increase protection
| against exploits in a number of ways. Unfortunately, there is
| still a lot of untapped potential. My number one priority would
| be to implement fine-grained ephemeral containers. Guix has
| these already.
|
| This would make it convenient to run _every_ single process
| with restricted privileges, including no access to ~ /, except
| those directories that are needed by the task. That would
| prevent e.g. a rogue pip package from stealing SSH keys.
|
| Still, I think the xz backdoor did not work on NixOS because
| its unusual non FHS-compliant filesystem structure.
| ratorx wrote:
| You don't even need to run in a container for this. It's
| possible to do this entirely in systemd service
| configuration. The easiest way is just to have separate user
| for every service and reduce stuff running as root. You can
| also restrict filesystem access, network access and even
| syscall access (although some of this may be implemented as a
| container under the hood).
|
| Unfortunately, this wouldn't help with the xz vulnerability
| because the SSH server is the one loading the compromised
| library in that case (indirectly). Since SSH itself needs to
| have access to the private keys, it's not really easy to
| secure it against vulnerabilities in the library it loads
| itself.
|
| On the flip side, unless the vulnerability is in one of the
| important binaries/shared libraries, the amount of damage it
| can cause it probably quite contained with simply having good
| user isolation. Nix can make this analysis really simple
| (because of explicitly specified dependencies), so you can
| crack down on critical dependencies a lot more easily.
| lolinder wrote:
| > I think the xz backdoor did not work on NixOS because its
| unusual non FHS-compliant filesystem structure.
|
| Right, but this is not part of the security model, it's an
| incidental attribute of the OS that's there for other reasons
| and easily solved for if the attacker had prioritized it. The
| only reason why it didn't work is because the attacker didn't
| bother making it work on NixOS, not because he couldn't have
| if he'd wanted to.
| lotharcable wrote:
| So the argument hinges on the fact that the XZ maintainer hid
| malicious code in the tarballs that were not checked into Git.
|
| The author demonstrates that Nix can be configured to generate
| the tarballs from git that go into building the binaries.
|
| What I don't see, however, is how is this a feature that requires
| Nix or NixOS?
|
| Any build system out there (including the stuff that goes into
| RPMs and Debs) can be configured to generate tarballs as a
| intermediate step.
|
| In fact making reproducible builds is a major thing that Debian
| has been working on for some time now.
|
| https://wiki.debian.org/ReproducibleBuilds
| rowanG077 wrote:
| it's somehow immensely funny to me that some state probably had
| an entire project to land this backdoor in xz, spend literal
| years to make it happen. And then it was immediately detected and
| all effort was for nothing.
| danieldk wrote:
| Or they have N other such projects in flight. The xz backdoor
| wasn't that much work, just playing the long game. The person
| doing the xz project could easily do several other projects at
| the same time.
|
| A lot of issues do get undetected. E.g. the Debian OpenSSL
| security accident was only detected when a gazillion servers
| had predictable SSH keys.
| rowanG077 wrote:
| Most likely yes. That doesn't really impact the funny factor
| for me.
| Dwedit wrote:
| Yeah, without the latency regression, it probably would have
| gone undetected much longer. Using a secondary thread and
| spreading the CPU load over a few seconds would have made it
| not even register as a spike in CPU usage.
| datadeft wrote:
| could have / should have => being smart retrospectively
| mcint wrote:
| Excellent descriptive analysis. Wrong, misleading title, perhaps
| "technically correct," but at best with a "backdoored" meaning.
|
| It points out the need and use for build-manager tools that go a
| step beyond union file system layers, but track then enforce that
| e.g. tests cannot pollute build artifacts. Take a causal trace
| graph of files affecting files, in the build process, make that
| trace graph explicit, and then build a way to enforce that graph,
| or report on deviations from previous trace graphs.
| nialv7 wrote:
| I feel the author is a bit tunnel visioned by what happens to
| happen this time. The Jiatan incident has a sample size of one,
| it'd be a bit short sighted to think that's the only way it could
| happen. You can imagine various scenarios where the defenses
| suggested here will not have worked.
|
| Also I (as a nix user myself) think it's unlikely NixOS would
| have caught it. As evidenced by the fact that it didn't. (Yeah I
| realize I just said next time it might happen differently but
| it'd be foolish to put faith in nix without evidence).
___________________________________________________________________
(page generated 2025-03-22 23:00 UTC)