[HN Gopher] Debian bookworm live images now reproducible
___________________________________________________________________
Debian bookworm live images now reproducible
Author : bertman
Score : 372 points
Date : 2025-03-26 17:22 UTC (5 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| imcritic wrote:
| I don't get how someone achieves reproducibility of builds: what
| about files metadata like creation/modification timestamps? Do
| they forge them? Or are these data treated as not important
| enough (like it 2 files with different metadata but identical
| contents should have the same checksum when hashed)?
| c0l0 wrote:
| Yes.
| HideousKojima wrote:
| Those aren't needed to generate a hash of a file. And that
| metadata isn't part of the file itself (or at least doesn't
| need to be), it's part of the filesystem or OS
| imcritic wrote:
| That's an acceptable answer for the simple case when you
| distribute just a file, but what if your distribution is
| something more complex, like an archive with some sub-
| archives? Metadata in the internal files will affect the
| checksum of the resulting archive.
| exe34 wrote:
| unless you fix them to a known epoch.
| o11c wrote:
| Timestamps are easiest part - you just set everything according
| to the chosen epoch.
|
| The hard things involve things like unstable hash orderings,
| non-sorted filesystem listing, parallel execution, address-
| space randomization, ...
| koolba wrote:
| ASLR shouldn't be an issue unless you intend to capture the
| entire memory state of the application. It's an intermediate
| representation in memory, not an output of any given step of
| a build.
|
| Annoying edge cases come up for things like internal object
| serialization to sort things like JSON keys in config files.
| sodality2 wrote:
| Let's say a compiler is doing something in a multi-threaded
| manner - isn't it possible that ASLR would affect the
| ordering of certain events which could change the compiled
| output? Sure you could just set threads to 1 but there's
| probably some more edge cases in there I haven't thought
| of.
| zamadatix wrote:
| I think you'd need the compiler to guarantee
| serialization order of such operations regardless if you
| used ASLR or not. Otherwise you're just hoping thread
| scheduling, core clocking, thread memory access, and many
| other things are the same between every system trying to
| do a reproducible build. Even setting threads to 1 may
| not solve that problem class if asynchronous
| functions/syscalls come into play.
| cperciva wrote:
| FreeBSD tripped over an issue recently where a C++ program
| (I think clang?) used a collection of pointers and output
| values in an order based on the pointers rather than the
| values they pointed to.
|
| ASLR by itself shouldn't cause reproducibility issues, but
| it can certainly _expose_ bugs.
| purkka wrote:
| Generally, yes: https://reproducible-
| builds.org/docs/timestamps/
|
| Since the build is reproducible, it should not matter when it
| was built. If you want to trace a build back to its source,
| there are much better ways than a timestamp.
| ryandrake wrote:
| C compilers offer __DATE__ and __TIME__ macros, which expand
| to string constants that describe the date and time that the
| preprocessor was invoked. Any code using these would have
| different strings each time it was built, and would need to
| be modified. I can't think of a good reason for them to be
| used in an actual production program, but for whatever
| reason, they exist.
| fmbb wrote:
| Toolchains for reproducible software likely let you set
| these values, or ensure they are 1970-01-01 00:00:00
| mikepurvis wrote:
| Nix sets everything to the epoch, although I believe
| Debian's approach is to just use the date of the newest
| file in the dsc tarballs.
| yjftsjthsd-h wrote:
| Nix _can_ also set it to things other than 0; I think my
| favorite is to set it by the time of the commit from
| which you 're building.
| mananaysiempre wrote:
| And that's why GCC (among others) accepts SOURCE_DATE_EPOCH
| from the environment, and also has -Wdate-time. As for
| using __DATE__ or __TIME__ in code, I suspect that was more
| helpful in the age before ubiquitous source control and
| build IDs.
| repiret wrote:
| > I can't think of a good reason for them
|
| I work on a product whose user interface in one place says
| something like "Copyright 2004-2025". The second year there
| is generated from __DATE__, that way nobody has to do
| anything to keep it up to date.
| Arelius wrote:
| I mean, you could do that, it's sort-of a lie though,
| maybe something better would be using the date of the
| most recent commit, which would be both more accurate, as
| far as authorship goes, and actually deterministic..
|
| Pipe something like this into your build system:
| date --date "$(git log HEAD --author-date-order
| --pretty=format:"%ad" --date=iso | head -n1)" +"%Y"
| jzb wrote:
| Debian uses a tool called `strip-nondeterminism` to help with
| this in part: https://salsa.debian.org/reproducible-
| builds/strip-nondeterm...
|
| There's lots of info on the Debian site about their
| reproducibility efforts, and there's a story from 2024's
| DebConf that may be of interest:
| https://lwn.net/Articles/985739/
| echoangle wrote:
| Maybe dumb question but why would this change the
| reproducibility? If you clone a git repo, do you not get the
| meta data as it is stored in git? Or would the files have the
| modification date of the cloning?
|
| I never actually checked that.
| mathfailure wrote:
| You clone source from git, but then you use them to build
| some artifacts. The artifacts build time may differ, yet with
| reproducible builds - the artifact should match.
| echoangle wrote:
| Right, but if you only clone and build, why would the files
| modification date be different compared to the version that
| was committed to git? Does just cloning a repo already lead
| to different file modification dates in my local copy?
| hoten wrote:
| Git does not store or restore file modification times.
| codetrotter wrote:
| And the reason for that in turn is because if you are on
| one commit and check out and older commit, then restoring
| file modification times to what they were at the time of
| the older commit would cause build tools that look at
| file modification times to sometimes not pick up on all
| the changes.
| echoangle wrote:
| Ah ok, that explains it.
| paulddraper wrote:
| > Do they forge them?
|
| Yes. All archive entries and date source code macros and any
| other timestamps are set to a standardized date (in the past).
| c0l0 wrote:
| I never really understood the hype around reproducible builds. It
| seems to mostly be a vehicle to enable tivoization[0] while
| keeping users sufficiently calm. With reproducible buiilds, a
| vendor can prove to users that they did build $binary from
| $someopensourceproject, and then digitally sign the result so
| that it - and only it - would load and execute on the vendor-
| provided and/or vendor-controlled platform. But that still kills
| effective software freedom as long as I, the user, cannot do the
| same thing with _my own build_ (whether it is unmodified or not)
| of $someopensourceproject.
|
| Therefore, I side with Tavis Ormandy on this debate:
| https://web.archive.org/web/20210616083816/https://blog.cmpx...
|
| [0]: https://en.wikipedia.org/wiki/Tivoization
| klysm wrote:
| One of the big advantages from my perspective is you can cache
| a lot more effectively throughout the build process when things
| are deterministic.
| c0l0 wrote:
| To achieve that it is enough to hash inputs, and cache
| resulting outputs. Repeating a build from scratch with an
| emtpy cache would not necessarily have to yield the same
| hashes all they way down to the last artifact, but that's
| actually a simplification of the whole process, and not a bad
| thing per se.
| mschuster91 wrote:
| > To achieve that it is enough to hash inputs, and cache
| resulting outputs.
|
| Thing is, inputs can be nondeterministic too - some
| programs (used to) embed the current git commit hash into
| the final binary so that a `./foo --version` gives a quick
| and easy way for bug triage to check if the user isn't
| using a version from years ago.
| layer8 wrote:
| This is only a problem if those nondeterministic inputs
| are actually included in the hash. This is often not the
| case, because the values are included implicitly in the
| build rather than explicitly.
|
| (Just playing devil's advocate here.)
| telotortium wrote:
| Adding the Git hash is reproducible, assuming you build
| from a clean tree (which the build script can check).
| Embedding the current date and time is the canonical
| cause of non-reproducibility, but that can be worked
| around in most cases by embedding the commit and/or
| author date of the commit instead.
| klysm wrote:
| Outputs are used as inputs later. If everything is
| deterministic, you can actually cache everything by hash
| oulipo wrote:
| Reproducible builds are important also for: - caching artefacts
| - ensuring there's no malware somewhere that's been added in
| the build process
| mjevans wrote:
| Auditors can take a copy of the source, reproducibly build it
| themselves, and thus prove that the binaries someone would
| like to run match the provided source code.
| AceJohnny2 wrote:
| > _ensuring there 's no malware somewhere that's been added
| in the build process_
|
| i.e. supply-chain safety
|
| It doesn't _entirely_ resolve Thompson 's "Trusting Trust"
| problem, but it goes a long way.
| 0cf8612b2e1e wrote:
| Is it possible for mortals to rebuild gcc from scratch? Can
| I start with some minimal, auditable compiler (tcc?) and
| build up to a modern gcc? Or would it be some byzantine
| path where I need to compile gcc v1998, then perl, then
| Python 1.8, enabling you to compile gcc v2005, which lets
| you build Python2.3, etc.
| uecker wrote:
| It is a byzantine path, also because gcc switched to C++
| at some point (for no good reason IMHO). But there is a
| project that maintains such a bootstrap path:
| https://www.gnu.org/software/mes/
| tetha wrote:
| Mh. Though, if you have deterministic builds for GCC,
| imagine how much of a problem some nerd in Northern
| Washington or Scandinavia with their own strange C build
| chain would be to inject something strange into these
| compilers into the build process.
|
| Like, you spend millions to get that one backdoor into
| the compiler. And then this guy is like "Uhm. Guys. I
| have this largely perl-based build process reproducing a
| modern GCC on a Pentium with 166 Mhz swapping RAM to disk
| because the motherboard can't hold that much memory. But
| the desk fan helps cooling. It takes about 2 months or 3
| to build, but that's fine. I start it and then I work in
| the woods. It was identical to your releases about 5
| times in the last 2 years (can't build more often), and
| now it isn't somewhere deep in the code sections. My
| arduino based floppy emulator is currently moving the
| binaries through the network"
|
| Sure, it's a cyberpunk hero-fantasy, but deterministic
| builds would make these kind of shenanigans possible.
|
| And at the end of the day, independent validation is one
| of the strongest ways to fight corruption.
| XorNot wrote:
| It is sort of like that. It's been documented:
| https://github.com/fosslinux/live-bootstrap/
|
| (This is an alternative to the Guix/Scheme thing).
| fsflover wrote:
| https://news.ycombinator.com/item?id=41368835
| __MatrixMan__ wrote:
| You can still slip malware into a reproducible build, but you
| have to do it in the open. If you do it via injecting a
| tampered-with artifact via some side channel which is specific
| to your target, they will end up with a hash that doesn't agree
| with the one that is trusted by rest of the community, and will
| have reason for suspicion.
|
| That benefit goes away if the rest of the community all have
| hashes that don't agree with each other. Then the tampered-with
| one doesn't stand out.
| inglor_cz wrote:
| It is not that different from tamper-proofing medications. It
| proves that no one added poison to whatever you are consuming,
| after that thing left its "factory".
| myrmidon wrote:
| Lets turn this around. Why would you ever want _non-
| reproducible_ builds?
|
| Every bit of nondeterminism in your binaries, even if it's just
| memory layout alone, might alter the behavior, i.e. break
| things on _some_ builds, which is just really not desirable.
|
| Why would you ever want builds from the same source to have
| potentially different performance, different output size or
| otherwise different behavior?
|
| IMO tivoization is completely unrelated, because the vendor
| most certainly does not need reproducible builds in order to
| lock down a platform.
| RJIb8RBYxzAMX9u wrote:
| > Lets turn this around. Why would you ever want non-
| reproducible builds?
|
| It's not about _wanting_ non-reproducible builds, but what am
| I _sacrificing_ to achieve reproducible builds. Debian 's
| reproducible build efforts have been going for ten years, and
| it's still not yet complete. Arguably Debian could have
| diverted ten years of engineering resources elsewhere.
| There's no end to the list of worthwhile projects to tackle,
| and clearly Debian believes that reproducible builds is high
| priority, but reasonable people can disagree on that.
|
| This not to say reproducible builds are not worth doing, just
| that depending on your project / org lifecycle and available
| resources (plus a lot of subjective judgement), you may want
| to do something else first.
| progval wrote:
| Debian didn't "divert engineering resources" to this
| project. People, some of whom happen to be Debian
| developers, decided to work on it for their own reasons. If
| the Reproducible Builds effort didn't exist, it doesn't
| mean they would have spent more time working on other areas
| of Debian. Maybe even less, because the RB effort was an
| opportunity to find and fix other bugs.
| RJIb8RBYxzAMX9u wrote:
| Yes, the system is not closed and certainly people may
| simply not contribute to Debian at all. However, my main
| point is that reasonable people disagree on the relative
| importance of RR among other things, so it's not about
| "want[ing] non-reproducible builds" even if one has
| unlimited resources, but rather wanting RR, but not at
| the expense of X, where X differs from person to person.
| ahlCVA wrote:
| For me as a developer, reproducible builds are a boon during
| debugging because I can be sure that I have reproduced the
| build environment corresponding to an artifact (which is not
| trivial, particularly for more complex things like whole OS
| image builds which are common in the embedded world, for
| example) in the real world _precisely_ when I need to
| troubleshoot something.
|
| Then I can be sure that I only make the changes I intend to do
| when building upon this state (instead of, for example,
| "fixing" something by accident because the link order of
| something changed which changed the memory layout which hides a
| bug).
| layer8 wrote:
| There is merit to some of the security arguments. However, one
| thing reproducible builds enable is to reliably identify the
| source code version from which a particular build was produced.
| If a build artifact is found to have undesirable behavior
| (whether malicious or just a genuine bug or misdesign),
| reproducible builds allow to reliably trace that behavior back
| to the source code, and then to only modify the undesired
| behavior. If, on the other hand, you can't identify the
| corresponding source code version with certainty, and therefore
| have to fix the behavior based on a possibly different version
| of the source code (or of the build environment), then you
| don't know that it doesn't additionally contain any _new_
| undesired behaviors.
| IshKebab wrote:
| Tivoisation doesn't depend on reproducible builds at all.
| Vendors don't need to mathematically prove the exact origin of
| their binaries.
| rcxdude wrote:
| It basically means that not everybody needs to build from
| source code if they want to verify that the binaries they're
| using haven't had malware injected during the build process.
| I.e. so long as enough people check that they can reproduce the
| build, and call out any case where it doesn't, everyone else
| can just use the binaries without building from source. This
| means auditing efforts can focus just on the source code, which
| is a lot more tractable (but still hard, and imperfect. But it
| means a potential attacker needs to work a lot harder, as
| opppsed to a compromise of the build servers basically giving
| them free reign without much risk of detection).
|
| It doesn't really do anything at all for tivoisation, Tivo
| managed it just fine without reproducable builds.
| pavon wrote:
| Tavis makes some good arguments, but since that post I've seen
| a couple real-world situations where reproducible builds are
| valuable.
|
| One is where the upstream software developer wants to build and
| sign their software so that users know it came from them, but
| distributors also want to be the ones to build and sign the
| software so they know what exactly it is they are distributing.
| The most public example is FDroid[1]. Reproducible builds allow
| both the software developer and the distributor to sign-off on
| a single binary, giving users addition assurance that neither
| are sneaking something in. This is similar to the last example
| that Tavis gave, but shows that it is a workable process that
| provides real security benefit to the user, not just a
| hypothetical stretch.
|
| The second is license enforcement. Companies that distribute
| (A/L)GPL software are required to distribute the exact source
| code that the binary was created from, and ability to compile
| and replace the software with a modified version (for GPLv3).
| However, a lot of companies are lazy about this and publish
| source code that doesn't include all their changes. A
| reproducible build demonstrates that the source they provided
| is what was used to create the binary. Of course, the lazy ones
| aren't going to go out of their way to create reproducible
| builds, but the more reproducible the upstream code build
| system is the fewer extraneous differences downstream builds
| should have. And it allows greater confidence in the good guys
| who are following the license.
|
| And like others have said, I don't see the Tivoization argument
| at all. TiVo didn't have reproducible builds, and they Tivo'd
| their software just fine. At worst a reproducible build might
| pacify some security minded folks that would otherwise object
| to Tivoization, but there will still be people who object to it
| out of the desire to modify the system.
|
| [1] https://f-droid.org/docs/Reproducible_Builds/
| bobmcnamara wrote:
| > This diagram demonstrates how to get a trusted binary without
| reproducible builds.
|
| Ages ago our device firmware release processes caught the early
| stage of a malware infection because the hash of one of our
| intermediate code generators (win32 exe) changed between two
| adjacent releases without any commits that should've impacted
| that tool.
|
| Turns out they had hooked something into windows to monitor for
| exe accesses and were accidentally patching out codegen.
|
| Eventually you just top trusting anything and live in the woods
| I guess.
| geocrasher wrote:
| What is the significance of a reproducible build, and how is it
| different than a normal distribution?
| bbarnett wrote:
| It means you can build it yourself, and know the source code
| you have, is all there is.
|
| It validates that publicly available downloads aren't different
| from what is claimed.
| genpfault wrote:
| https://en.wikipedia.org/wiki/Reproducible_builds
|
| https://wiki.debian.org/ReproducibleBuilds/About
| csense wrote:
| Reproducible: If Alice and Bob both download and compile the
| same source code, Alice's binary is byte-for-byte identical to
| Bob's binary.
|
| Normal: Before Debian's initiative to handle this problem, most
| people didn't think hard about all the ways system-specific
| differences might wind up in binaries. For example: __DATE__
| and __TIME__ macros in C, parallel builds finishing in
| different order, anything that produces a tar file (or zip
| etc.) usually by default asks the OS for the input files'
| modification time and puts that into the bytes of the tar file,
| filesystems may list files in a directory in different order
| and this may also get preserved in tar/zip files or other
| places...
|
| Why it's important: With reproducible builds, anyone can check
| the official binaries of Debian match the source code. This
| means going forward, any bad actors who want to sneak backdoors
| or other malware into Debian will have to find a way to put it
| in the source code, where it will be easier for people to spot.
| walrus01 wrote:
| as the 'xz' backdoor was in the source code, and remained
| there for a while before anyone spotted it, it doesn't
| necessarily guarantee that backdoors/malware won't make their
| way into the source of a very-widely-redistributed project.
| jkaplowitz wrote:
| Certainly true. But removing some attack vectors still
| helps security and trustworthiness. These are not all or
| nothing questions.
| rstuart4133 wrote:
| It's a link in a chain that allows you to trust programs you
| run.
|
| - At the start of the chain, developers write software they
| claim is secure. But very few people trust the word of just one
| developer.
|
| - Over time other developers look at the code and also
| pronounce it secure. Once enough independent developers from
| different countries and backgrounds do this, people start to
| believe it really is secure. As measure of security this isn't
| perfect, but it is verifiable and measurable in the sense more
| is always better, so if you set the bar very high you can be
| very confident.
|
| - Somebody takes that code, goes through a complex process to
| produce a binary, releases it, and pronounces it is secure
| because it is only based on code that you trust, because of the
| process above. You should not believe this. That somebody could
| have introduced malicious code and you would never know.
|
| - Therefore before reproducible builds, your only way to get a
| binary you knew was built from code you had some level of trust
| in was to build it yourself. But most people can't do that, so
| they have to trust that Debian, Google, Apple, Microsoft or
| whoever that are no backdoors have been added. Maybe people do
| place their faith in those companies, but is is misplaced. It's
| misplaced because countries like Australia have laws that allow
| them to compel such companies to silently introduce malicious
| code and distribute it to you. Australia's law is called the
| "Assistance and Access Bill (2018)". Countries don't introduce
| such laws for no reason. It's almost certain it is being used
| now.
|
| - But now the build can be reproducible. That means many
| developers can obtain the same trusted source code from the
| source the original builder claimed he used, build the binary
| themselves, verify it is identical to the original so publicly
| validate the claim. Once enough independent developers from
| different countries and backgrounds do this, people start to
| believe it really built from the trusted sources.
|
| - Ergo reproducible builds allow everyone, as opposed to just
| software developers, to run binaries they can be very confident
| was built just from code have some measurable and verifiable
| level of trustworthiness.
| zozbot234 wrote:
| Nice, these live images could become the foundation for a Debian-
| based "immutable OS" workflow.
| abdullahkhalids wrote:
| Is the build infrastructure for Debian also reproducible? It
| seems like we if someone wants to inject malware in Debian
| package binaries (without injecting them into the source), they
| have to target the build infrastructure (compilers, linkers and
| whatever wrapper code is written around them).
|
| Also, is someone else also compiling these images, so we have
| evidence that the Debian compiling servers were not compromised?
| jzb wrote:
| There's a page that includes reproducibility results for Debian
| here: https://tests.reproducible-
| builds.org/debian/bookworm/index_...
|
| I _think_ there 's also a similar thing for the images, but I
| might be wrong and I definitely don't have the link handy at
| the moment.
|
| There's lots of documentation about all of the things on
| Debian's site at the links in the brief. And LWN also had a
| story last year about Holger Levsen's talk on the topic from
| DebConf: https://lwn.net/Articles/985739/
| layer8 wrote:
| And what about the hardware on which the build runs? Is it
| reproducible? ;)
| nikisweeting wrote:
| well little johnny, when one hardware loves another hardware
| very much...
| kragen wrote:
| Working on it! But in general the answer is that for most
| purposes it's good enough to show that many independently
| produced pieces of hardware can reproduce the same results.
| ratmice wrote:
| And who trusting trusted the original RepRap?
| abdullahkhalids wrote:
| You are joking. But solving this problem is probably amongst
| the most important we can have in the information age we live
| in.
|
| Every country in the world should have the capability of
| producing "good enough" hardware.
| paulddraper wrote:
| A la xz.
|
| You must ultimately root trust in some set of binaries and any
| hardware that you use.
| XorNot wrote:
| For user space? No you can definitely do a stage 0 build
| which depends only on about 364 bytes of x86_64 binary
| (though ironically I haven't managed to get this to work for
| me yet).
|
| The liability is EFI underneath that, and the Intel ring -1
| stuff (which we should be mandating is open source).
| jcmfernandes wrote:
| Insane effort. This sounded like a pipe dream just a couple of
| years ago. Congrats to everyone involved, especially to those who
| drove the effort.
| Joel_Mckay wrote:
| The Debian group is admirable, and have positively changed the
| standards for OS design several times. Reminds me I should
| donate to their coffee fund around tax time =3
| kragen wrote:
| This is a huge milestone: https://lists.reproducible-
| builds.org/pipermail/rb-general/2...
| Cort3z wrote:
| I'm a noob to this subject. How can a build be non-reproducible?
| By that, I mean, what part of the build process could return non-
| deterministic output? Are people putting timestamps into the
| build and stuff like that?
| r3trohack3r wrote:
| File paths, timestamps, unstable ordering of inputs/outputs,
| locals, version info, variations in the build environment, etc.
|
| This pages has a good write up
|
| https://reproducible-builds.org/docs/
| jcranmer wrote:
| Timestamps, timestamps, absolute paths (i.e., differences
| between building /src versus /home/Cort3z/source), timestamps,
| file inode numbering ("for file in directory" defaults to inode
| order rather than alphabetical order in many languages, and
| that means it's effectively pseudorandom), more timestamps,
| using random data in your build process (e.g., embedding a
| generated private key, or signing something), timestamps, and
| accidental nondeterminism within the compiler.
|
| _By far_ the most prevalent source of nondeterminism is
| timestamps, especially since timestamps crop up in file formats
| you don 't expect (e.g., running gzip stuffs a timestamp in its
| output for who knows what reason). After that, it's the two big
| filesystem issues (absolute paths and directory iteration
| nondeterminism), and then it's basically a long tail of
| individual issues that affect but one or two packages.
| perdomon wrote:
| Can someone please ELI5? When I hear live images, I think of iOS
| videos that go along with pictures you take
| mjg59 wrote:
| Live images are Linux distributions that can be run directly
| from removable media instead of having to be installed to local
| storage.
| stuporglue wrote:
| A live image is an operating system image which you can boot
| from and use vs. an install disk which can only install, but
| there's no usable environment available).
|
| A reproducable build means you can get the same source code and
| compile it, and it will be identical to the published image.
| This is important because otherwise you don't know if the
| published image actually used some other source code. If it
| used some other source code, the published image might have a
| backdoor, or something that you can't find by reading the
| source code.
| perdomon wrote:
| Is that the idea behind Tails OS? It runs from removable
| media and disappears when ejected?
| yupyupyups wrote:
| This is amazing news. Well done!
| moondev wrote:
| Do these live images come ready with cloud-init? A cloud-init in-
| memory live iso seems perfect for immutable infrastructure
| "anywhere"
| selfhoster wrote:
| Used to like Debian until they started playing politics.
___________________________________________________________________
(page generated 2025-03-26 23:00 UTC)