https://blog.josefsson.org/2024/07/10/towards-idempotent-rebuilds/ Skip to content Simon Josefsson's blog Search for: [ ] Search Menu Primary menu * Home * About Towards Idempotent Rebuilds? Posted on 2024-07-10 by simon -- 2 Comments | After rebuilding all added/modified packages in Trisquel, I have been circling around the elephant in the room: 99% of the binary packages in Trisquel comes from Ubuntu, which to a large extent are built from Debian source packages. Is it possible to rebuild the official binary packages identically? Does anyone make an effort to do so? Does anyone care about going through the differences between the official package and a rebuilt version? Reproducible-build.org's effort to track reproducibility bugs in Debian (and other systems) is amazing. However as far as I know, they do not confirm or deny that their rebuilds match the official packages. In fact, typically their rebuilds do not match the official packages, even when they say the package is reproducible, which had me surprised at first. To understand why that happens, compare the buildinfo file for the official coreutils 9.1-1 from Debian bookworm with the buildinfo file for reproducible-build.org's build and you will see that the SHA256 checksum does not match, but still they declare it as a reproducible package. As far as I can tell of the situation, the purpose of their rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match. I have felt that something is lacking, and months have passed and I haven't found any project that address the problem I am interested in. During my earlier work I created a project called debdistreproduce which performs rebuilds of the difference between two distributions in a GitLab pipeline, and display diffoscope output for further analysis. A couple of days ago I had the idea of rewriting it to perform rebuilds of a single distribution. A new project debdistrebuild was born and today I'm happy to bless it as version 1.0 and to announces the project! Debdistrebuild has rebuilt the top-50 popcon packages from Debian bullseye, bookworm and trixie, on amd64 and arm64, as well as Ubuntu jammy and noble on amd64, see the summary status page for links. This is intended as a proof of concept, to allow people experiment with the concept of doing GitLab-based package rebuilds and analysis. Compare how Guix has the guix challenge command. Or I should say debdistrebuild has attempted to rebuild those distributions. The number of identically built packages are fairly low, so I didn't want to waste resources building the rest of the archive until I understand if the differences are due to consequences of my build environment (plain apt-get build-dep followed by dpkg-buildpackage in a fresh container), or due to some real difference. Summarizing the results, debdistrebuild is able to rebuild 34% of Debian bullseye on amd64, 36% of bookworm on amd64, 32% of bookworm on arm64. The results for trixie and Ubuntu are disappointing, below 10%. So what causes my rebuilds to be different from the official rebuilds? Some are trivial like the classical problem of varying build paths, resulting in a different NT_GNU_BUILD_ID causing a mismatch. Some are a bit strange, like a subtle difference in one of perl's headers file. Some are due to embedded version numbers from a build dependency. Several of the build logs and diffoscope outputs doesn't make sense, likely due to bugs in my build scripts, especially for Ubuntu which appears to strip translations and do other build variations that I don't do. In general, the classes of reproducibility problems are the expected. Some are assembler differences for GnuPG's gpgv-static, likely triggered by upload of a new version of gcc after the original package was built. There are at least two ways to resolve that problem: either use the same version of build dependencies that were used to produce the original build, or demand that all packages that are affected by a change in another package are rebuilt centrally until there are no more differences. The current design of debdistrebuild uses the latest version of a build dependency that is available in the distribution. We call this a "idempotent rebuild". This is usually not how the binary packages were built originally, they are often built against earlier versions of their build dependency. That is the situation for most binary distributions. Instead of using the latest build dependency version, higher reproducability may be achieved by rebuilding using the same version of the build dependencies that were used during the original build. This requires parsing buildinfo files to find the right version of the build dependency to install. We believe doing so will lead to a higher number of reproducibly built packages. However it begs the question: can we rebuild that earlier version of the build dependency? This circles back to really old versions and bootstrappable builds eventually. While rebuilding old versions would be interesting on its own, we believe that is less helpful for trusting the latest version and improving a binary distribution: it is challenging to publish a new version of some old package that would fix a reproducibility bug in another package when used as a build dependency, and then rebuild the later packages with the modified earlier version. Those earlier packages were already published, and are part of history. It may be that ultimately it will no longer be possible to rebuild some package, because proper source code is missing (for packages using build dependencies that were never part of a release); hardware to build a package could be missing; or that the source code is no longer publicly distributable. I argue that getting to 100% idempotent rebuilds is an interesting goal on its own, and to reach it we need to start measure idempotent rebuild status. One could conceivable imagine a way to rebuild modified versions of earlier packages, and then rebuild later packages using the modified earlier packages as build dependencies, for the purpose of achieving higher level of reproducible rebuilds of the last version, and to reach for bootstrappability. However, it may be still be that this is insufficient to achieve idempotent rebuilds of the last versions. Idempotent rebuilds are different from a reproducible build (where we try to reproduce the build using the same inputs), and also to bootstrappable builds (in which all binaries are ultimately built from source code). Consider a cycle where package X influence the content of package Y, which in turn influence the content of package X. These cycles may involve several packages, and it is conceivable that a cycle could be circular and infinite. It may be difficult to identify these chains, and even more difficult to break them up, but this effort help identify where to start looking for them. Rebuilding packages using the same build dependency versions as were used during the original build, or rebuilding packages using a bootsrappable build process, both seem orthogonal to the idempotent rebuild problem. Our notion of rebuildability appears thus to be complementary to reproducible-builds.org's definition and bootstrappable.org's definition. Each to their own devices, and Happy Hacking! This entry was posted in general and tagged bootstrappable, debian, gitlab, gnu, guix, pureos, reproducible, supply-chain, trisquel, ubuntu by simon. Bookmark the permalink. 2 Replies to "Towards Idempotent Rebuilds?" 1. [9b96419e]ntyni on 2024-07-10 at 10:46 said: Thanks for looking at this! > Some are a bit strange, like a subtle difference in one of perl's headers file. Not that strange, it's a result of linux_5.10.209-1 in bullseye adding DECLARE_FLEX_ARRAY() into stddef.h after the current perl packages in bullseye were built. https://salsa.debian.org/kernel-team/linux/-/commit/ dcb6b3af721aeabdfd0fb82c22e26e04b45f4351 Reply | + [f49a]simon on 2024-07-10 at 11:24 said: Thanks for reading and explaining! Indeed, so this may better be described as an(other) example of how a change in one package (linux) leads to a change in the content of another binary package (perl). /Simon Reply | Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name *[ ] Email *[ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Post navigation - Previous Previous post: Reproducible and minimal source-only tarballs Primary Sidebar Widget Area Tags android (7) crypto (4) debian (41) devuan (4) ed25519 (6) fsdg (4) fsf (5) git (5) gitlab (6) gnome (5) gnu (29) gnuk (8) gnupg (17) gnutls (4) gsasl (5) guix (7) i9300 (4) ietf (10) key (4) keyring (3) laptop (5) lenovo (4) linux (6) neo (4) nv41pz (3) openpgp (20) openssh (4) openwrt (6) pgp (5) pureos (10) replicant (7) reproducible (5) rsa (5) ryf (4) s3 (5) sasl (8) security (17) sigstore (5) smartcard (6) smartcards (4) ssh (4) sysadmin (3) trisquel (19) ubuntu (10) yubikey (6) Recent Posts * Towards Idempotent Rebuilds? 2024-07-10 * Reproducible and minimal source-only tarballs 2024-04-13 * Towards reproducible minimal source code tarballs? On *-src.tar.gz 2024-04-01 * Apt archive mirrors in Git-LFS 2024-03-18 * Trisquel on arm64: Ampere Altra 2024-01-10 * Validating debian/copyright: licenserecon 2023-12-29 * Classic McEliece goes to IETF and OpenSSH 2023-12-10 * Trisquel on ppc64el: Talos II 2023-09-01 * Enforcing wrap-and-sort -satb 2023-08-16 * Coping with non-free software in Debian 2023-07-11 * Streamlined NTRU Prime sntrup761 goes to IETF 2023-05-12 * How To Trust A Machine 2023-04-29 * A Security Device Threat Model: The Substitution Attack 2023-04-27 * Sigstore for Apt Archives: apt-cosign 2023-04-20 * More on Differential Reproducible Builds: Devuan is 46% reproducible! 2023-04-17 * Sigstore protects Apt archives: apt-verify & apt-sigstore 2023-04-15 * Trisquel is 42% Reproducible! 2023-04-10 * OpenPGP master key on Nitrokey Start 2023-03-27 * Apt Archive Transparency: debdistdiff & apt-canary 2023-02-01 * Understanding Trisquel 2023-01-22 * Preseeding Trisquel Virtual Machines Using "netinst" Images 2022-12-30 * OpenPGP key on FST-01SZ 2022-12-24 * Second impressions of Guix 1.4 2022-12-19 * Guix 1.4 on NV41PZ 2022-12-16 * Trisquel 11 on NV41PZ: First impressions 2022-12-10 * How to complicate buying a laptop 2022-12-10 * On language bindings & Relaunching Guile-GnuTLS 2022-10-14 * Privilege separation of GSS-API credentials for Apache 2022-09-20 * Static network config with Debian Cloud images 2022-08-22 * Towards pluggable GSS-API modules 2022-07-14 * What's wrong with SCRAM? 2021-06-08 * OpenPGP smartcard with GNOME on Debian 11 Bullseye 2021-05-01 * Passive Icinga Checks: icinga-pusher 2019-12-16 * OpenPGP smartcard under GNOME on Debian 10 Buster 2019-06-21 * Offline Ed25519 OpenPGP key with subkeys on FST-01G running Gnuk 2019-03-21 * Installing Gnuk on FST-01G running NeuG 2019-03-21 * OpenPGP 2019 Key Transition Statement 2019-03-21 * Planning for a new OpenPGP key 2019-03-21 * Vikings D16 server first impressions 2017-08-03 * OpenPGP smartcard under GNOME on Debian 9.0 Stretch 2017-06-19 * GPS on Replicant 6 2017-03-04 * Why I don't Use 2048 or 4096 RSA Key Sizes 2016-11-03 * Let's Encrypt Clients 2015-12-17 * Automatic Replicant Backup over USB using rsync 2015-11-28 * Combining Dnsmasq and Unbound 2015-10-26 * Cosmos - A Simple Configuration Management System 2015-09-24 * SSH Host Certificates with YubiKey NEO 2015-06-16 * Scrypt in IETF 2015-05-19 * Certificates for XMPP/Jabber 2015-05-12 * Laptop decision fatigue 2015-05-11 * Laptop indecision 2015-03-25 * EdDSA and Ed25519 goes to IETF 2015-03-04 * Laptop Buying Advice? 2015-02-24 * Replicant 4.2 0003 on I9300 2015-01-14 * OpenPGP Smartcards and GNOME 2015-01-02 * Dice Random Numbers 2014-11-12 * The Case for Short OpenPGP Key Validity Periods 2014-08-26 * Wifi on S3 with Replicant 2014-08-10 * Replicant 4.2 0002 and NFC on I9300 2014-08-05 * Offline GnuPG Master Key and Subkeys on YubiKey NEO Smartcard 2014-06-23 * OpenPGP Key Transition Statement 2014-06-23 * Creating a small JPEG photo for your OpenPGP key 2014-06-19 * Replicant 4.2 on Samsung S3 2014-02-27 * Necrotizing Fasciitis 2014-01-05 * Replicant 4.0 on Samsung Galaxy S III 2013-11-11 * BLURB: Software repository metadata convention 2013-09-24 * Portable Symmetric Key Container (PSKC) Library 2012-10-11 * Using OATH Toolkit with Dropbox 2012-08-27 * Small syslog server 2011-12-12 * Unattended SSH with Smartcard 2011-10-11 * OpenWRT with Huawei E367 and TP-Link TL-WR1043ND 2011-05-22 * Introducing the OATH Toolkit 2011-01-20 * On Password Hashing and RFC 6070 2011-01-07 * GNU SASL with SCRAM-SHA-1-PLUS 2010-11-17 * Debian on Lenovo X201 2010-10-25 * GS2-KRB5 using GNU SASL and MIT Kerberos for Windows 2010-09-27 * Bridging SASL and GSS-API: GS2 2010-07-13 * OpenWRT 10.03 "Backfire" 2010-05-03 * GS2-KRB5 in GNU SASL 1.5.0 2010-03-31 * Fellowship interview 2010-01-08 * Nordic Free Software Award 2009 2009-11-15 * Storing OpenPGP keys in the DNS 2009-10-29 * Thread Safe Functions 2009-06-23 * CACert and GnuTLS 2009-04-16 * OpenWRT 8.09 plus Huawei E220 2009-03-05 * Redmine on Debian Lenny Using Lighttpd 2008-10-17 * FSCONS / Nordic Free Software Award Nomination 2008-10-14 * Cyclomatic Code Complexity 2008-10-07 * My blog uses Yubikey authentication 2008-06-30 * Home Wireless Network 2008-05-08 * Real-world Performance Tuning with Callgrind 2008-02-27 * IDNA flaws with regard to U+2024 2008-01-14 * PAM module for Yubico 2008-01-14 * Response to GnuTLS in Exim Debate 2007-11-09 * FSCONS 2007-10-23 * On TLS-AUTHZ 2007-10-18 * Home Audio Server 2007-09-25 * GnuTLS v2.0 2007-09-05 * Building GnuTLS and GNU SASL without running ./configure 2007-08-21 * 1 TeraByte 2007-08-14 * OpenMoko first impressions 2007-08-02 * OpenMoko Neo1973 order confirmed 2007-07-22 * Linksys WRT54G3G + Huawei E600 + OpenWRT Kamikaze = Internet at summer house 2007-07-22 * Neo1973 / OpenMoko ordered 2007-07-15 * GNU General Public License version 3 2007-06-29 * Porting to uClinux 2007-06-07 * Libidn now uses Git 2007-05-31 * Free-ietf-review 2007-05-30 * Youbico 2007-05-24 * Hacking Jobo device 2007-04-27 * First TLS v1.2 HTTPS browser in the world? 2007-04-19 * Buggy IMAP authentication on Nokia 6233 2007-04-17 * Jobo Giga Vu Pro Evolution 80GB 2007-04-14 * TLS-AUTHZ Patent Concerns 2007-04-11 * Boycott scan.coverity.com! 2007-04-02 * EnigForm - HTML/HTTP forms with OpenPGP 2007-04-01 * Password-based Authentication Protocol 2007-03-29 * New SASL GS2 document published 2007-03-29 * Libntlm 0.3.13 2007-03-27 * Debian etch on Dell Precision M65 2007-03-24 * Announcing krb5dissect 2007-03-14 * gitco 2007-03-14 * LibIDN 0.6.11 2007-03-13 * Cypak LoginKey 2006-10-18 * Base encoding 2006-10-17 * Update of Kerberos V5 over TLS draft 2006-10-03 * Kerberos 5 Credential Cache file format 2006-09-20 * RSS Feed * Email * Github Copyright (c) 2024 Simon Josefsson's blog. All Rights Reserved. Theme: Catch Box by Catch Themes Scroll Up