https://stephan.lachnit.xyz/posts/2023-02-08-debian-sbuild-mmdebstrap-apt-cacher-ng/

profile picture
Personal Blog

Physics student. Debian Developer.

  *  
  *  
  *  
  *  
  *  

  * (c) Stephan Lachnit 2023

  * Home
  * Posts
  * Talks
  * About

  * 

Setting Up Fast Debian Package Builds Using Sbuild, Mmdebstrap and
Apt-Cacher-Ng

In this post I will give a quick tutorial on how to set up fast
Debian package builds using sbuild with mmdebstrap and apt-cacher-ng.

The usual tool for building Debian packages is dpkg-buildpackage, or
a user-friendly wrapper like debuild, and while these are geat tools,
if you want to upload something to the Debian archive they lack the
required separation from the system they are run on to ensure that
your packaging also works on a different system. The usual candidate
here is sbuild. But setting up a schroot is tedious and performance
tuning can be annoying. There is an alternative backend for sbuild
that promises to make everything simpler: unshare. In this tutorial I
will show you how to set up sbuild with this backend.

Additionally to the normal performance tweaking, caching downloaded
packages can be a huge performance increase when rebuilding packages.
I do rebuilds quite often, mostly when a new dependency got
introduced I didn't specify in debian/control yet or lintian notices
a something I can easily fix. So let's begin with setting up this
caching.

Setting up apt-cacher-ng

Install apt-cacher-ng:

sudo apt install apt-cacher-ng

A pop-up will appear, if you are unsure how to answer it select no,
we don't need it for this use-case.

To enable apt-cacher-ng on your system, create /etc/apt/apt.conf.d/
02proxy and insert:

Acquire::http::proxy "http://127.0.0.1:3142";
Acquire::https::proxy "DIRECT";

In /etc/apt-cacher-ng/acng.conf you can increase the value of
ExThreshold to hold packages for a shorter or longer duration. The
length depends on your specific use case and resources. A longer
threshold takes more disk space, a short threshold like one day
effecitvely only reduces the build time for rebuilds.

If you encounter weird issues on apt update at some point the future,
you can try to clean the cache from apt-cacher-ng. You can use this
script:

Setting up mmdebstrap

Install mmdebstrap:

sudo apt install mmdebstrap

We will create a small helper script to ease creating a chroot. Open
~/.local/bin/mmupdate and insert:

#!/bin/sh
mmdebstrap \
  --variant=buildd \
  --aptopt='Acquire::http::proxy "http://127.0.0.1:3142";' \
  --arch=amd64 \
  --components=main,contrib,non-free \
  unstable \
  ~/.cache/sbuild/unstable-amd64.tar.xz \
  http://deb.debian.org/debian

Notes:

  * aptopt enables apt-cacher-ng inside the chroot.
  * --arch sets the CPU architecture (see Debian Wiki).
  * --components sets the archive components, if you don't want
    non-free pacakges you might want to remove some entries here.
  * unstable sets the Debian release, you can also set for example
    bookworm-backports here.
  * unstable-amd64.tar.xz is the output tarball containing the
    chroot, change accordingly to your pick of the CPU architecture
    and Debian release.
  * http://deb.debian.org/debian is the Debian mirror, you should set
    this to the same one you use in your /etc.apt/sources.list.

Make mmupdate executable and run it once:

chmod +x ~/.local/bin/mmupdate
mkdir -p ~/.cache/sbuild
~/.local/bin/mmupdate

If you execute mmupdate again you can see that the downloading stage
is much faster thanks to apt-cacher-ng. For me the difference is from
about 115s to about 95s. Your results may vary, this depends on the
speed of your internet, Debian mirror and disk.

If you have used the schroot backend and sbuild-update before, you
probably notice that creating a new chroot with mmdebstrap is slower.
It would be a bit annoying to do this manually before we start a new
Debian packaging session, so let's create a systemd service that does
this for us.

First create a folder for user services:

mkdir -p ~/.config/systemd/user

Create ~/.config/systemd/user/mmupdate.service and add:

[Unit]
Description=Run mmupdate
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=%h/.local/bin/mmupdate

Start the service and test that it works:

systemctl --user daemon-reload
systemctl --user start mmupdate
systemctl --user status mmupdate

Create ~/.config/systemd/user/mmupdate.timer:

[Unit]
Description=Run mmupdate daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer:

systemctl --user enable mmupdate.timer

Now every day mmupdte will be run automatically. You can adjust the
period if you think daily rebuilds are a bit excessive.

A neat advantage of period rebuilds is that they the base files in
your apt-cacher-ng cache warm every time they run.

Setting up sbuild:

Install sbuild and (optionally) autopkgtest:

sudo apt install --no-install-recommends sbuild autopkgtest

Create ~/.sbuildrc and insert:

# backend for using mmdebstrap chroots
$chroot_mode = 'unshare';

# build in tmpfs
$unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXXXX';

# upgrade before starting build
$apt_update = 1;
$apt_upgrade = 1;

# build everything including source for source-only uploads
$build_arch_all = 1;
$build_arch_any = 1;
$build_source = 1;
$source_only_changes = 1;

# go to shell on failure instead of exiting
$external_commands = { "build-failed-commands" => [ [ '%SBUILD_SHELL' ] ] };

# always clean build dir, even on failure
$purge_build_directory = "always";

# run lintian
$run_lintian = 1;
$lintian_opts = [ '-i', '-I', '-E', '--pedantic' ];

# do not run piuparts
$run_piuparts = 0;

# run autopkgtest
$run_autopkgtest = 1;
$autopkgtest_root_args = '';
$autopkgtest_opts = [ '--apt-upgrade', '--', 'unshare', '--release', '%r', '--arch', '%a' ];

# set uploader for correct signing
$uploader_name = 'Stephan Lachnit <stephanlachnit@debian.org>';

You should adjust uploader_name. If you don't want to run autopkgtest
or lintian by default you can also disable it here. Note that for
packages that need a lot of space for building, you might want to
comment the unshare_tmpdir_template line to prevent a OOM build
failure.

You can now build your Debian packages with the sbuild command :)

Finishing touches

You can add these variables to your ~/.bashrc as bonus (with adjusted
name / email):

export DEBFULLNAME="<your_name>"
export DEBEMAIL="<your_email>"
export DEB_BUILD_OPTIONS="parallel=<threads>"

In particular adjust the value of parallel to ensure parallel builds.

If you are new to signing / uploading your package, first install the
required tools:

sudo apt install devscripts dput-ng

Create ~/.devscripts and insert:

DEBSIGN_KEYID=<your_gpg_fingerpring>
USCAN_SYMLINK=rename

You can now sign the .changes file with:

debsign ../<pkgname_version_arch>.changes

And for source-only uploads with:

debsign -S ../<pkgname_version_arch>_source.changes

If you don't introduce a new binary package, you always want to go
with source-only changes.

You can now upload the package to Debian with

dput ../<filename>.changes

Resources for further reading:

https://wiki.debian.org/sbuild
https://www.unix-ag.uni-kl.de/~bloch/acng/html/index.html
https://wiki.ubuntu.com/SimpleSbuild
https://wiki.archlinux.org/title/Systemd/Timers
Thanks for reading!

howtoDebiandebootstrapsystemd

  * (c) Stephan Lachnit 2023