https://kevinboone.me/systemd_embedded.html
Kevin Boone
Home Contact CV Software Articles
[ ]
Why systemd is a problem for embedded Linux
[tux_win]
The Internet is full of rants about systemd, and I don't want this
post to be another one. Many of the complaints people make about it
don't stand up to much scrutiny, even the technical ones; and many
complaints are not even technical. My particular interest in Linux is
primarily for embedded applications; and there, I suggest, systemd is
creating a potential (technical) problem. In this article I will try
to articulate what the problem is; but I have no solution to offer.
Recapping the last ten years
systemd is a set of integrated applications concerned with system
management. It replaces not only the traditional init process that
brings up long-lived processes, but also much of the other system
infrastructure: user session management, device management, logging,
timing, and an increasing number of other functions.
The majority of Linux users are uninterested in the pros and cons of
systemd. A small number are violently opposed to it, and a small
number are violently opposed to those who are opposed to it.
Nevertheless, most mainstream Linux distributions have adopted it
after a shorter (Fedora) or longer (Debian) period of argument.
I think there's little argument that the main target for systemd is a
general-purpose computer, with a modern, integrated graphical desktop
(Gnome, KDE). systemd does well in systems like this because it can
handle process initialization on demand and in parallel. This
potentially makes boot times faster, and keeps resource usage down,
because it isn't necessary to start a lot of services that are used
only occasionally. I don't think these advantages are the main
reasons for systemd's widespread adoption (see below), but they're
certainly important.
Embedded Linux: the problem
Unfortunately, what makes systemd good for general-purpose desktop
applications potentially makes it unsatisfactory for embedded Linux
systems. As an illustration, I'll show some memory figures from the
Raspberry Pi 3B that's currently on my workbench. The board is
running the DietPi Linux distribution - probably the best
fully-maintained Pi distribution, if you want a minimal system.
Although DietPi uses systemd (it has little alternative, as I'll
explain later) it doesn't necessarily use the full set of components.
In fact, a minimal installation of DietPi, for console operation,
installs only the systemd service manager (the 'init' process), the
log daemon, and the udev daemon.
This is the resource usage, as reported by top, for the systemd init
process (alone) on the Raspberry Pi 3B.
PID USER PR NI VIRT RES SHR S %CPU %MEM
1 root 20 0 168144 11404 8592 S 0.0 0.3
systemd init is not a particularly greedy user of memory by
contemporary standards - its virtual address space is 168Mb, but only
~8Mb is currently mapped to RAM. That's about 0.3% of the Pi 3's 4Gb
of RAM. But here's the same figures for SystemV init, on exactly the
same hardware:
PID USER PR NI VIRT RES SHR S %CPU %MEM
1 root 20 0 2744 1532 1436 S 0.0 0.2
It's much smaller. Just to be clear - I got the systemd
implementation and the SystemV init implementation from the same
Debian ARM binary repository. I haven't used fiddly compiler
optimizations or anything like that, to bias the resource usage
figures.
Now let's look at the systemd logging daemon, systemd-journald.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
147 root 20 0 41508 7744 6652 S 0.0 0.2 0:12.05 systemd-jour+
Again, it's a small, but noticeable, user of the 4Gb RAM. And, for
comparison, these are the figures from my own syslogd-lite, which I
wrote specifically for embedded applications.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
117 root 20 0 1976 72 0 S 0.0 0.0 0:00.00 syslogd
Note that the memory and CPU usage of this logging daemon are both
essentially zero. This is an unfair comparison, in a way, because I
wrote syslogd-lite specifically to minimize resource usage, and it
has few features. But it shows that it's plausible to write utilities
that
target embedded systems specifically, and sometimes it's necessary.
systemd does not do this, and was never intended to. Running a
general-purpose software set like systemd on minimal, embedded
hardware can't be expected to be effective.
With care, though, a minimal installation of systemd does run on a
low-resource ARM board like the Pi 3. In fact, it will run on a board
with 1Gb RAM, perhaps even lower. But, as the RAM decreases, the
proportion of it occupied by systemd increases.
What's less obvious is the effect on boot time of the use of systemd
init. Proponents of systemd argue - correctly, I think - that it
decreases boot time in the kinds of system for which it was designed.
But on my Pi 3 it increases boot time quite noticeably. That is, the
systemd 'init' process takes an extra half-second or so to start,
compared to SystemV init. Half a second isn't much, but in an
embedded application I actually care about boot time. I couldn't care
less on my desktop computers, because they spend most of their lives
suspended. I rarely actually reboot them.
The extra start-up time of the init process is the result, I guess,
of the additional size and complexity of the systemd executable. It's
about 200kB in size itself, and dynamically links 26 other libraries.
SystemV init, on the same hardware, is 40kB, and links only the
standard C library. The additional complexity of systemd is not
wasted: it's needed for the additional functionality that systemd
offers, in its intended environment. But this functionality is mostly
not needed in an embedded application, so the additional complexity
of systemd is a cost without a benefit.
I've found that most of the services that systemd replaces have an
alternative that is smaller, and faster to start, in an embedded
environment. Unfortunately, some of these services don't have an
alternative any more.
So what?
I'm not obliged to run systemd on my Raspberry Pi systems and, in
fact, usually I do not. I build my own Linux installation, using
binaries that I cherry-pick from the Debian repositories, and code I
write myself. Most of the binaries work without systemd. Some
complain about not finding systemd, but work anyway. Some things
don't work without systemd: the Gnome display manager, for example,
as it is built for the standard Raspberry Pi, won't work. It can be
made to work, but you have to build it from source. How long it will
continue to work, even if built from source, is open to question. But
I'm not going to be running Gnome on an embedded Linux board, so I
don't see this as a problem for my applications.
The more fundamental problem is that the people who most like systemd
are distribution managers. Sure, there are administrators who like
it, and defend it vigorously; but most end users and administrators
don't really care. But for the maintainers of mainstream Linux
distributions, systemd is like Christmas. systemd works reasonably
well for a whole range of usage scenarios and, best of all, it all
comes in one bundle. So a distribution manager doesn't have to
maintain, integrate, and support a whole bunch of system utilities
from different sources - systemd provides everything in one huge
build.
There are just a few Linux distributions that don't use systemd, and
they are not widely used. Maintaining them is difficult, with just a
handful of volunteers. All the large, commercial Linux providers have
moved to systemd.
Only Gentoo and its derivatives (so far as I know) make systemd
optional, with fully-supported alternatives. And even Gentoo can't be
made 100% free of systemd - not in practice.
Is it even practical to avoid systemd any more?
Take, for example, the udev daemon. This service monitors the kernel,
and makes configuration changes when devices are added and removed.
It's not an essential part of Linux, but it removes the need for a
whole heap of manual configuration.
At the time of writing there is no fully-featured udev implementation
outside of systemd. The original udev code was absorbed into the
systemd project about ten years ago. The Gentoo alternative eudev is
no longer fully maintained. At present, if your Linux distribution
requires udev, you're forced to use the version from systemd. This
version can be used without the rest of systemd, but it nevertheless
does systemd operations. In particular, it tries to communicate with
systemd over DBus. This communication (so far) fails gracefully if
systemd is not present, but it's not clear how long this will
continue to be the case.
systemd is a tightly-coupled set of services. I don't think a design
goal of the systemd maintainers is to make its components modular.
udev doesn't need to talk to systemd but, clearly, there's some
benefit to its doing so in a systemd installation. I understand that
there was a kind of 'gentlemen's agreement' between the systemd
maintainers and the Gentoo maintainers, to keep udev independent of
the rest of systemd. I think we can see that this agreement has
broken down a little already; I suspect it will break down more, if
the systemd folks think that tighter integration will make systemd
work better.
Many parts of systemd continue - for now - to have non-systemd
alternatives. For example, systemd has a clock synchronizer
systemd_systemtimed. The systemd maintainers are perfectly honest
that this software lacks features that exist in alternatives like
Chrony and OpenNTPD, and it's less accurate. systemd_timed is
included because it's fast to start, and satisfactory for
applications where exact time synchronization is not critical.
At present, Chrony remains widely used, even in some distributions
(like Fedora) that use systemd. But with systemd becoming ubiquitous,
what motivation will there be to maintain non-systemd alternatives?
These alternatives could fall into disrepair, even though some are
superior to the systemd utilities - and even the systemd maintainers
admit this.
Similarly, systemd has a DHCP client. It isn't the only DHCP client
that is available, but my concern is that one day it might be. In my
tests, I've found that the systemd components are larger, and slower
to start, than the traditional alternatives (where they still exist).
Again, this isn't a criticism of systemd itself - I'm testing them in
an environment they were not designed for.
So where does that leave embedded Linux?
I've found that many systemd components are less effective in an
embedded environment than the traditional alternatives. I've shown
some illustrative examples in this article, but I really don't think
there's much controversy here: this simply isn't the environment that
systemd was designed for. But it's getting increasingly difficult to
find a mainstream Linux distribution that doesn't use systemd - even
Raspberry Pi distributions use it. As systemd absorbs more
functionality into itself, there's going to be little motivation to
maintain alternatives. After all, if everybody uses systemd, what
motivation is there to support anything else? My concern is that
we're moving towards a future where Linux is inconceivable without
systemd. That will be a problem for those environments where systemd
really doesn't shine.
I wish I knew the solution to this problem. There's no point
complaining about systemd, because distribution maintainers have
grown to like it too much. And, in any event, well-reasoned,
technical concerns are drowned out by all the ranting and conspiracy
theories. All we can do - if we care - is to continue to use and
support Linux distributions that don't insist on systemd, and stand
ready to develop or maintain alternatives to those bits of Linux that
it absorbs.
# Categories: Linux Last update Aug 01 2024