[HN Gopher] Systemd replacing ELF dependencies with dlopen
___________________________________________________________________
Systemd replacing ELF dependencies with dlopen
Author : klooney
Score : 91 points
Date : 2024-04-12 16:28 UTC (6 hours ago)
(HTM) web link (mastodon.social)
(TXT) w3m dump (mastodon.social)
| rwmj wrote:
| The interesting part is:
| https://github.com/systemd/systemd/pull/32234
| AlotOfReading wrote:
| It's nice to see a proposal that fits so neatly within existing
| infrastructure and norms. I just hope they either standardize
| or drop the extensible type field idea instead of letting it
| turn into a semi-documented mess other projects can't easily
| adopt.
| stefan_ wrote:
| If they are "optional", why do initrd generators need to know
| them? This truly has all the hallmarks of a systemd
| "solution"..
| Denvercoder9 wrote:
| A dependency being optional on a technical level doesn't mean
| that it's optional on a functional level _for everyone_.
| `libfido2` mentioned in the pull request is a great example:
| if your rootdisk is encrypted using a FIDO token, you need
| that library in the initrd; but otherwise you don 't. By
| making the dependency optional and initrd generators aware of
| that optional dependency, people that need it will have
| things just work, while others don't need to install the
| library.
| Spivak wrote:
| Because the optional dependencies actually do things. By
| default an initrd-builder would probably copy all deps
| including optional ones but it now enables you to choose not
| to do that as well, an avenue that was previously not
| available.
| turminal wrote:
| I don't understand the initial motivation for converting regular
| dynamic library dependencies to dlopen dependencies. How does
| that help with reducing the footprint?
| ongy wrote:
| dynloading libraries can have some side effects.
|
| Which the recent xz attack used to mess with `sshd`, even if it
| never actually used functions from the library.
|
| dlopen loading only loads the library *when it's actually
| used*, and doesn't include everything and the kitchen sink on
| startup.
|
| If the libraries are used seldomly enough, it might also allow
| to make more dependencies optional in package management.
| marcosdumay wrote:
| So... I repeat the GP.
|
| What you gain is that the vulnerabilities will be harder to
| track down?
| ongy wrote:
| The specific way sshd was infected would not have happened
| with libxz as dlopen library.
|
| Debian's sshd only uses libsystemd for the notify api. I.e.
| it doesn't need any feature that uses libxz. If it's
| dlopen()ed, it does not need to be loaded into the process
| context to use an unrelated feature.
|
| FWIW, IMO upstream systemd should split their monolithic
| library and allow users to pick better that way, but this
| has other implications on DX.
| NewJazz wrote:
| _FWIW, IMO upstream systemd should split their monolithic
| library and allow users to pick better that way, but this
| has other implications on DX._
|
| They've done the exact opposite afaict. Libsysd used to
| be split up, now it is monolithic.
| newpavlov wrote:
| sshd should not have used libsystemd in the first place
| for the trivial notification. And the ifunc stuff is its
| own security nightmare. Papering over this by dlopen-ing
| some libs in libsystemd does not address the deeper
| issues.
| Tuna-Fish wrote:
| > FWIW, IMO upstream systemd should split their
| monolithic library and allow users to pick better that
| way, but this has other implications on DX.
|
| FWIW, upstream systemd has the opinion that no-one should
| load the library for startup notification, instead they
| should use the well documented api and just write a
| message to a socket.
| idoubtit wrote:
| Indeed, that's what the Debian maintainer of OpenSSH did
| soon after the quick security fix. He replaced the
| dependency on libsystemd with some hand-made code that
| notifies systemd by socket. https://salsa.debian.org/ssh-
| team/openssh/-/commit/cc5f37cb8...
| ptx wrote:
| That's not what they said in their update to the
| documentation[0] last week, which says that "although
| using libsystemd is a good choice, this protocol can also
| be reimplemented without external dependencies".
|
| It calls it a "good choice". Did they say somewhere else
| that no-one should do it despite it supposedly being a
| good choice?
|
| [0] https://github.com/systemd/systemd/pull/32030/files
| marcosdumay wrote:
| Upstream systemd should just cut the unused crap from
| their notification protocol and promise it's stable.
|
| They probably couldn't change it anymore if they tried
| to.
| Denvercoder9 wrote:
| The systemd notification protocol has been stable and
| documented as such for years (probably even a decade at
| this point):
| https://systemd.io/PORTABILITY_AND_STABILITY/
| advisedwang wrote:
| It meant that had the bad xz version been shipped to
| distros, only some people would been vulnerable instead of
| everyone. That is valuable.
| rcxdude wrote:
| Though only with this particular approach to the
| backdoor. If systemd had always had this approach (or
| distros hadn't patched sshd to link it in), the attackers
| would have focused on a different path from delivering
| malicious code to widely-used distros which executes in a
| priviledged context to network RCE.
| cryptonector wrote:
| > What you gain is that the vulnerabilities will be harder
| to track down?
|
| No, they're just as easily tracked.
|
| What you gain is that you can refuse to install optional
| dependencies _because they are now optional_ when they used
| to be required.
|
| That's a fairly big deal.
|
| Of course, in practice all those optional dependencies will
| be installed anyways because of other things in the distro
| needing them. You can fairly object that not much changed,
| at least for now.
|
| A better approach would be to eliminate a lot of these
| dependencies somehow. Another approach would be to sandbox
| the dependencies that cannot be eliminated.
| Tuna-Fish wrote:
| I think it's also relevant that the xz exploit made use of
| the fact that by running code before main, they could modify
| areas of memory that later get turned read-only. Any library
| that does get loaded with dlopen can of course still attack
| the process it's in, but it has less tools available to it
| for evading detection.
| dv_dt wrote:
| Wouldn't this systemd feature add a convenient centralized
| point of attack to inject libraries? Not as open at the user
| level, but similar to a LD_PRELOAD kind of vulnerability.
| ajross wrote:
| Not in a way that isn't otherwise accessible, IMHO. I mean,
| if you're really concerned about injected vulnerabilities
| into high-trust software (and you should be!) you should be
| suspicious of any dynamic linkage at all. But if you're
| going to do it, doing it late and under affirmative control
| is almost certainly the right choice.
| rcxdude wrote:
| It makes the presence of those libraries optional: you no
| longer need them to execute the relevant tool at all. It'll
| just mean you can't use features which depend on those
| libraries. For libraries that are only pulled in for less-
| commonly used features it makes a lot of sense (the specific
| case they are doing it for is generating the initrd, which
| needs a copy of any libraries used by anything running in the
| initrd, which is almost always going to include systemd, but
| the systemd in the initrd is very unlikely to use any of these
| optional features)
| kelnos wrote:
| It's because of how dynamic linking/loading works on Linux. An
| ELF dependency means that symbols in the library can override
| symbols in the binary or in other libraries. That doesn't work
| when the library is loaded with dlopen().
| blueflow wrote:
| At least Poettering acknowledged that Systemd's dependency creep
| is a problem. _[edited out]_
| Starlevel004 wrote:
| The PR for changing compression libraries to use dlopen() was
| opened several weeks before the xz-utils backdoor was revealed.
|
| https://github.com/systemd/systemd/pull/31550
| blueflow wrote:
| Then its good the systemd team changed their minds even if
| its a decade late.
| cratermoon wrote:
| "Jia Tan" was pushing hard to get systemd updated to use the
| new xz because he saw this change in progress and wanted to
| get it in a release before this went through.
| Denvercoder9 wrote:
| Do you have a source for that? The backdoor was added to a
| minor release without a SONAME change, and generally such
| updates are done by distributions and not something
| upstreams concern themselves with.
| panick21_ wrote:
| It was mentioned in the recent Oxide and Friends podcast
| with the guy who, discovered the backdoor. Maybe there is
| a link there somewhere.
| Denvercoder9 wrote:
| I'm not going to listen to a 1.5 hour podcast to find the
| exact quote, but someone must've misspoke or
| misunderstood. I did some searching in the meantime, but
| there's no evidence at all of "Jia Tan" interacting with
| the systemd developers. They pressured distributions to
| update xz-utils, but not systemd upstream.
| blueflow wrote:
| You could link the systemd issue.
| bcrl wrote:
| Why does an init replacement even need a compression
| library built into it? At some point the bloat is more risk
| than feature.
| panick21_ wrote:
| I guess this has to be said, for the another billion
| times, systemd is not and has never been 'only' an init
| replacement. And even earlier 'init' systems did more
| then init.
| dralley wrote:
| No they weren't. Jia Tan has never interacted with the
| systemd developers in any way as far as I know.
|
| It isn't systemd's decision what version of xz-utils to
| use, it's the distro's decision. And Jia Tan did push the
| distro maintainers to update xz-utils, but the systemd
| developers have _absolutely nothing_ to do with that, so
| your statement is incorrect.
| fsniper wrote:
| Wouldn't dlopen make you more vulnerable to runtime library
| attacks? Instead of having dependencies defined at the
| build/start up time, they will be loaded at runtime on demand. So
| the dependency library could be changed at any time and you
| wouldn't have any idea what code you are loading at this time?
|
| So upgrading XZ could even attack already running sshd too?
|
| I would be more inclined to have static linked libraries instead
| of having lazy loading of them for preventing this kind of
| attacks..
| rcxdude wrote:
| This seems a bit nitpicky: you just move the threshold for
| where you might load a backdoored library from first exec to
| first use which dlopens that library (which might even just be
| at exec anyway, I haven't checked the source to see how systemd
| handles it). I think the focus on the specific means that the
| xz backdoor used to get into sshd is misdirected: it's code
| that is run in all kinds of priviledged contexts and has
| thousands of different ways to turn that into RCE. If sshd
| didn't load it directly the developers of the backdoor would
| have simply taken a different route.
| wakawaka28 wrote:
| Loading at runtime is a feature. You're supposed to lock down
| the environment for execution so the right libraries get used.
| The trouble with statically linking is that you have to relink
| everything to update. You also can't easily audit which
| versions are in a binary. It seems appropriate for some small
| userspace programs but not for complex software like services
| with many dependencies.
| jcranmer wrote:
| > So upgrading XZ could even attack already running sshd too?
|
| With the specifics of how the xz injection attack worked,
| actually it wouldn't. libsystemd would have been loaded to
| implement the service-started notification, and then there is
| no reason to use anything further and therefore nothing to
| trigger the loading of the xz library.
|
| Which is kind of the point of this change: the goal is to load
| libraries on-demand instead of eagerly, so if there is nothing
| demanding the library load, the library is never loaded.
| cryptonector wrote:
| Lazy loading would have achieved the same thing with less
| codebase churn. However, the `dlopen()` thing means that the
| dependencies are then optional as far as the packaging system
| go, and therefore you can run with fewer dependencies
| installed. Also, lazy loading can be turned off with an
| environment variable -- we would need a way to indicate that
| a dependency is optional in order to have something like
| always-lazy loading for it.
|
| There's also "filters" and "auxiliary filters", which can be
| used to make dependencies optional without having to go to
| `dlopen()`. However, IIRC filter functionality is very bare-
| bones or non-workin in the Linux link-editors and ld.so.
| iforgotpassword wrote:
| > So upgrading XZ could even attack already running sshd too?
|
| Doesn't that work both ways, without dlopen updating to the
| patched version would still leave your sshd running with the
| old backdoored one.
| im3w1l wrote:
| I think this isn't motivated by security at all. That it messed
| up the backdoor was a complete coincidence.
| Denvercoder9 wrote:
| That's correct, as evidenced by the fact that the systemd
| change to dlopen() liblzma was proposed and merged before the
| backdoor was discovered.
| arp242 wrote:
| To be honest, liblzma using systemd seems like an unimportant
| detail more than anything else.
|
| On my laptop there's 141 binaries in /bin/ that link against
| lzma; an abbreviated list with similar entries omitted below. And
| that's a fairly minimal Void Linux system - the list for an
| average Ubuntu system is probably a lot longer. Sneaking in via
| kmod, grub-install, udevd, or xbps-install seems just as viable
| as sneaking in via libsystemd. Perhaps a little bit harder, but
| not much.
|
| It will remain problem as long as you have a single command that
| runs your malicious code as root. Being directly linked against
| sshd was convenient, but there's all sorts of devious shit you
| can pull once you've got root, and it doesn't take that much more
| effort. % for f in /bin/*(*); ldd $f 2>&1 | grep
| -q lzma && print $f /bin/amdgpu-arch /bin/bsdcat
| /bin/bsdcpio /bin/bsdtar /bin/bsdunzip
| /bin/c-index-test /bin/clang-* /bin/clangd
| /bin/diagtool /bin/ffmpeg /bin/ffplay
| /bin/ffprobe /bin/find-all-symbols /bin/grub-editenv
| /bin/grub-install /bin/grub-mkimage /bin/grub-
| mknetdir /bin/grub-mkrescue /bin/grub-mkstandalone
| /bin/kmod /bin/lspci /bin/modularize /bin/mpv
| /bin/nvptx-arch /bin/pp-trace /bin/pskctool
| /bin/qemu-* /bin/rtorrent /bin/tiffcp
| /bin/tiffdump /bin/tiffinfo /bin/tiffset
| /bin/tiffsplit /bin/udevadm /bin/udevd
| /bin/update-mime-database /bin/xbps-alternatives
| /bin/xbps-checkvers /bin/xbps-create /bin/xbps-dgraph
| /bin/xbps-digest /bin/xbps-fbulk /bin/xbps-fetch
| /bin/xbps-install /bin/xbps-pkgdb /bin/xbps-query
| /bin/xbps-reconfigure /bin/xbps-remove /bin/xbps-
| rindex /bin/xbps-uchroot /bin/xbps-uhelper
| /bin/xbps-uunshare /bin/xmlcatalog /bin/xmllint
| /bin/xsltproc /bin/zstd
| jwilk wrote:
| They are doing dlopen() wrong.
|
| Instead of dlopen()ing libfoo.so.7, you should build a private
| wrapper library, normally linked with -lfoo, and then dlopen()
| that private wrapper. Then you get dependency tracking for free.
| Denvercoder9 wrote:
| For some definition of free, that includes writing and
| maintaining a private wrapper library.
|
| This seems a lot like a workaround for our tools not supporting
| optional dependencies. Let's fix that instead.
| KerrAvon wrote:
| This seems like kind of a janky workaround when what you really
| want is linker support for weak/on-demand linking.
| mike_hearn wrote:
| Right, as Lennart points out macOS has supported this for
| years. And adding a DT_WEAK_NEEDED to ELF wouldn't be that
| hard. The glibc maintainers even work for Red Hat. Feels like
| systemd is the wrong place to solve this.
| gumby wrote:
| I thought mastodon supported long posts. I thought breaking these
| things into a chain of tiny links was a twitter pathology.
| amluto wrote:
| Sigh.
|
| On the one hand, this seems like a slight improvement.
|
| On the other hand, while it's a bit of a slog, it's possible to
| actually sandbox dependencies. libxz, for example, has well
| defined inputs and outputs. It's possible to load it in such a
| way that all it can do is consume inputs and produce outputs that
| depend on those inputs. (And allocate memory and waste CPU, but
| that's DoS at worst.)
| 10000truths wrote:
| A shared object that is loaded in the same thread/process as
| the rest of your application is not sandboxed from your
| application.
| amluto wrote:
| That's true. But both in-process and out-of-process
| sandboxing are doable, as is writing code in a manner that is
| provably (for various definitions of provability) free of
| various classes of side effects.
| ptx wrote:
| In can be with RLBox[0], which compiles the code to
| WebAssembly and then back to C. It's used in Firefox to
| sandbox various format parsing libraries.
|
| [0] https://rlbox.dev/
___________________________________________________________________
(page generated 2024-04-12 23:02 UTC)