https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27

Skip to content
 
[                    ]
All gists Back to GitHub Sign in Sign up
 
Sign in Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}

Instantly share code, notes, and snippets.

@thesamesam

thesamesam/xz-backdoor.md

Last active March 29, 2024 22:55

  * Star 40 You must be signed in to star a gist
  * Fork 4 You must be signed in to fork a gist

Star You must be signed in to star a gist
Code Revisions 17 Stars 40 Forks 4
Embed

  * Embed Embed this gist in your website.
  * Share Copy sharable link for this gist.
  * Clone via HTTPS Clone using the web URL.
  * 
  * Learn more about clone URLs

Clone this repository at <script src="https://gist.github.com
/thesamesam/223949d5a074ebc3dce9ee78baad9e27.js"></script&
gt;
[<script src="https:/]
Save thesamesam/223949d5a074ebc3dce9ee78baad9e27 to your computer and
use it in GitHub Desktop.
Download ZIP
xz-utils backdoor situation
Raw
xz-backdoor.md

FAQ on the xz-utils backdoor

 

Background

 

On March 29th, 2024, a backdoor was discovered in xz-utils, a suite
of software that gives developers lossless compression. This package
is commonly used for compressing release tarballs, software packages,
kernel images, and initramfs images. It is very widely distributed,
statistically your average Linux or macOS system will have it
installed for convenience.

This backdoor is very indirect and only shows up when a few specific
criteria are met. However, this backdoor is triggerable by remote
unprivileged systems connecting to public SSH ports. This has been
seen in the wild where it gets activated by connections, but we do
not know yet what is required to bypass authentication (etc) with it.

TL;DR: We think the following things need to be true for your system
to be vulnerable:

  * You need to be running a rolling-release distro and updating
    religiously
  * You need to be running a distro that uses glibc and systemd
  * You need to have versions 5.6.0 or 5.6.1 of xz or liblzma
    installed (xz-utils provides the library liblzma)
  * You need to have your SSH port exposed to the public internet

If all of these are the case, please update your systems to mitigate
this threat. For more information about affected systems and how to
update, please see this article or check the xz-utils page on
Repology.

This is still a new situation. There is a lot we don't know. We don't
know if there are more possible exploit paths. We only know about
this one path. Please update your systems regardless. Unknown
unknowns are safer than known unknowns.

This is a living document. Everything in this document is made in
good faith of being accurate, but like I just said; we don't know
much about what's going on.

This is not a fault of sshd, systemd, or glibc, that is just how it
was made exploitable.

Design

 

This backdoor has several components. At a high level:

  * The release tarballs upstream publishes don't have the same code
    that GitHub has. This is common in C projects so that downstream
    consumers don't need to remember how to run autotools and
    autoconf. The version of build-to-host.m4 in the release tarballs
    differs wildly from the upstream on GitHub.
  * There are crafted test files in the tests/ folder. These files
    are in the following commits:
      + tests/files/bad-3-corrupt_lzma2.xz (
        cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0,
        74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
      + tests/files/good-large_compressed.lzma (
        cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0,
        74b138d2a6529f2c07729d7c77b1725a8e8b16f1)
  * A script called by build-to-host.m4 that unpacks this malicious
    test data and uses it to modify the build process.
  * IFUNC, a mechanism in glibc that allows for indirect function
    calls, is used to perform runtime hooking/redirection of
    OpenSSH's authentication routines. IFUNC is a tool that is
    normally used for legitimate things, but in this case it is
    exploited for this attack path.

Normally upstream publishes release tarballs that are different than
the automatically generated ones in GitHub. In these modified
tarballs, a malicious version of build-to-host.m4 is included to
execute a script during the build process.

This script (at least in versions 5.6.0 and 5.6.1) checks for various
conditions like the architecture of the machine. Here is a snippet of
the malicious version of build-to-host.m4 and an explanation of what
it does:

    if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) &&
    (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

  * If amd64/x86_64 is the target of the build
  * And if the target uses the name linux-gnu (mostly checks for the
    use of glibc)

It also checks for the toolchain being used:

      if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
      exit 0
      fi
      if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
      exit 0
      fi
      LDv=$LD" -v"
      if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
      exit 0

And if you are trying to build a Debian or Red Hat package:

    if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" =
    "xx86_64";then

This attack thusly seems to be targeted at amd64 systems running
glibc using either Debian or Red Hat derived distributions. Other
systems may be vulnerable at this time, but we don't know.

Payload

 

If those conditions check, the payload is injected into the source
tree. We have not analyzed this payload in detail. Here are the main
things we know:

  * The payload only activates if the running program has the process
    name /usr/bin/sshd. This means that systems that put sshd in /usr
    /sbin or another folder are not vulnerable. This further suspects
    targeting systemd systems due to their usrmerge initiative
    putting all binaries in /usr/bin.
  * We don't know what the payload is intended to do. We are
    investigating.
  * Vanilla upstream OpenSSH isn't affected unless one of its
    dependencies links liblzma. We are not aware of any cases of this
    in practical production systems.
  * The payload is loaded into sshd indirectly. sshd is often patched
    to support systemd-notify so that other services can start when
    sshd is running. liblzma is loaded because it's depended on by
    other parts of libsystemd. This is not the fault of systemd, this
    is more unfortunate. The patch that most distributions use is
    available here: openssh/openssh-portable#375.
  * If this payload is loaded in openssh sshd, the RSA_public_decrypt
    function will be redirected into a malicious implementation. We
    have observed that this malicious implementation can be used to
    bypass authentication. Further research is being done to explain
    why.

People

 

We do not want to speculate on the people behind this project in this
document. This is not a productive use of our time, and law
enforcement will be able to handle identifying those responsible.
They are likely patching their systems too.

xz-utils has two maintainers:

  * Lasse Collin (Larzhu) who has maintained xz since the beginning
    (~2009), and before that, lzma-utils.
  * Jia Tan (JiaT75) who started contributing to xz in the last 2-2.5
    years and gained commit access, and then release manager rights,
    about 1.5 years ago.

Lasse regularly has internet breaks and is on one at the moment,
started before this all kicked off. We believe CISA may be trying to
get in contact with him.

Acknowledgements

 

  * Andres Freund who discovered the issue and reported it to
    linux-distros and then oss-security.
  * All the hard-working security teams helping to coordinate a
    response and push out fixes.
  * Xe Iaso who resummarized this page for readability.

References

 

  * https://lwn.net/Articles/967180/
  * https://www.openwall.com/lists/oss-security/2024/03/29/4

@roycewilliams
Copy link

roycewilliams commented Mar 29, 2024

Another useful roll-up (not affiliated, just looks good) - hopefully
some opportunities to cross-pollenate.

https://boehs.org/node/everything-i-know-about-the-xz-backdoor

Sorry, something went wrong.

@boehs
Copy link

boehs commented Mar 29, 2024 *
edited

Thanks, I agree! I will replace my poor technical component with a
link to this gist

Sorry, something went wrong.

@DanielRuf
Copy link

DanielRuf commented Mar 29, 2024 *
edited

Thanks for the overview and very interesting links @boehs.

Short question, any source for this?

    However, this backdoor is triggerable by remote unprivileged
    systems connecting to public SSH ports. This has been seen in the
    wild.

Sorry, something went wrong.

@thesamesam
Copy link
Author

thesamesam commented Mar 29, 2024 *
edited

@DanielRuf I think this got introduced when paraphrasing https://
www.openwall.com/lists/oss-security/2024/03/29/4, because the issue
got noticed in the wild rather than necessarily an exploitation of
it. I'll adjust the text now. Thank you!

EDIT: OK, hope that edit looks alright.

Sorry, something went wrong.

@DanielRuf
Copy link

DanielRuf commented Mar 29, 2024

@thesamesam thanks, that is much better worded.

    However, this backdoor is triggerable by remote unprivileged
    systems connecting to public SSH ports. This has been seen in the
    wild where it gets activated by connections, but we do not know
    yet what is required to bypass authentication (etc) with it.

Exactly, I interpreted "this has been seen in the wild" as "the
(successful) exploitation has been seen in the wild".

Sorry, something went wrong.

@herzeleid02
Copy link

herzeleid02 commented Mar 29, 2024

public SSH ports as in port 22 or any port that sshd listens?

Sorry, something went wrong.

@DanielRuf
Copy link

DanielRuf commented Mar 29, 2024

    public SSH ports as in port 22 or any port that sshd listens?

The port number does not matter. Excerpt from the oss-security
mailinglist post:

    I have not yet analyzed precisely what is being checked for in
    the injected
    code, to allow unauthorized access. Since this is running in a
    pre-authentication context, it seems likely to allow some form of
    access or
    other form of remote code execution.

Sorry, something went wrong.

@duracell
Copy link

duracell commented Mar 29, 2024

    @thesamesam thanks, that is much better worded.

        However, this backdoor is triggerable by remote unprivileged
        systems connecting to public SSH ports. This has been seen in
        the wild where it gets activated by connections, but we do
        not know yet what is required to bypass authentication (etc)
        with it.

    Exactly, I interpreted "this has been seen in the wild" as "the
    (successful) exploitation has been seen in the wild".

I understand this as "the (exploit) code got executed" and not "it
was successful exploited/used".
So it checked for the key of the bad actor, but this key was never
present (as we know).

Sorry, something went wrong.

@duracell
Copy link

duracell commented Mar 29, 2024

    public SSH ports as in port 22 or any port that sshd listens?

It's triggered after connecting. So if you're using port 22 with ipv4
it's much easier to find your server with port scans. So the chances
of being a (early) target are higher, but the attack could be on any
port.

Sorry, something went wrong.

@mirabilos
Copy link

mirabilos commented Mar 29, 2024

/usr/sbin/sshd, which is the normal path, not /usr/bin/sshd (even
shitsystemd UsrMove does not (yet...) go there)

Sorry, something went wrong.

@DanielRuf
Copy link

DanielRuf commented Mar 29, 2024

@duracell the text was changed, see the previous version by checking
the revisions: https://gist.github.com/thesamesam/
223949d5a074ebc3dce9ee78baad9e27/revisions

You can also see it there: https://gist.github.com/thesamesam/
223949d5a074ebc3dce9ee78baad9e27?permalink_comment_id=5005741#
gistcomment-5005741

"it has been seen in the wild" is a common phrase for seeing attacks
and exploit attempts in the real world. Nonetheless, the changed text
is better to understand and less likely to misinterpret.

Sorry, something went wrong.

@waterkip
Copy link

waterkip commented Mar 29, 2024

I already pinged @boehs on IRC about this (because of his blog post).
This commit/PR seems suspicious as well: https://github.com/google/
oss-fuzz/pull/10667/files.

Sorry, something went wrong.

@claui
Copy link

claui commented Mar 29, 2024

    So it checked for the key of the bad actor, but this key was
    never present (as we know).

Not sure whether the malicious code allowlists a specific key, or
rather enables some kind of shared secret or magic sequence to bypass
public-key authentication altogether.

Sorry, something went wrong.

@duracell
Copy link

duracell commented Mar 29, 2024 *
edited

EDIT: Okay, I maybe interpreted some code and analysis not correctly.
I read it again and it seems not clear what the exploit code is
doing.

Old Comment:

    "it has been seen in the wild" is a common phrase for seeing
    attacks and exploit attempts in the real world. Nonetheless, the
    changed text is better to understand and less likely to
    misinterpret.

I know, but "the exploit" is, as far as I know, the replacement of
the ssh keys. This happened in the wild (it just got detected,
because the exploit has bugs), but I didn't read anything about
someone used the key to successfully login or to use the exploit in
another way. If you read anything about this, I would be very
interested. Or did I miss anything else?

Sorry, something went wrong.

@rileywilddog
Copy link

rileywilddog commented Mar 29, 2024

    The payload only activates if the running program has the process
    name /usr/bin/sshd. This means that systems that put sshd in /usr
    /sbin or another folder are not vulnerable. This further suspects
    targeting systemd systems due to their usrmerge initiative
    putting all binaries in /usr/bin.

The oss-security post says:

    Observed requirements for the exploit: [...]
    b) argv[0] needs to be /usr/sbin/sshd

Debian and EL, the targeted families, traditionally use /usr/sbin/
sshd, and usrmerge does not change this - as the linked debian page
says:

    Is this about merging /usr/bin/ and /usr/sbin/? No, there are no
    plans to do that.

Sorry, something went wrong.

Sign up for free to join this conversation on GitHub. Already have an
account? Sign in to comment

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.