[HN Gopher] Backdoor in upstream xz/liblzma leading to SSH serve...
___________________________________________________________________
Backdoor in upstream xz/liblzma leading to SSH server compromise
Author : rkta
Score : 2104 points
Date : 2024-03-29 16:16 UTC (6 hours ago)
(HTM) web link (www.openwall.com)
(TXT) w3m dump (www.openwall.com)
| Rucadi wrote:
| Saw this on nix, which was using a compromised version in the
| unstable channel, I hope not too many systems are affected.
| jeffbee wrote:
| Safety through obscurity and weirdness! If you disable ifunc,
| like any sensible person, this backdoor disables itself.
| prydt wrote:
| I'm curious now. What is ifunc? (Had difficulty finding it
| through a search)
| jeffbee wrote:
| ifunc is a GNU method of interposing function calls with
| platform-optimized versions of the function. It is used to
| detect CPU features at runtime and insert, for example,
| AVX2-optimized versions of memcmp. It is seen in crypto a
| lot, because CPUs have many crypto-specific instructions.
|
| However, I don't like it much and I think software should be
| compiled for the target machine in the first place. My 1
| hardened system that is reachable from the public network is
| based on musl, built mostly with llvm, and with ifunc
| disabled.
| cesarb wrote:
| > However, I don't like it much and I think software should
| be compiled for the target machine in the first place.
|
| That means you either have to compile software locally on
| each machine, or you have a combinatorial explosion of
| possible features.
|
| Compiling locally has several drawbacks. It needs the full
| compilation environment installed on every machine, which
| uses a lot of disk space, and some security people dislike
| it (because then attackers can also compile software
| locally on that machine); compiling needs a lot of memory
| and disk space, and uses a lot of processor time and
| electric power. It also means that signature schemes which
| only allow signed code cannot be used (or you need to have
| the signing key available on the target machine, making it
| somewhat pointless).
|
| The combinatorial explosion of features has been somewhat
| tamed lately, by bundling sets of feature into feature
| levels (x86_64-v1, etc), but that still quadruples the
| amount of compiled code to be distributed, and newer
| features still have to be selected at runtime.
| jeffbee wrote:
| I don't think you can really say it is "combinatorial"
| because there's not a mainstream machine with AES-NI but
| not, say, SSSE3. In any case if there were such a machine
| you don't need to support it. The one guy with that box
| can do scratch builds.
| myself248 wrote:
| Compiled _on_ and compiled _for_ are not the same. There
| must be a way to go to the target machine, get some
| complete dump of CPU features, copy that to the compile-
| box, do the build, and copy the resulting binaries back.
| derefr wrote:
| > That means you either have to compile software locally
| on each machine, or you have a combinatorial explosion of
| possible features.
|
| Or you just have to buy a lot of the exact same hardware.
| Secure installations tend to do that.
| eklitzke wrote:
| Obviously compiling for the target architecture is best,
| but for most software (things like crypto libraries
| excluded) 95% of the benefit of AVX2 is going to come from
| things like vectorized memcpy/memcmp. Building glibc using
| ifuncs to provide optimized implementations of these
| routines gives most users most of the benefit of AVX2 (or
| whatever other ISA extension) while still distributing
| binaries that work on older CPU microarchitectures.
| jeffbee wrote:
| ifunc memcpy also makes short copies suck ass on those
| platforms, since the dispatch cost dominates regardless
| of the vectorization. It's an open question whether ifunc
| helps or harms the performance of general use cases.
|
| By "open question" I meant that there is compelling
| research indicating that GNU memcpy/memcmp is
| counterproductive, but the general Linux-using public did
| not get the memo.
|
| https://storage.googleapis.com/gweb-
| research2023-media/pubto...
|
| "AsmDB: Understanding and Mitigating Front-End Stalls in
| Warehouse-Scale Computers" Section 4.4 "Memcmp and the
| perils of micro-optimization"
| wiml wrote:
| On the other hand, it also means that your distro can
| supply a microarchitecture-specific libc and every
| program automatically gets the memcpy improvements.
| (Well, except for the golang/rust people.)
|
| Wasn't this the point of Gentoo, back in the day? It was
| more about instruction scheduling and register allocation
| differences, but your system would be built with
| everything optimized for your uarch.
| aaronmdjones wrote:
| https://sourceware.org/glibc/wiki/GNU_IFUNC
| canada_dry wrote:
| Is there a way to easily/reliably disable ifunc globally on a
| system (e.g. ubuntu/debian) without breaking a bunch of things?
|
| FYI this looks for pkgs with liblzma:
|
| > dpkg -l |grep liblzma
|
| Versions >= 5.6 are compromised
| selckin wrote:
| https://github.com/google/oss-fuzz/pull/10667
| travem wrote:
| Interesting, I used https://ossinsight.io/analyze/JiaT75 to
| identify contributions from the account used by author of the
| backdoor. It looks like the account made other potentially
| problematic contributions to other projects.
|
| The disabling of ifunc in this PR against Google's oss-fuzz
| project maybe one way they tried to prevent this particular
| backdoor being flagged by that tool?
| https://github.com/google/oss-fuzz/pull/10667
| mxmlnkn wrote:
| There is a related issue for LLVM/clang by this person:
|
| https://github.com/llvm/llvm-project/issues/63957
| dlachausse wrote:
| > openssh does not directly use liblzma. However debian and
| several other distributions patch openssh to support systemd
| notification, and libsystemd does depend on lzma.
|
| It looks to be limited to Linux systems that are running certain
| patches. macOS and BSD seem unaffected?
| delphij wrote:
| FreeBSD is not affected as the payloads in question were
| stripped out, however we are looking into improvements to our
| workflow to further improve the import process.
| rasengan wrote:
| > One portion of the backdoor is _solely in the distributed
| tarballs_. For easier reference, here 's a link to debian's
| import of the tarball, but it is also present in the tarballs for
| 5.6.0 and 5.6.1:
|
| Ubuntu 22.04 version:
|
| dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-
| format compression library
|
| Whew!
| yogorenapan wrote:
| Very strange behavior from the upstream developers. Possible
| government involvement? I have a feeling LANG is checked to
| target servers from particular countries
| acheong08 wrote:
| One thing to note is that the person that added the commits
| only started contributing around late 2022 and appears to have
| a Chinese name. Might be required by law to plant the backdoor.
|
| That would be quite scary considering they have contributed to
| a wide variety of projects including C++
| https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...
| yorwba wrote:
| I don't think you need to worry about the C++ contribution:
| https://github.com/MicrosoftDocs/cpp-
| docs/commit/9a96311122a...
| acdha wrote:
| This does make me wonder how much they made a deliberate
| effort to build an open source portfolio so they'd look
| more legitimate when time came to mount an attack. It seems
| expensive but it's probably not really much at the scale of
| an intelligence agency.
| bombcar wrote:
| If I were doing it, I'd have a number of these "burner
| committers" ready to go when needed.
|
| If I were doing it AND amoral, I'd also be willing to
| find and compromise committers in various ways.
| kube-system wrote:
| What's the salary for a software engineer in urban China?
| 60-80k/yr USD? Two years of that salary is cheaper than a
| good single shoulder fired missile. Seems like a pretty
| cheap attack vector to me. A Javelin is a quarter million
| per pop and they can only hit one target.
| acdha wrote:
| Yeah, exactly - when your army is measured in the
| millions, picking n hundred with technical aptitude is
| basically a rounding error.
| hartator wrote:
| Until you figure there are very subtle unicode changes in
| the URL that don't diff on GitHub. :)
| ComputerGuru wrote:
| No one is being "required by law" to add vulnerabilities,
| it's more likely they are foreign agents to begin with.
| bombcar wrote:
| Depends on the law and where you are. Publicly we have
| https://www.eff.org/issues/national-security-letters/faq
| and it's likely that other requests have occurred from time
| to time, even in the USA.
| computerfriend wrote:
| > No one is being "required by law" to add vulnerabilities
|
| This is absolutely not the case in many parts of the world.
| anarazel wrote:
| LANG only needs to have _some_ value, the concrete value does
| not seem to matter.
| cf100clunk wrote:
| I am *not* a security researcher, nor a reverse engineer.
| There's lots of stuff I have not analyzed and most of what
| I observed is purely from observation rather than
| exhaustively analyzing the backdoor code.
|
| I love this sort of technical writing from contributors outside
| the mainstream debugging world who might be averse to sharing.
| What an excellently summarized report of his findings that should
| be seen as a template.
| anarazel wrote:
| FWIW, it felt intimidating as hell. And I'm fairly established
| professionally. Not sure what I'd have done earlier in my
| career (although I'd probably not have found it in the first
| place).
| internetter wrote:
| > Not sure what I'd have done earlier in my career
|
| To anybody in this sorta situation, you should absolutely
| share whatever you have. It doesn't need to be perfect, good,
| or 100% accurate, but if there's a risk you could help a lot
| of people
| bonzini wrote:
| For what it's worth the author is a PostgreSQL committer, he's
| not a security researcher but he's a pretty damn good engineer!
| vhiremath4 wrote:
| Honestly, you only get this kind of humility when you're
| working with absolute wizards on a consistent basis. That's how
| I read that whole analysis. Absolutely fascinating.
| agwa wrote:
| > openssh does not directly use liblzma. However debian and
| several other distributions patch openssh to support systemd
| notification, and libsystemd does depend on lzma.
|
| The systemd notification protocol could have been as simple as
| just writing a newline to a pipe, but instead you have to link to
| the libsystemd C library, so now security-critical daemons like
| openssh have additional dependencies like liblzma loaded into
| their address space (even if you don't use systemd as PID 1),
| increasing the risks of supply chain attacks. Thanks, systemd.
| capitainenemo wrote:
| FWIW, I did a quick check on a Devuan system. The sshd in
| Devuan does link to a libsystemd stub - this is to cut down on
| their maintenance of upstream packages. However that stub does
| not link to lzma.
| cf100clunk wrote:
| On an MX Linux (non-systemd Debian-derived distro) box I ran
| ldd on /sbin/ssh and also ran:
|
| [EDIT: this string gives cleaner results:]
| lsof -w -P -T -p $(pgrep sshd)|grep mem
|
| and saw liblzma in the results of both, so there is some sort
| of similar trickery going on.
| capitainenemo wrote:
| Huh. That's rather surprising. Do you know how MX Linux
| handles systemd? Devuan does that shimming of upstream. Do
| they perhaps just try to leave out certain packages?
|
| Anyway. I did not see lzma in the results on Devuan running
| a process check (just in case). I did see it on a Debian.
| cf100clunk wrote:
| It turns out MX uses a package called systemd-shim that
| seems to be the Debian one: $aptitude
| show systemd-shim Package: systemd-shim
| Version: 10-6 State: installed Automatically
| installed: no Priority: extra Section: admin
| Maintainer: Debian QA Group <packages@qa.debian.org>
| Architecture: amd64 Uncompressed Size: 82.9 k
| Depends: libc6 (>= 2.34), libglib2.0-0 (>= 2.39.4),
| cgmanager (>= 0.32) Suggests: pm-utils
| Conflicts: systemd-shim:i386 Breaks: systemd (<
| 209), systemd:i386 (< 209) Description: shim for
| systemd This package emulates the systemd function
| that are required to run the systemd helpers without
| using the init service
| delroth wrote:
| > The systemd notification protocol could have been as simple
| as just writing a newline to a pipe
|
| It basically is. libsystemd links to liblzma for other features
| not related to notifications.
|
| (The protocol is that systemd passes the path to a unix socket
| in the `NOTIFY_SOCKET` env variable, and the daemon writes
| "READY=1" into it.)
| agwa wrote:
| Is that protocol documented/stable? For whatever reason,
| daemons are choosing to link to libsystemd instead of
| implementing it themselves.
|
| It doesn't matter that libsystemd links to liblzma for other
| reasons. It's still in the address space of any daemon that
| is using libsystemd for the notification protocol.
| wickberg wrote:
| I know Golang has their own implementation of sd_notify().
|
| For Slurm, I looked at what a PITA pulling libsystemd into
| our autoconf tooling would be, stumbled on the Golang
| implementation, and realized it's trivial to implement
| directly.
| tripflag wrote:
| indeed; it should be trivial in any language. Here's
| python: https://github.com/9001/copyparty/blob/a080759a03
| ef5c0a6b06c...
| cesarb wrote:
| It should be trivial in any language _which has AF_UNIX_.
| Last time I looked, Java didn 't have it, so the only way
| was to call into non-Java code.
| reftel wrote:
| Then I suggest you have another look =)
| https://inside.java/2021/02/03/jep380-unix-domain-
| sockets-ch...
| fullstop wrote:
| Under Limitations: Datagram support
| reftel wrote:
| It appears you are correct. What an odd limitation!
| fullstop wrote:
| At first I thought that this surely could not be true as
| of today, but it looks like it is.
|
| There is AF_UNIX support, but only for streams and not
| datagrams: https://bugs.openjdk.org/browse/JDK-8297837
|
| What an odd decision. I suppose that you could execute
| systemd-notify but that's a solution that I would not
| like.
| cesarb wrote:
| > I suppose that you could execute systemd-notify but
| that's a solution that I would not like.
|
| What I did was to use JNA to call sd_notify() in
| libsystemd.so.0 (when that library exists), which works
| but obviously does not avoid using libsystemd. I suppose
| I could have done all the socket calls into glibc by
| hand, but doing that single call into libsystemd directly
| was simpler (and it can be expected to exist whenever
| systemd is being used).
| ptx wrote:
| It looks like the FFI (Project Panama) finally landed in
| Java 22, released a few days ago:
| https://openjdk.org/jeps/454
|
| Unless that feature also has some weird limitation, you
| could probably use that to call the socket API in libc.
| pkaye wrote:
| Can me point me to the Golang implementation? Is it a
| standard package?
| yencabulator wrote:
| Most likely https://github.com/coreos/go-systemd
| tonyg wrote:
| Strange protocol. Why not pass a path to a _file_ that should
| be `touch`d and /or written to, I wonder? Would avoid the
| complexity of sockets.
| wiml wrote:
| > libsystemd links to liblzma for other features not related
| to notifications
|
| Which is pretty emblematic of systemd's primary architectural
| fault!
| IAmNotACellist wrote:
| systemd getting its tentacles everywhere they can squeeze
| is a feature, not a bug
| fullstop wrote:
| Also thanks to Debian for modifying openssh.
| cassianoleal wrote:
| You're not wrong. Had Debian not patched it in this way, OP
| might have never found it, leaving all other distros who do
| the same vulnerable.
|
| Note that OP found this in Debian sid as well, which means
| it's highly unlikely this issue will find its way into any
| Debian stable systems.
| fullstop wrote:
| Right, the systemd notification framework is very simple
| and I've used it in my projects. I didn't even know that
| libsystemd provided an implementation.
|
| My Arch system was not vulnerable because openssh was not
| linked to xz.
|
| IMO every single commit from JiaT75 should be reviewed and
| maybe even rolled back, as they have obliterated their
| trust.
|
| edit:
|
| https://github.com/google/oss-fuzz/pull/10667
|
| Even this might be nefarious.
| gopher_space wrote:
| > the systemd notification framework is very simple and
| I've used it in my projects
|
| Have you come across an outline or graph of systemd that
| you really like, or maybe a good example of a minimal
| setup?
| SAI_Peregrinus wrote:
| If they hadn't been modifying SSH their users would never
| have been hit by this backdoor. Of course if it is actually
| intended to target SSH on Debian systems, the attacker
| would likely have picked a different dependency. But adding
| dependencies like Debian did here means that those
| dependencies aren't getting reviewed by the original
| authors. For security-critical software like OpenSSH such
| unaudited dependencies are prime targets for attacks like
| this.
| sitkack wrote:
| It takes a village.
| cassianoleal wrote:
| My point was, this is not "Debian did a thing". Lots of
| other distros do the same thing. In this particular case,
| it was in fact fortunate for users of all these other
| distros that Debian did it, lest this vulnerability might
| have never been found!
|
| Also, only users on sid (unstable) and maybe testing seem
| to have been affected. I doubt there are many Debian
| servers out there running sid.
|
| Debian stable (bookworm) has xz-utils version 5.4.1:
| https://packages.debian.org/bookworm/xz-utils
| fullstop wrote:
| > Debian stable (bookworm) has xz-utils version 5.4.1:
| https://packages.debian.org/bookworm/xz-utils
|
| Guess who released 5.4.1? JiaT75!
| cassianoleal wrote:
| 5.4.1 doesn't even have the `m4/build-to-host.m4` script
| that pulls the backdoor's tarball.
|
| https://salsa.debian.org/debian/xz-utils/-/tree/v5.4.1/m4
| fullstop wrote:
| Neither does https://salsa.debian.org/debian/xz-
| utils/-/tree/v5.6.0/m4
|
| The script was not present in the git tree, only in the
| released archives.
|
| I'm also suggesting that there could be more than one
| exploit present. All of their commits should be rolled
| back, none of it can be trusted.
| sroussey wrote:
| Not just commits, but all tarballs released with his key.
| cassianoleal wrote:
| > The script was not present in the git tree, only in the
| released archives.
|
| I confess I couldn't quite figure out the branching and
| tagging strategy on that repo. Very weird stuff. That
| script seems to have been added by Sebastian Andrzej
| Siewior just ahead of the 5.6.0 release. It's definitely
| present in the Debian git tree, and probably in many
| other distros since others seem to be affected.
|
| The commit where the script was added to Debian is tagged
| `upstream/v5.6.0` despite the script itself not being
| present on that tag upstream: https://github.com/tukaani-
| project/xz/tree/v5.6.0/m4
|
| > I'm also suggesting that there could be more than one
| exploit present. All of their commits should be rolled
| back, none of it can be trusted.
|
| I agree.
| Jasper_ wrote:
| That is all the protocol is. From https://www.freedesktop.org/s
| oftware/systemd/man/latest/sd_n...:
|
| > These functions send a single datagram with the state string
| as payload to the socket referenced in the $NOTIFY_SOCKET
| environment variable.
|
| The simplest implementation (pseudocode, no error handling, not
| guaranteed to compile), is something like:
| const char *addrstr = getenv("NOTIFY_SOCKET"); if
| (addrstr) { int fd = socket(AF_UNIX, SOCK_DGRAM,
| 0); struct sockaddr_un addr = { .sun_family =
| AF_UNIX }; strncpy(addr.sun_path,
| sizeof(addr.sun_path), addrstr); connect(fd,
| (struct sockaddr*) &addr); write(fd, "READY=1");
| close(fd); }
| throwaway71271 wrote:
| goddamnit leftpad got us too :)
| iforgotpassword wrote:
| This is what I did for a daemon I'm maintaining. Type=notify
| support was requested but I'm really allergic to adding new
| libs to a project until they really do some heavy lifting and
| add enough value. I was pleasantly surprised the protocol was
| that simple and implemented it myself. I think systemd should
| just provide a simple standalone reference implementation and
| encourage people to copy it into their project directly. (But
| maybe they already do, I did that almost a decade ago IIRC
| when the feature was relatively new.)
| bbarnett wrote:
| One of the objections that many people do not understand, is
| that systemd adds complexity. Unnecessary complexity. Boats
| full, loads full, mountains full of complexity.
|
| Yes, there are things delivered with that complexity. However,
| as an example, sysvinit is maybe, oh, 20k lines of code
| including binaries, heck including all core init scripts.
|
| What's systemd? 2M lines? It was >1M lines 4+ years ago.
|
| For an init system, a thing that is to be the core of
| stability, security, and most importantly glacial, stable
| change -- that is absurdly complex. It's exceedingly over
| engineered.
|
| And so you get cases like this. And cases like that, and that
| over there, and that case over there too. All which could not
| exist, if systemd didn't try to overengineer, over complicate
| everything.
|
| Ah well. I'm still waiting for someone to basically fork
| systemd, remove all the fluff (udev, ntp, dns, timers, restart
| code, specialized logging, on and on and on), and just end up
| with systemd compatible service files.
|
| But not yet. So... well, oh well.
| gavinhoward wrote:
| I have a design in the works to do just this.
|
| The problem? It's on the backburner because I don't think I
| could find a business model to make money from it.
|
| I don't think offering support for a price would work, for
| example.
| adr1an wrote:
| What about sponsors? Actually, now I have the idea of a
| platform similar to Kickstarter but for software
| development, and with just sponsors. It wouldn't work,
| sure... Except in some cases. Like when things like this
| happen...
| gavinhoward wrote:
| Sponsors are fickle, unfortunately, and they tend to
| remove "donations" when money gets tight.
|
| If I am considered a full vendor, though, and a vendor
| for a critical piece of software, they might keep me
| around.
| iforgotpassword wrote:
| What's the point of your implementation? systemd is totally
| modular, you can use just the init system without networkd,
| timesyncd, resolved, nspawn, whatever else I forgot about.
|
| If you want you can just use systemd as PID1 for service
| management and enjoy a sane way to define and manage
| services - and do everything in archaic ways like 20 years
| ago.
| gavinhoward wrote:
| There are two points to the implementation:
|
| * Choice. If I have a separate implementation, my users
| do not have to be subject to systemd's choices. And I do
| not either.
|
| * The same implementation will have the same bugs, so in
| the same way that redundant software has multiple
| independent implementations, having an independent
| implementation will avoid the same bugs. It may have
| different bugs, sure, but my goal would be to test like
| SQLite and achieve DO-178C certification. Or as close as
| I could, anyway.
| iforgotpassword wrote:
| I'd assume chances of monetizing this are incredibly low.
| There already is an init system that understands systemd
| unit files, the name escapes my mind unfortunately.
| DO-178C might be a selling point literally, but whether
| there's enough potential customers for ROI is
| questionable.
| gavinhoward wrote:
| I unfortunately agree with you. Hence why it's on the
| backburner.
| jorvi wrote:
| > Ah well. I'm still waiting for someone to basically fork
| systemd, remove all the fluff (udev, ntp, dns, timers,
| restart code, specialized logging, on and on and on)
|
| Most of the things you named there are modular and can be
| easily disabled.
|
| Furthermore, udev precedes systemd and systemd has in fact
| its own replacement for it (though the name escapes me).
|
| Kind of a classic, people loving harping on systemd without
| properly understanding it.
| johnny22 wrote:
| systemd subsumed udev. Eudev is what folks who don't have
| systemd use.
| nottorp wrote:
| > are modular and can be easily disabled.
|
| That's a common defense for any bloatware. If they're
| modular and easily disabled then why are they all enabled
| by default?
| matheusmoreira wrote:
| Systemd is actually pretty damn good _and_ it 's GPL licensed
| free software.
|
| I understand that people don't like the way it seems to work
| itself into the rest of Linux user space as a dependency but
| that's actually our own fault for not investing the man power
| that Red Hat invests. We have better things to do than make
| our own Linux user space and so they have occupied that
| niche. It's free software though, we always have the freedom
| to do whatever we want.
|
| By the way, all the stuff you mentioned is not really part of
| the actual init system, namely PID 1. There's an actual
| service manager for example and it's entirely separate from
| init. It manages services really well too, it's measurably
| better than all that "portable" nonsense just by virtue of
| using cgroups to manage processes which means it can actually
| supervise poorly written double forking daemons.
| pessimizer wrote:
| People are complaining that it's too big, labyrinthine, and
| arcane to audit, not that it doesn't work. They would
| prefer other things that work, but don't share those
| characteristics.
|
| Also, the more extensive the remit (of this init), the more
| complexly interconnected the interactions between the
| components; the fewer people understand the architecture,
| the fewer people understand the code, the fewer people read
| the code. This creates a situation where the codebase is
| getting larger and larger at a rate faster than the growth
| of the number of man-hours being put into reading it.
|
| This has to make it easier for people who are systemd
| specialists to put in (intentionally or unintentionally)
| backdoors and exploitable bugs that will last for years.
|
| People keep defending systemd by talking about its UI and
| its features, but that completely misses the point. If
| systemd were replaced by something comprehensible and less
| internally codependent, _even if_ the systemd UI and
| features were preserved, most systemd complainers would be
| over the moon with happiness. Red Hat invests too much into
| completely replacing linux subsystems, they should take a
| break. Maybe fix the bugs in MATE.
| dralley wrote:
| >the more complexly interconnected the interactions
| between the components
|
| This is a bit of a rich criticism of systemd, given the
| init scripts it replaced.
|
| > Red Hat invests too much into completely replacing
| linux subsystems, they should take a break. Maybe fix the
| bugs in MATE.
|
| MATE isn't a Red Hat project. And nobody complains about
| Pipewire.
| _factor wrote:
| Let's not get started on how large the kernel is. Large
| code bases increase attack surface, period. The only
| sensible solution is to micro service out the pieces and
| only install the bare essentials. Why does the an x86
| server come with Bluetooth drivers baked in?
|
| The kernel devs are wasting time writing one offs for
| every vendor known to man, and it ships to desktops too.
| matheusmoreira wrote:
| > Red Hat invests too much into completely replacing
| linux subsystems, they should take a break.
|
| They should do whatever they feel is best for them, as
| should we. They're releasing free as in freedom GPL Linux
| software, high quality software at that. Thus I have no
| moral objections to their activities.
|
| You have to realize that this is really a symptom of
| _others_ not putting in the required time and effort to
| produce a better alternative. I know because I reinvent
| things regularly just because I enjoy it. People
| underestimate by many orders of magnitude the effort
| required to make something like this.
|
| So I'm really thankful that I got systemd, despite many
| valid criticisms. It's a pretty good system, and it's not
| proprietary nonsense. I've learned to appreciate it.
| ongy wrote:
| How is the service manager different from PID1/init?
| matheusmoreira wrote:
| They are completely different things.
|
| Init just a more or less normal program that Linux starts
| by default and by convention. You can make it boot
| straight into bash if you want. I created a little
| programming language with the ultimate goal of booting
| Linux directly into it and bringing up the entire system
| from inside it.
|
| It's just a normal process really. Two special cases that
| I can think of: no default signal handling, and it can't
| ever exit. Init will not get interrupted by signals
| unless it explicitly configures the signal dispositions,
| even SIGKILL will not kill it. Linux will panic if PID 1
| ever exits so it can't do that.
|
| Traditionally, it's also the orphaned child process
| reaper. Process descriptors and their IDs hang around in
| memory until something calls wait on them. Parent
| processes are supposed to do that but if they don't it's
| up to init to do it. Well, that's the way it works
| traditionally on Unix. On Linux though that's
| customizable with prctl and PR_SET_CHILD_SUBREAPER so you
| actually can factor that out to a separate process. As
| far as I know, systemd does just that, making it _more_
| modular and straight up _better_ than traditional Unix,
| simply because this separate process won 't make Linux
| panic if it ever crashes.
|
| As for the service manager, this page explains process
| and service management extremely well:
|
| https://mywiki.wooledge.org/ProcessManagement
|
| Systemd does it right. It does everything that's
| described in there, does it correctly, uses powerful
| Linux features like cgroups for even better process
| management and also solves the double forking problem
| described in there. It's essentially a solved problem
| with systemd. Even the people who hate it love the unit
| files it uses and for good reason.
| dralley wrote:
| This is a bit like complaining that the Linux kernel has 30
| million lines of code, while ignoring that 3/4 of that is in
| hardware support (drivers) or filesystems that nobody is
| actually required to use at any given time.
|
| systemd is a collection of tools, one of which is an init
| system. Nobody accused GNU yes of being bloated just because
| it's in a repository alongside 50 other tools.
| msm_ wrote:
| Gnu yes is actually pretty bloated. It's 130 lines of code
| for something so trivial [1]! ;)
|
| [1] https://github.com/coreutils/coreutils/blob/master/src/
| yes.c
| bananapub wrote:
| > One of the objections that many people do not understand,
| is that systemd adds complexity. Unnecessary complexity.
| Boats full, loads full, mountains full of complexity.
|
| this is and always has been such a dumb take.
|
| if you'd like to implement an init (and friends) system that
| doesn't have "unnecessary complexity" and still provides all
| the functionality that people currently want, then go and do
| so and show us? otherwise it's just whinging about things not
| being like the terrible old days of init being a mass of
| buggy and racey shell scripts.
| simoncion wrote:
| > about things not being like the terrible old days of init
| being a mass of buggy and racey shell scripts.
|
| Zero of the major distros used System V init by default.
| Probably only distros like Slackware or Linux From Scratch
| even suggested it.
|
| It's unfortunate that so many folks uncritically swallowed
| the Systemd Cabal's claims about how they were the first to
| do this, that, or the other.
|
| (It's also darkly amusing to note that every service that
| has nontrivial pre-start or post-start configuration and/or
| verification requirements ends up using systemd to run at
| least one shell script... which is what would have often
| been inlined into their init script in other init systems.)
| marcosdumay wrote:
| As long as Gnome requires bug-compatibility with systemd,
| nobody will rewrite it.
| quotemstr wrote:
| > One of the objections that many people do not understand,
| is that systemd adds complexity. Unnecessary complexity.
| Boats full, loads full, mountains full of complexity.
|
| Complexity that would otherwise be distributed to a sea of
| ad-hoc shell scripts? systemd is a win
| bennyhill wrote:
| > so now security-critical daemons like openssh have additional
| dependencies like liblzma
|
| Systemd itself seems security-critical to me. Would removing
| other dependencies on libsystemd really make a secure system
| where systemd was compromised through its library?
| agwa wrote:
| 1. systemd (at least the PID 1 part) does not talk to the
| network, so a remotely-accessible backdoor would need to be
| more complex (and thus more likely to be detected) than a
| backdoor that can be loaded into a listening daemon like
| openssh.
|
| 2. You can run Debian systems without systemd as PID 1, but
| you're still stuck with libsystemd because so many daemons
| now link with it.
| capitainenemo wrote:
| .. well, you can use a shim package as devuan did.
| chasil wrote:
| > systemd... does not talk to the network...
|
| Socket activation and the NFS automounter appear to.
|
| If I run "netstat -ap" I see pid 1 listening on enabled
| units.
|
| Edit: tinysshd is specifically launched this way.
|
| Edit2: there is also substantial criticism of xz on
| technical grounds.
|
| https://www.nongnu.org/lzip/xz_inadequate.html
| poettering wrote:
| Uh. systemd documents the protocol at various places and the
| protocol is trivial: a single text datagram sent to am AF_UNIX
| socket whose path you get via the NOTIFY_SOCKET. That's trivial
| to implement for any one with some basic unix programming
| knowledge. And i tell pretty much anyone who wants to listen
| that they should just implement the proto on their own if thats
| rhe only reason for a libsystemd dep otherwise. In particular
| non-C environments really should do their own native impl and
| not botjer wrapping libsystemd just for this.
|
| But let me stress two other things:
|
| Libselinux pulls in liblzma too and gets linked into _tons_
| more programs than libsystemd. And will end up in sshd too (at
| the very least via libpam /pam_selinux). And most of the really
| big distros tend do support selinux at least to some level.
| Hence systemd or not, sshd remains vulnerable by this specific
| attack.
|
| With that in mind libsystemd git dropped the dep on liblzma
| actually, all compressors are now dlopen deps and thus only
| pulled in when needed.
| o11c wrote:
| Deferring the load of the library often just makes things
| harder to analyze, not necessarily more secure. I imagine
| many of the comments quoting `ldd` are wrongly forgetting
| about `dlopen`.
|
| (I really wish there were a way to link such that the library
| isn't actually loaded but it still shows in the metadata, so
| you can get the performance benefits of doing less work but
| can still analyze the dependency DAG easily)
| iforgotpassword wrote:
| > And i tell pretty much anyone who wants to listen that they
| should just implement the proto on their own if thats rhe
| only reason for a libsystemd dep otherwise.
|
| That's what I think too. Do the relevant docs point this out
| too? Ages ago they didn't. I think we should try to avoid
| that people just google "implement systemd notify daemon" and
| end up on a page that says "link to libsystemd and call
| sd_notify()".
| agwa wrote:
| > And i tell pretty much anyone who wants to listen that they
| should just implement the proto on their own if thats rhe
| only reason for a libsystemd dep otherwise
|
| Could you point out where the man page (https://www.freedeskt
| op.org/software/systemd/man/latest/sd_n...) says this?
| mardifoufs wrote:
| What? I don't get it? Isn't it on Debian if they modified the
| package to do something like this? Why would you blame systemd
| for maintainers doing something that upstream has never
| required or recommended?
| hnald wrote:
| It's unfortunate that the anti-systemd party lost the war...
| years ago. But I don't blame systemd, Lennart Pottering or the
| fanboys (though it would have been so much better if the guy
| never worked in open source or wasn't such a prolific
| programmer). I blame Debian and its community for succumbing to
| this assault on Unix philosophy (again, years ago).
| 0x0 wrote:
| Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as
| well). Hopefully not affected on mac?
| stephenr wrote:
| The issue is caused by patches to add integration with systemd,
| so no, this won't affect SSH on a Mac.
| 0x0 wrote:
| Just because macs don't use systemd, doesn't mean the
| backdoor won't work. The oss-sec post talks about liblzma
| having backdoors in crc32_resolve() and crc64_resolve() and
| that it has not been fully reversed. This could perhaps
| affect more than just sshd on x86-64 linux?
| anarazel wrote:
| > Just because macs don't use systemd, doesn't mean the
| backdoor won't work.
|
| Practically speaking it can't - For one the script injected
| into the build process tests that you're running on x86-64
| linux, for another, the injected code is elf code, which
| wouldn't link on a mac. It also needs to manipulate dynamic
| linker datastructures, which would also not work the same
| on a mac.
|
| > This could perhaps affect more than just sshd on x86-64
| linux?
|
| This however is true - /usr/sbin/sshd was the only argv[0]
| value that I found to "work", but it's possible there are
| others. "/usr/sbin/sshd" isn't a string directly visible in
| the injected code, so it's hard to tell.
| stephenr wrote:
| The article explains _numerous_ concurrent conditions that
| have to be met for the backdoor to even be activated (at
| build time, not runtime), which combined make it extremely
| unlikely this will affect SSH on macOS:
|
| - linux
|
| - x86-64
|
| - building with gcc & the GNU linker
|
| - part of a .deb or .rpm build
|
| Add to that, as the article explains: openssh does not
| directly use liblzma, the only reason SSH is affected at
| all, is because some Linux Distros patch openssh to link it
| against systemd, which _does_ depend on liblzma.
|
| Could it affect things _other_ than SSH on a Mac? Unlikely.
| The compromise was introduced in 5.6.0, but macOS Sonoma
| has 5.4.4 (from August last year).
| woodruffw wrote:
| Homebrew reverted to 5.4.6 once the maintainers became aware.
| The current understanding is that macOS is not affected, but
| that's not certain.
|
| [1]: https://github.com/Homebrew/homebrew-core/pull/167512
| asveikau wrote:
| That's completely crazy, the backdoor is introduced through a
| very cryptic addition to the configure script. Just looking at
| the diff, it doesn't look malicious at all, it looks like build
| script gibberish.
| zb3 wrote:
| Yeah, now imagine they succeeded and it didn't cause any
| performance issues...
|
| Can we even be sure no such successful attempt has already been
| made?
| gpvos wrote:
| No, we can't.
| coldpie wrote:
| You can be certain it has happened, many times. Now think of
| all the software we mindlessly consume via docker, language
| package managers, and the like.
|
| Remember, there is no such thing as computer security. Make
| your decisions accordingly :)
| agwa wrote:
| Thanks to autoconf, we're now used to build scripts looking
| like gibberish. A perfect place to hide a backdoor.
| rwmj wrote:
| This is my main take-away from this. We must stop using
| upstream configure and other "binary" scripts. Delete them
| all and run "autoreconf -fi" to recreate them. (Debian
| already does something like this I think.)
| anthk wrote:
| I always run autoreconfig -ifv first.
| rwmj wrote:
| In this case it wouldn't be sufficient. You had to also
| delete m4/build-to-host.m4 for autoreconf to recreate it.
| cesarb wrote:
| > We must stop using upstream configure and other "binary"
| scripts. Delete them all and run "autoreconf -fi" to
| recreate them.
|
| I would go further than that: all files which are in a
| distributed tarball, but not on the corresponding git
| repository, should be treated as suspect.
|
| Distributing these generated autotools files is a relic of
| times when it could not be expected that the target machine
| would have all the necessary development environment
| pieces. Nowadays, we should be able to assume that whoever
| wants to compile the code can also run
| autoconf/automake/etc to generate the build scripts from
| their sources.
|
| And other than the autotools output, and perhaps a couple
| of other tarball build artifacts (like cargo simplifying
| the Cargo.toml file), there should be no difference between
| what is distributed and what is on the repository. I recall
| reading about some project to find the corresponding commit
| for all Rust crates and compare it with the published
| crate, though I can't find it right now; I don't know
| whether there's something similar being done for other
| ecosystems.
| nolist_policy wrote:
| One small problem with this is that autoconf is not
| backwards-compatible. There are projects out there that
| need older autoconf than distributions ship with.
| rwmj wrote:
| There are, and they need to be fixed.
| cesarb wrote:
| The test code generated by older autoconf is not going to
| be work correctly with newer GCC releases due to the
| deprecation of implicit int and implicit function
| declarations (see
| https://fedoraproject.org/wiki/Changes/PortingToModernC),
| so these projects already have to be updated to work with
| newer autoconf.
| londons_explore wrote:
| Why do we distribute tarballs at all? A git hash should
| be all thats needed...
| cesarb wrote:
| > Why do we distribute tarballs at all? A git hash should
| be all thats needed...
|
| A git hash means nothing without the repository it came
| from, so you'd need to distribute both. A tarball is a
| self-contained artifact. If I store a tarball in a CD-
| ROM, and look at it twenty years later, it will still
| have the same complete code; if I store a git hash in a
| CD-ROM, without storing a copy of the repository together
| with it, twenty years later there's a good chance that
| the repository is no longer available.
|
| We could distribute the git hash together with a shallow
| copy of the repository (we don't actually need the
| history as long as the commit with its trees and blobs is
| there), but that's just reinventing a tarball with more
| steps.
|
| (Setting aside that currently git hashes use SHA-1, which
| is not considered strong enough.)
| londons_explore wrote:
| except it isn't reinventing the tarball, because the git
| hash _forces_ verification that every single file in the
| repo matches that in the release.
|
| And git even has support for "compressed git repo in a
| file" or "shallow git repo in a file" or even "diffs from
| the last release, compressed in a file". They're called
| "git bundle"'s.
|
| They're literally perfect for software distribution and
| archiving.
| bombcar wrote:
| People don't know how to use git hashes, and it's not
| been "done". Whereas downloading tarballs and verifying
| hashes of the tarball has been "good enough" because the
| real thing it's been detecting is communication faults,
| not supply chain attacks.
|
| People also like version numbers like 2.5.1 but that's
| not a hash, and you can only indirectly make it a hash.
| nicolas_17 wrote:
| The backdoor is in an .m4 file that gets parsed by autoconf
| to generate the configure script. Running autoconf yourself
| won't save you.
| rwmj wrote:
| That's not entirely true. autoreconf will regenerate
| m4/build-to-host.m4 but only if you delete it first.
| tobias2014 wrote:
| It seems like this was the solution for archlinux, pull
| directly from the github tag and run autogen: https://gitla
| b.archlinux.org/archlinux/packaging/packages/xz...
| k8svet wrote:
| Oh come on, please, let's put autotools out to pasture.
| I've lost so much of my life fighting autotools crap
| compared to "just use meson".
| omoikane wrote:
| It looks like an earlier commit with a binary blob "test data"
| contained the bulk of the backdoor, then the configure script
| enabled it, and then later commits patched up valgrind errors
| caused by the backdoor. See the commit links in the
| "Compromised Repository" section.
|
| Also, seems like the same user who made these changes are still
| submitting changes to various repositories as of a few days
| ago. Maybe these projects need to temporarily stop accepting
| commits until further review is done?
| 20after4 wrote:
| > "Given the activity over several weeks, the committer is
| either directly involved or there was some quite severe
| compromise of their system. Unfortunately the latter looks like
| the less likely explanation, given they communicated on various
| lists about the "fixes" mentioned above."
|
| Crazy indeed.
| tetromino_ wrote:
| A big part of the problem is all the tooling around git (like
| the default github UI) which hides diffs for binary files like
| these pseudo-"test" files. Makes them an ideal place to hide
| exploit data since comparatively few people would bother
| opening a hex editor manually.
| acdha wrote:
| How many people read autoconf scripts, though? I think those
| filters are symptom of the larger problem that many popular
| C/C++ codebases have these gigantic build files which even
| experts try to avoid dealing with. I know why we have them
| but it does seem like something which might be worth
| reconsidering now that the tool chain is considerably more
| stable than it was in the 80s and 90s.
| bonzini wrote:
| How many people read build.rs files of all the transitive
| dependencies of a moderately large Rust project?
|
| Autoconf is bad in this respect but it's not like the
| alternatives are better (maybe Bazel).
| salawat wrote:
| I mean, autoconf is basically a set of template programs
| for snffing out whether a system has X symbol available to
| the linker. Any replacement for it would end up morphing
| into it over time.
|
| Some things are just that complex.
| acdha wrote:
| We have much better tools now and much simpler support
| matrices, though. When this stuff was created, you had
| more processor architectures, compilers, operating
| systems, etc. and they were all much worse in terms of
| features and compatibility. Any C codebase in the 90s was
| half #ifdef blocks with comments like "DGUX lies about
| supporting X" or "SCO implemented Y but without option Z
| so we use Q instead".
| johnny22 wrote:
| I don't see how showing the binary diffs would help.
| 99.99999% of people would just scroll right past them
| anyways.
| AeroNotix wrote:
| Even in binary you can see patterns. Not saying it's
| perfect to show binary diffs (but it is better than showing
| nothing) but I know even my slow mammalian brain can spot
| obvious human readable characters in various binary
| encoding formats. If I see a few in a row which doesn't
| make sense, why wouldn't I poke it?
| ok123456 wrote:
| What should I look for? The evil bit set?
| johnny22 wrote:
| Sure, the same person who's gonna be looking is the same
| person who'd click "show diff"
| janc_ wrote:
| This particular file was described as an archive file
| with corrupted data somewhere in the middle. Assuming you
| wanted to scroll that far through a hexdump of it, there
| could be pretty much any data in there without being
| suspicious.
| londons_explore wrote:
| testdata should not be on the same machine as the build is
| done. testdata (and tests generally) aren't as well audited,
| and therefore shouldn't be allowed to leak into the finished
| product.
|
| Sure - you want to test stuff, but that can be done with a
| special "test build" in it's own VM.
| bangoodirro wrote:
| 00011900: 0000 4883 f804 7416 b85f 5f5f 5f33 010f
| ..H...t..____3.. | 00011910: b651 0483 f25a 09c2 0f84 5903
| 0000 488d .Q...Z....Y...H. | 00011920: 7c24 40e8 5875 0000
| 488b 4c24 4848 3b4c |$@.Xu..H.L$HH;L | 00011930: 2440 7516
| 4885 c074 114d 85ff 0f84 3202 $@u.H..t.M....2. | 00011940:
| 0000 498b 0ee9 2c02 0000 b9fe ffff ff45 ..I...,........E |
| 00011950: 31f6 4885 db74 0289 0b48 8bbc 2470 1300
| 1.H..t...H..$p.. | 00011960: 0048 85ff 0f85 c200 0000 0f57
| c00f 2945 .H.........W..)E | 00011970: 0048 89ac 2470 1300
| 0048 8bbc 2410 0300 .H..$p...H..$... | 00011980: 0048 8d84
| 2428 0300 0048 39c7 7405 e8ad .H..$(...H9.t... | 00011990:
| e6ff ff48 8bbc 24d8 0200 0048 8d84 24f0 ...H..$....H..$. |
| 000119a0: 0200 0048 39c7 7405 e893 e6ff ff48 8bbc
| ...H9.t......H.. | 000119b0: 2480 0200 0048 8d84 2498 0200
| 0048 39c7 $....H..$....H9. | 000119c0: 7405 e879 e6ff ff48
| 8bbc 2468 0100 004c t..y...H..$h...L | Please tell me what
| this code does, Sheldon
| tetromino_ wrote:
| You're right - the two exploit files are lzma-compressed
| and then deliberately corrupted using `tr`, so a hex dump
| wouldn't show anything immediately suspicious to a
| reviewer.
|
| Mea culpa!
| maxcoder4 wrote:
| Is this lzma compressed? Hard to tell because of the lack
| of formatting, but this looks like amd64 shellcode to me.
|
| But that's not really important to the point - I'm not
| looking at a diff of every committed favicon.ico or ttf
| font or a binary test file to make sure it doesn't
| contain a shellcode.
| ptx wrote:
| The use of "eval" stands out, or at least it _should_ stand out
| - but there are two more instances of it in the same script,
| which presumably are not used maliciously.
|
| A while back there was a discussion[0] of an arbitrary code
| execution vulnerability in exiftool which was also the result
| of "eval".
|
| Avoiding casual use of this overpowered footgun might make it
| easier to spot malicious backdoors. Usually there is a better
| way to do it in almost all cases where people feel the need to
| reach for "eval", unless the feature you're implementing really
| is "take a piece of arbitrary code from the user and execute
| it".
|
| [0] https://news.ycombinator.com/item?id=39154825
| bonzini wrote:
| Unfortunately eval in a shell script has an effect on the
| semantics but is not necessary to do some kind of parsing of
| the contents of a variable, unlike Python or Perl or
| JavaScript. A $goo
|
| line (without quotes) will already do word splitting, though
| it won't do another layer of variable expansion and
| unquoting, for which you'll need eval
| "$goo"
|
| (This time with quotes).
| jwilk wrote:
| eval in autoconf macros is nothing unusual.
|
| In (pre-backdoor) xz 5.4.5: $ grep -wl eval
| m4/* m4/gettext.m4 m4/lib-link.m4 m4/lib-
| prefix.m4 m4/libtool.m4
| youainti wrote:
| Summary: "The upstream xz repository and the xz tarballs have
| been backdoored."
|
| It is known to be in version 5.6.0 and 5.6.1, and the obfuscated
| code is found in the test directory.
| buildbot wrote:
| This potentially could be a full automated rootkit type breach
| right? Great - is any system with 5.6.1 possibly vulnerable?
|
| Also super weird a contributor thought they could slip this in
| and not have it be noticed at some point. It may point to burning
| that person (aka, they go to jail) for whatever they achieved
| with this. (And whoever they are...)
| q3k wrote:
| NixOS/Pkgs 23.11 unaffected, unstable contains backdoored
| implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not
| seem to link against systemd/liblzma, and the backdoor doesn't
| get configured in (only happens on .deb/.rpm systems).
| jchw wrote:
| It may not have really mattered much for NixOS:
|
| > b) argv[0] needs to be /usr/sbin/sshd
|
| For once, the lack of FHS interoperability is a benefit, if
| only on accident.
| q3k wrote:
| Right, but in this case it's not even compiled it, which is
| arguably better than compiled in but assumed dormant :) (at
| least until someone actually does a full analysis of the
| payload).
| o11c wrote:
| Note that NixOS has a unique advantage in that `dlopen` is
| easier to analyze, but you do have to check for it. A lot of
| people are looking only at `ldd` and missing that they can be
| vulnerable at runtime.
| dandanua wrote:
| That's one of the advantages of NixOS - viruses and mass hacks
| have lesser chance to function due to how different this OS is.
| Until it gets more popular, of course.
| rany_ wrote:
| It's actually not an advantage. The reason why the exploit
| wasn't included is because the attacker specifically decided
| to only inject x86_64 Debian and RHEL to reduce the chances
| of this getting detected.
| AdmiralAsshat wrote:
| > Red Hat assigned this issue CVE-2024-3094.
|
| Does that mean this affects RHEL and Fedora?
| formerly_proven wrote:
| RHEL no, Fedora 41 and Rawhide yes.
|
| https://www.redhat.com/en/blog/urgent-security-alert-fedora-...
|
| https://lists.debian.org/debian-security-announce/2024/msg00...
| dralley wrote:
| Note that Fedora _40_ isn 't even released yet, it's in beta,
| Fedora 41 / rawhide is basically a development branch used
| only by a small number of people.
| jethro_tell wrote:
| RHEL won't get this bug for 2 years =)
| richardwhiuk wrote:
| Red Hat helps to do the job of making sure OSS has CVEs so
| there's common vernacular for the problem.
| lpapez wrote:
| So many security companies publishing daily generic blog posts
| about "serious supply chain compromises" in various distros on
| packages with 0 downloads, and yet it takes a developer debugging
| performance issues to find an actual compromise.
|
| I worked in the software supply chain field and cannot resist
| feeling the entire point of that industry is to make companies
| pay for a security certificate so you can shift the blame onto
| someone else when things go wrong.
| r0ckarong wrote:
| > cannot resist feeling the entire point of that industry is to
| make companies pay for a security certificate so you can shift
| the blame onto someone else when things go wrong.
|
| That's the entire point. You did everything you could by
| getting someone else look at it and saying it's fine.
| bawolff wrote:
| > the entire point of that industry is to make companies pay
| for a security certificate so you can shift the blame onto
| someone else when things go wrong.
|
| That is actually a major point of a lot of corporate security
| measures (shifting risk)
| keepamovin wrote:
| If you installed xz on macOS using brew, then you have
| xz (XZ Utils) 5.6.1 liblzma 5.6.1
|
| which are within the release target for the vuln. As elsewhere
| in these comments, people say macOS effect is uncertain. If
| concerned you can revert to 5.4.6 with brew
| upgrade xz
| neodypsis wrote:
| It's been reverted now: https://github.com/Homebrew/homebrew-
| core/blob/9a0603b474804...
| pjl wrote:
| Similarly if you're using MacPorts, make sure to sync and
| upgrade xz if you have it installed.
|
| 5.6.1 was available for a few days and just rolled back ~20
| minutes ago: https://github.com/macports/macports-
| ports/commit/a1388aee09...
| CableNinja wrote:
| Thats basically the whole point actually... A company pays for
| insurance for the business. The insurance company says sure we
| will insure you, but you need to go through audits A B and C,
| and you need certifications X and Y to be insured by us. Those
| audits are often industry dependent, mostly for topics like
| HIPAA, PCI, SOC, etc.
|
| Insurance company hears about supply chain attacks. Declares
| that insured must have supply chain validation. Company goes
| and gets a shiny cert.
|
| Now when things go wrong, the company can point to the cert and
| go "it wasnt us, see we have the cert you told us to get and
| its up to date". And the company gets to wash their hands of
| liability (most of the time).
| markus_zhang wrote:
| That's the entire point of certification, and any certification
| at all. Certification does not guarantee performance. Actually,
| I would always cast a suspect glance to anyone who is FOCUSED
| on getting certification after certification without any side
| project.
| fourfour3 wrote:
| Looks like Arch Linux shipped both compromised versions - and
| 5.6.1-2 is out to hopefully resolve it.
| tutfbhuf wrote:
| I upgraded Arch Linux on my server a few hours ago. Arch Linux
| does not fetch one of the compromised tarballs but builds from
| source and sshd does not link against liblzma on Arch.
| [root@archlinux ~]# pacman -Qi xz | head -n2 Name
| : xz Version : 5.6.1-2 [root@archlinux
| ~]# pacman -Qi openssh | head -n2 Name :
| openssh Version : 9.7p1-1 [root@archlinux
| ~]# ldd $(which sshd) | grep liblzma [root@archlinux ~]#
|
| It seems that Arch Linux is not affected.
| gpm wrote:
| 5.6.1-1 was built from what I understand to be one of the
| affected tarballs. This was patched in 5.6.1-2: https://gitla
| b.archlinux.org/archlinux/packaging/packages/xz...
|
| I agree on the sshd linking part.
| tutfbhuf wrote:
| Interesting, they just switched from tarballs to source 19
| hours ago. It seems to me that Frederik Schwan had prior
| knowledge of the security issue, or it is just a rare
| coincidence.
| ComputerGuru wrote:
| Distributions were notified under embargo.
| gpm wrote:
| On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I
| think it's unaffected? Obviously still not great to be shipping
| malicious code that just happens to not trigger.
| fullstop wrote:
| My Arch setup is the same, they must not patch openssh.
| altairprime wrote:
| Deleted per below
| gpm wrote:
| This is what the `detect_sh.bin` attached to the email
| does. I can only assume that the pesron who reported the
| vulnerability checked that this succeeds in detecting it.
|
| Note that I'm not looking for the vulnerable symbols, I'm
| looking for the library that does the patching in the first
| place.
| altairprime wrote:
| Deleted, thanks.
| Macha wrote:
| 5.6.1-2 is not an attempted fix, it's just some tweaks to
| Arch's own build script to improve reproducibility. Arch's
| build script ultimately delegates to the compromised build
| script unfortunately, but it also appears the payload itself is
| specifically targeting deb/RPM based distros, so a narrow miss
| for Arch here.
|
| (EDIT: as others have pointed out, part of the exploit is in
| the artifact from libxz, which Arch is now avoiding by
| switching to building from a git checkout)
| gpm wrote:
| Are you sure about that? The diff moves away from using the
| compromised tarballs to the not-compromised (by this) git
| source. The comment message says it's about reproducibility,
| but especially combined with the timing it looks to me like
| that was just to avoid breaking an embargo.
| tutfbhuf wrote:
| So, you suggest that Frederik Schwan had prior knowledge of
| the security issues but hid the real purpose of the commit
| under "improve reproducibility"?
| gpm wrote:
| Yes.
|
| I've never had to do it myself but I believe that's
| common practice with embargos on security
| vulnerabilities.
| bombcar wrote:
| It can lead to amusing cases where the intentional vuln
| comes in "to improve x" and the quiet fix comes in "to
| improve x".
| jethro_tell wrote:
| And, If you break the embargo too many times then you
| just find out with the rest of us and that's not a great
| way to run a distro. I believe openbsd is or was in that
| position around the time of the intel speculative
| execution bugs.
| Starlevel004 wrote:
| xz was masked in the Gentoo repositories earlier today
| with the stated reason of "Investigating serious bug". No
| mention of security. It's pretty likely.
| donio wrote:
| 5.6.1 is masked specifically.
|
| Also, https://mastodon.social/@mgorny@treehouse.systems/1
| 121802382... from a Gentoo dev mentions that Gentoo
| doesn't use the patch that results in sshd getting linked
| against liblzma.
|
| As far as I know this is not an official communication
| channel so don't take it as such.
| NekkoDroid wrote:
| This is very likely the case. Arch maintainers do get
| early information on CVEs just like any other major
| distro.
|
| But with pacman/makepkg 6.1 (which recently released) git
| sources can also now be check summed IIRC which is a
| funny coincidence.
| mook wrote:
| The writeup indicates that the backdoor only gets applied when
| building for rpm or deb, so Arch probably would have been okay
| either way? Same with Nix, Homebrew, etc.
| aquova wrote:
| The project has made an official post on the subject
|
| https://archlinux.org/news/the-xz-package-has-been-backdoore...
| bawolff wrote:
| The terrifying part is that this was primarily found because the
| backdoor was poorly made and causing performance problems.
|
| Makes you wonder what more competent actors can do.
| rwmj wrote:
| I've analysed the backdoor myself and it's very sophisticated,
| not poorly made at all. The performance problem is surprising
| in this context, but I think next time they won't make that
| mistake.
| Nextgrid wrote:
| Do you have a writeup or any details as to what it does? The
| logical thing based on this post is that it hooks the SSH key
| verification mechanism to silently allow some attacker-
| controlled keys but I wonder if there's more to it?
| rwmj wrote:
| I was starting one, but the openwall message linked here is
| far more detailed and gets much further than I did. It's
| fiendishly difficult to follow the exploit.
| bawolff wrote:
| I guess it seems like the operational parts are a bit poorly
| done. Valgrind issues, adding a new version with symbols
| removed, the aforementioned performance issues. Like i would
| assume the type of person who would do this sort of thing,
| over a 2 year period no less, would test extensively and be
| sure all their i's are dotted. Its all kind of surprising
| given how audacious the attack is.
| dist-epoch wrote:
| There are so many variations of Linux/FreeBSD and weird
| setups and environments that it's almost guaranteed that
| you'll hit a snag somewhere if you do any major
| modification like inserting a backdoor.
| bombcar wrote:
| It's hard enough to get code to work correctly; getting
| it to be also doing something else is even harder.
|
| The way they went around it, however, was brilliant.
| Completely reduce the variables to directly target
| whatever it is you're attacking. Reminds me of stuxnet
| somewhat.
| SkiFire13 wrote:
| Note that in this case the backdoor was only inserted in
| some tarballs and enabled itself only when building
| deb/rpm packages for x86-64 linux and with gcc and the
| gnu linker. This should already filter out the most
| exotic setups and makes it harder to reproduce.
| rdtsc wrote:
| But they almost got away with it. We could have found
| ourselves 5 years later with this code in all stable
| distribution versions, IoT devices etc.
|
| Also, we only catch the ones that we ... catch. The ones
| that do everything perfectly, unless they come out and
| confess eventually, we don't get to "praise" them for their
| impeccable work.
| pinko wrote:
| s/can do/have done/
| aidenn0 wrote:
| So many malicious actors have been caught because they
| accidentally created a mild annoyance for someone that went on
| to bird-dog the problem.
| cpach wrote:
| Case in point: https://news.ycombinator.com/item?id=39843930
| londons_explore wrote:
| Which is why a really good backdoor is a one line logic bug
| somewhere which is fiendishly difficult to trigger.
| bombcar wrote:
| http://underhanded-c.org if people want examples of what
| could (and probably, somewhere, IS) being done.
| tialaramex wrote:
| Sure, however the problem that software is really hard
| _also_ impacts bad actors. So it 's probably at least as
| hard to write that one line logic bug _and have it do
| exactly what you intended_ as to write equivalent real code
| that works precisely as intended.
| Lammy wrote:
| Like the 2003 Linux kernel attempt
| https://lwn.net/Articles/57135/
| rwmj wrote:
| Very annoying - the apparent author of the backdoor was in
| communication with me over several weeks trying to get xz 5.6.x
| added to Fedora 40 & 41 because of it's "great new features". We
| even worked with him to fix the valgrind issue (which it turns
| out now was caused by the backdoor he had added). We had to race
| last night to fix the problem after an inadvertent break of the
| embargo.
|
| He has been part of the xz project for 2 years, adding all sorts
| of binary test files, and to be honest with this level of
| sophistication I would be suspicious of even older versions of xz
| until proven otherwise.
| AdmiralAsshat wrote:
| Yikes! Do you have any info on the individual's background or
| possible motivations?
| bbarnett wrote:
| Yikes indeed. This fix is being rolled out very fast, but
| what about the entire rest of the codebase? And scripts? I
| mean, years of access? I'd trust no aspect of this code until
| a full audit is done, at least of every patch this author
| contributed.
|
| (note: not referring to fedora here, a current fix is
| required. But just generally. As in, everyone is rolling out
| this fix, but... I mean, this codebase is poison in my eyes
| without a solid audit)
| bbarnett wrote:
| This seems to be the account, correct me if wrong (linked
| from the security email commit link):
|
| https://github.com/JiaT75
|
| I hope authors of all these projects have been alerted.
|
| STest - Unit testing framework for C/C++. Easy to use by
| simply dropping stest.c and stest.h into your project!
|
| libarchive/libarchive - Multi-format archive and
| compression library
|
| Seatest - Simple C based Unit Testing
|
| Everything this account has done should be investigated.
|
| Woha, is this legit or some sort of scam on Google in some
| way?:
|
| https://github.com/google/oss-fuzz/pull/11587
|
| edit: I have to be missing something, or I'm confused. The
| above author seems to be primary contact for xz? Have they
| just taken over?? Or did the bad commit come from another
| source, and a legit person applied it?
|
| A bit confused here.
| tux3 wrote:
| The concern about other projects is fine, but let's be
| careful with attacks directed at the person.
|
| Maybe their account is compromised, maybe the username
| borrows the identity of an innocent person with the same
| name.
|
| Focus on the code, not people. No point forming a mob.
|
| (e: post above was edited and is no longer directed at
| the person. thanks for the edit.)
| simiones wrote:
| It's important to focus on people, not just code, when
| suspecting an adversary. Now, I have no idea if this is
| the right account, and if it has recently been
| compromised/sold/lost, or if it has always been under the
| ownership of the person who committed the backdoor. But
| IF this is indeed the right account, then it's important
| to block any further commit from it to any project, no
| matter how innocuous it seems, and to review thoroughly
| any past commit. For the most security-conscious
| projects, it would be a good idea to even consider
| reverting and re-implementing any work coming from this
| account if it's not fully understood.
|
| An account that has introduced a backdoor is not the same
| thing as an account who committed a bug.
| tux3 wrote:
| I agree we should look at the account and its
| contributions, I make a distinction between the account
| and the person.
|
| Sometimes the distinction is not meaningful, but better
| safe than sorry.
| tsimionescu wrote:
| Oh, agreed then.
| kevin_b_er wrote:
| They appear to have moved carefully to set this up over
| the course of weeks by setting up the framework to
| perform this attack.
|
| I would now presume this person to be a hostile actor and
| their contributions anywhere and everywhere must be
| audited. I would not wait for them to cry 'but my bother
| did it', because an actual malicious actor would say the
| same thing. The 'mob' should be pouring over everything
| they've touched.
|
| Audit now and audit aggressively.
| bbarnett wrote:
| My above post shows the primary domain for xz moving from
| tukaani.org to xz.tukaani.org. While it's hosted on
| github:
|
| $ host xz.tukaani.org
|
| _host xz.tukaani.org is an alias for tukaani-
| project.github.io._
|
| And originally it was not:
|
| $ host tukaani.org
|
| _tukaani.org has address 5.44.245.25_ (seemingly in
| Finland)
|
| It was moved there in Jan of this year, as per the commit
| listed in my prior post. By this same person/account.
| This means that instead of Lasse Collin's more
| restrictive webpage, an account directly under the
| control of the untrusted account, is now able to edit the
| webpage without anyone else's involvement.
|
| For example, to make subtle changes in where to report
| security issues to, and so on.
|
| So far I don't see anything nefarious, but at the same
| time, isn't this the domain/page hosting bad tarballs
| too?
| buildbot wrote:
| > tukaani.org has address 5.44.245.25 (seemingly in
| Finland)
|
| Hetzner?
| TimWolla wrote:
| No: route: 5.44.240.0/21
| descr: Zoner Oy origin: AS201692
| mnt-by: MNT-ZONER created:
| 2014-09-03T08:09:00Z last-modified:
| 2014-09-03T08:09:00Z source: RIPE
| buildbot wrote:
| Interesting, seems to be a tiny finnish hosting company:
| https://www.zoner.fi/english/
| whizzter wrote:
| It's Finnish, Oy is short for "Osake Yhtio" (share-
| association, basically a LLC), seems to be
| registered/hosted at https://www.zoner.fi/
| yencabulator wrote:
| For what it's worth, tukaani is how you spell toucan (the
| bird) in Finnish, and Lasse is a common Finnish name; the
| site being previously hosted in Finland is very
| plausible.
| Stagnant wrote:
| Yeah according to their website[0] it looks like majority
| of the past contributors were Finnish so nothing odd
| about the hosting provider. On the same page it says that
| Jia Tan became co-maintainer of xz in 2022.
|
| 0: https://tukaani.org/about.html
| pja wrote:
| This account changed the instructions for reporting
| security issues in the xz github as their very last
| commit: commit
| af071ef7702debef4f1d324616a0137a5001c14c (HEAD -> master,
| origin/master, origin/HEAD) Author: Jia Tan
| <jiat0218@gmail.com> Date: Tue Mar 26 01:50:02
| 2024 +0800 Docs: Simplify SECURITY.md.
| diff --git a/.github/SECURITY.md b/.github/SECURITY.md
| index e9b3458a..9ddfe8e9 100644 ---
| a/.github/SECURITY.md +++ b/.github/SECURITY.md
| @@ -16,13 +16,7 @@ the chance that the exploit will be
| used before a patch is released. You may submit
| a report by emailing us at
| [xz@tukaani.org](mailto:xz@tukaani.org), or through
| [Security Advisories](https://github.com/tukaani-
| project/xz/security/advisories/new). -While both
| options are available, we prefer email. In any case,
| please -provide a clear description of the
| vulnerability including: - -- Affected
| versions of XZ Utils -- Estimated severity (low,
| moderate, high, critical) -- Steps to recreate
| the vulnerability -- All relevant files (core
| dumps, build logs, input files, etc.) +While both
| options are available, we prefer email.
| This project is maintained by a team of volunteers on a
| reasonable-effort basis. As such, please give us
| 90 days to work on a fix before
|
| Seems innocuous, but maybe they were planning further
| changes.
| bombcar wrote:
| > Seems innocuous, but maybe they were planning further
| changes.
|
| Seems like an attempt to get 90 days of "use" of this
| vulnerability after discovery. If they only had checked
| performance before!
| hackernudes wrote:
| No, they just removed the bullet points about what to
| include in a report. The 90 days part was in both
| versions.
| bombcar wrote:
| True, but the "talk only to me" part was new, I think.
| meragrin_ wrote:
| Yes. An incomplete report allows for dragging out
| "fixing" the issue longer.
| mort96 wrote:
| If the owner of the account is innocent and their account
| was compromised, it's on them to come out and say that.
| All signs currently point to the person being a malicious
| actor, so I'll proceed on that assumption.
| londons_explore wrote:
| Does the person exist at all? Maybe they're a persona of
| a team working at some three letter agency...
| bonzini wrote:
| Or for some three letter party.
| Citizen8396 wrote:
| Probably not. I did some pattern of life analysis on
| their email/other identifiers. It looks exactly like when
| I set up a burner online identity- just enough to get
| past platform registration, but they didn't care enough
| to make it look real.
|
| For example, their email is only registered to GitHub and
| Twitter. They haven't even logged into their Google
| account for almost a year. There's also no history of it
| being in any data breaches (because they never use it).
|
| Burn the witch.
| soraminazuki wrote:
| Oh no, not libarchive! GitHub search shows 6 pull
| requests were merged back in 2021.
|
| https://github.com/search?q=repo%3Alibarchive%2Flibarchiv
| e+j...
|
| It does look innocent enough though. Let's hope there's
| no unicode trickery involved...
| steelframe wrote:
| Maybe not. They removed safe_fprintf() here and replaced
| it with the (unsafe) fprintf().
|
| https://github.com/libarchive/libarchive/commit/e37efc16c
| 866...
| buildbot wrote:
| If libarchive is also backdoored, would that allow
| specifically crafted http gzip encoded responses to do
| bad things?
| nicolas_17 wrote:
| What software is using libarchive to decode HTTP
| responses?
| buildbot wrote:
| I don't know, way outside my domain. Possibly none I
| guess?
| giantrobot wrote:
| FreeBSD's archive tools are built on top of libarchive
| IIRC. Not sure about the other BSDs.
| duskwuff wrote:
| No. There's no good reason HTTP response decoding would
| ever be implemented in terms of libarchive; using libz
| directly is simpler and supports some use cases (like
| streaming reads) which libarchive doesn't.
| billyhoffman wrote:
| EDIT: Ahh, I was wrong and missed the addition of
| "strerror"
|
| The PR is pretty devious.
|
| JiaT75 claims is "Added the error text when printing out
| warning and errors in bsdtar when untaring. Previously,
| there were cryptic error messages" and cites this as
| fixing a previous issue.
|
| https://github.com/libarchive/libarchive/pull/1609
|
| However it doesn't actually do that!
|
| The PR literally removes a new line between 2 arguments
| on the first `safe_fprintf()` call, and converts the
| `safe_fprintf()` to unsafe direct calls to `fprintf()`.
| In all cases, the arguments to these functions are
| exactly the same! So it doesn't actually make the error
| messages any different, it doesn't actually solve the
| issue it references. And the maintainer accepted it with
| no comments!
| zb3 wrote:
| But I see the "strerror" call is added
| londons_explore wrote:
| reread it...
|
| It does remove the safe prefixes... But it also adds one
| print statement to "strerror()", which could plausibly
| give better explanations for the error code...
|
| The only suspicious thing here is the lack for safe_
| prefix (and the potential for the strerror() function to
| already be backdoored elsewhere in another commit)
| dchest wrote:
| That seems to be fine. safe_fprintf() takes care of non-
| printable characters. It's used for
| archive_entry_pathname, which can contain them, while
| "unsafe" fprintf is used to print out
| archive_error_string, which is a library-provided error
| string, and strerror(errno) from libc.
| mbauman wrote:
| We know there's long-cons in action here, though. This PR
| needn't be the exploit. It needn't be anywhere
| _temporally_ close to the exploit. It could just be
| laying groundwork for later pull requests by potentially
| different accounts.
| datenwolf wrote:
| Exactly. If we assume the backdoor via liblzma as a
| template, this could be a ploy to hook/detour both
| fprintf and strerror in a similar way. Get it to diffuse
| into systems that rely on libarchive in their package
| managers.
|
| When the trap is in place deploy a crafted package file
| that appears invalid on the surface level triggers this
| trap. In that moment fetch the payload from the (already
| opened) archive file descriptor, execute it, but also
| patch the internal state of libarchive so that it will
| process the rest of the archive file as if nothing
| happened, and the desired outcome also appearing in the
| system.
| zrm wrote:
| Assuming there isn't another commit somewhere modifying a
| library-provided error string or anything returned by
| libc. There is all kinds of mischief to be had there,
| which may or may not have already happened, e.g. now you
| do some i18n and introduce Unicode shenanigans.
| Zetobal wrote:
| That looks like a repo that would sound alarms if you
| look at it from a security standpoint.
| davexunit wrote:
| JiaT75 also has commits in wasmtime according to
| https://hachyderm.io/@joeyh/112180082372196735
| pinko wrote:
| per https://hachyderm.io/@bjorn3/112180226784517099, "The
| only contribution by them to Wasmtime is a doc change. No
| actual code or binary blobs have been changed by them."
| mintplant wrote:
| Just a documentation change, fortunately:
|
| https://github.com/bytecodealliance/wasmtime/commits?auth
| or=...
|
| They've submitted little documentation tweaks to other
| projects, too; for example:
|
| https://learn.microsoft.com/en-us/cpp/overview/whats-new-
| cpp...
|
| I don't know whether this is a formerly-legitimate open
| source contributor who went rogue, or a deep-cover
| persona spreading innocuous-looking documentation changes
| around to other projects as a smokescreen.
| bombcar wrote:
| Minor documentation change PRs is a well known tactic
| used to make your GitHub profile look better (especially
| to potential employers).
|
| He could be doing the same thing for other reasons;
| nobody really digs into anything very deep so I could see
| someone handing over co-maintenance to a project based on
| a decent looking Github graph and some reasonability.
| mysidia wrote:
| Consider the possibility those type of submissions were
| part of the adversary's strategy in order to make their
| account appear more legitimate rather than appearing out
| of nowhere wanting to become the maintainer of some
| project.
| ikmckenz wrote:
| > The above author seems to be primary contact for xz?
|
| They made themselves the primary contact for xz for
| Google oss-fuzz about one year ago:
| https://github.com/google/oss-
| fuzz/commit/6403e93344476972e9...
| bed99 wrote:
| A SourceGraph search like this shows https://sourcegraph.
| com/search?q=context:global+JiaT75&patte...
|
| - Jia Tan <jiat75@gmail.com>
|
| - jiat75 <jiat0218@gmail.com>
|
| ``` amap = generate_author_map("xz")
| test_author = amap.get_author_by_name("Jia Cheong Tan")
| self.assertEqual( test_author.names,
| {"Jia Cheong Tan", "Jia Tan", "jiat75"}
| ) self.assertEqual(
| test_author.mail_addresses,
| {"jiat0218@gmail.com", "jiat75@gmail.com"}
| )
|
| ```
| pryce wrote:
| additionally, even though the commit messages they've
| made are mostly plain, there may be features of their
| commit messages that could provide leads, such as his
| using what looks like a very obscure racist joke of
| referring to a gitignore file as a 'gitnigore'. There's
| barely a handful of people on the whole planet making
| this 'joke'.
| metzmanj wrote:
| >Woha, is this legit or some sort of scam on Google in
| some way?:
|
| I work on OSS-Fuzz.
|
| As far as I can tell, the author's PRs do not compromise
| OSS-Fuzz in any way.
|
| OSS-Fuzz doesn't trust user code for this very reason.
| packetlost wrote:
| It looks more like they disabled a feature of oss-fuzz
| that would've caught the exploit, no?
| metzmanj wrote:
| That's what people are saying though I haven't had the
| chance to look into this myself.
|
| Fuzzing isn't really the best tool for catching bugs the
| maintainer intentionally inserted though.
| bombcar wrote:
| It's more likely that fuzzing would blow up on new code
| and they wanted an excuse to remove it.
|
| After all, if it hadn't had a performance regression
| (someone could submit a PR fixing whatever slowed it
| down, heh) it still wouldn't be known.
| jnxx wrote:
| There is also a variety of new, parallelized
| implementations of compression algorithms which would be
| good to have a close look at. Bugs causing undefined
| behaviour in parallel code are notoriously hard to see,
| and the parallel versions (which are actually much
| faster) could be take the place of well-established
| programs which have earned a lot of trust.
| formerly_proven wrote:
| Well that account also did most of the releases since
| 5.4.0.
| alwaysbeconsing wrote:
| +1 Can see from project homepage http://web.archive.org/w
| eb/20240329165859/https://xz.tukaani... they have some
| release responsibility from 5.2.12.
|
| > Versions 5.2.12, 5.4.3 and later have been signed with
| Jia Tan's OpenPGP key . The older releases have been
| signed with Lasse Collin's OpenPGP key .
|
| It must be assume that before acquiring that privilege,
| they also contributed code to project. Probably most was
| to establish respectable record. Still could be malicious
| code going back someways.
| rwmj wrote:
| I handed over all the emails I received to the security team,
| who I guess will send them "higher". I'll let them analyse
| it.
| 5kg wrote:
| There is zero web presence for this person and associated
| email address.
|
| Looks more likely a fake identity than compromised account.
| Retr0id wrote:
| Did you check Chinese social media?
| mrb wrote:
| Actually the "jiat0218" user part in his email address
| jiat0218@gmail.com has a bunch of matches on Taiwanese
| sites:
|
| https://char.tw/blog/post/24397301
|
| https://forum.babyhome.com.tw/topic/167439
|
| https://bmwcct.com.tw/forums/thread1828.html
| 5kg wrote:
| I think it's just a coincidence.
|
| - All the posts are from 2004/2006. - "jiat" can be
| abbreviation for many common Chinese names.
| mrb wrote:
| I agree, probably a coincidence. Just wanted to point out
| we can, actually, find the username online.
| johnny22 wrote:
| I've never had a web presencse for my associated emails due
| to wanting to avoid spammers. I don't have a false
| identity.
| johnisgood wrote:
| Keep in mind that having a "false identity" does not make
| you a malicious actor. I have a serious project I work on
| under another pseudonym, but it has to do more with the
| fact that I do not want my real name to be associated
| with that project AND having a serious case of impostor
| syndrome. :/
|
| That, and I used to contribute to various games (forks of
| ioquake3) when I was a teen and I wanted to keep my real
| name private.
| stephenr wrote:
| > I don't have a false identity.
|
| That's just what someone with a false identity would
| say.. get him boys!
|
| The biggest /S
| junon wrote:
| This is all I can find on them. carrd.co
| jiat0218@gmail.com business
| https://jiat0218@gmail.com.carrd.co eBay JiaT75
| shopping https://www.ebay.com/usr/JiaT75 giters
| jiat0218 coding https://giters.com/jiat0218 giters
| JiaT75 coding https://giters.com/JiaT75 GitHub
| jiat0218 coding https://github.com/jiat0218 GitHub
| JiaT75 coding https://github.com/JiaT75 Mastodon-
| meow.so.. jiat0218@gmail.com social
| https://meow.social/@jiat0218@gmail.com
|
| Beyond that, nothing surefire. (This is all publicly
| queryable information, if anyone is curious).
| wpietri wrote:
| I get why people are focusing on this bad actor. But the
| question that interests me more: how many other apparent
| individuals fit the profile that this person presented before
| caught?
| mrb wrote:
| I would presume it's a state actor. Generally in the blackhat
| world, attackers have very precise targets. They want to
| attack this company or this group of individuals. But someone
| who backdoors such a core piece of open source infrastructure
| wants to cast a wide net to attack as many as possible. So
| that fits the profile of a government intelligence agency who
| is interested in surveilling, well, everything.
|
| Or it could in theory be malware authors (ransomware, etc).
| However these guys tend to aim at the low hanging fruits.
| They want to make a buck _quickly_. I don 't think they have
| the patience and persistence to infiltrate an open source
| project for 2 long years to finally gain enough trust and
| access to backdoor it. On the other hand, a state actor is in
| for the long term, so they would spend that much time (and
| more) to accomplish that.
|
| So that's my guess: Jia Tan is an employee of some
| intelligence agency. He chose to present an asian persona,
| but that's not necessarily who he truly represents. Could be
| anyone, really: Russia, China, Israel, or even the US, etc.
|
| Edit: given that Lasse Collin was the only maintainer of xz
| utils in 2022 before Jia Tan, I wouldn't be surprised if the
| state actor interfered with Lasse somehow. They could have
| done anything to distract him from the project: introduce a
| mistress in his life, give him a high-paying job, make his
| spouse sick so he has to care for her, etc. With Lasse not
| having as many hours to spend on the project, he would have
| been more likely to give access to a developer who shows up
| around the same time and who is highly motivated to
| contribute code. I would be interested to talk to Lasse to
| understand his circumstances around 2022.
| janc_ wrote:
| Or they have just one or a small number of targets, but
| don't want the target(s) to know that they were the only
| target(s), so they backdoor a large number of victims to
| "hide in the crowd".
|
| I agree that this is likely a state actor, or at least a
| very large & wealthy private actor who can play the long
| game...
| dist-epoch wrote:
| According to top comment he committed multiple binary files
| to xz for the last two years.
|
| Most likely this is not the first backdoor, just the first
| one to be discovered, so it wasn't two years of work until
| there were results.
|
| But I still agree that he's probably a state actor.
| bombcar wrote:
| Don't forget that you could have state actors who are
| otherwise interested in open source code, and working to
| actually improve it.
|
| In fact, that'd be the best form of deep cover. It'll be
| interested to watch as people more knowledgable than I
| pour over every single commit and change.
| fullstop wrote:
| If anyone here happens to know Lasse, it might be good to
| check up on him and see how he's doing.
| Delk wrote:
| > I wouldn't be surprised if the state actor interfered
| with Lasse somehow
|
| People could also just get tired after years of active
| maintainership or become busier with life. Being the sole
| maintainer of an active open source project on top of work
| and perhaps family takes either a lot of enthusiasm or a
| lot of commitment. It's not really a given that people want
| to (or can) keep doing that forever at the same pace.
|
| Someone then spots the opportunity.
|
| I have no idea what the story is here but it might be
| something rather mundane.
| hk__2 wrote:
| > I haven't lost interest but my ability to care has been
| fairly limited mostly due to longterm mental health issues
| but also due to some other things. Recently I've worked
| off-list a bit with Jia Tan on XZ Utils and perhaps he will
| have a bigger role in the future, we'll see.
|
| https://www.mail-archive.com/xz-
| devel@tukaani.org/msg00567.h...
| mrb wrote:
| Dated June 2022. Good find!
| jnxx wrote:
| > They want to attack this company or this group of
| individuals. But someone who backdoors such a core piece of
| open source infrastructure wants to cast a wide net to
| attack as many as possible.
|
| The stuxnet malware, which compromised Siemens industrial
| controls to attack specific centrifuges in uranium
| enrichment plants in Iran, is a counterexample to that.
| 2OEH8eoCRo0 wrote:
| It's ridiculous to think it's the US as it would be an
| attack on Red Hat a US company and an attack on Americans.
| It's a good way to be dragged in front of Congress.
| guinea-unicorn wrote:
| The US has backdoored RSA's RNG and thus endangered the
| security of American companies. It is naive to think that
| US intelligence agencies will act in the best interest of
| US citizens or companies.
| 2OEH8eoCRo0 wrote:
| That is speculation and has never been confirmed.
| occamsrazorwit wrote:
| What type of confirmation do you want? The documents
| aren't going to be declassified in the next couple of
| decades, if ever.
|
| I've never heard anyone claim that Dual_EC_DRBG is most
| likely _not_ intentionally backdoored, but there 's
| literally no way to confirm because of how its written.
| If we can't analyze intention from the code, we can look
| at the broader context for clues. The NSA spent an
| unusual amount of effort trying to push forward an
| algorithm that kept getting shot down because it was
| slower than similar algorithms with no additional
| benefits (the $10 million deal specified it as a
| requirement [1]). If you give the NSA the benefit of the
| doubt, they spent a lot of time and money to...
| intentionally slow down random number generation?!
|
| As an American, I'd prefer a competent NSA than an
| incompetent NSA that spends my tax dollars to make
| technology worse for literally no benefit...
|
| [1] https://www.reuters.com/article/us-usa-security-rsa-
| idUSBRE9...
| hex4def6 wrote:
| You are understating the level of evidence that points to
| the NSA being fully aware of what it was doing.
|
| To be clear, the method of attack was something that had
| been described in a paper years earlier, the NSA
| literally had a program (BULLRUN) around compromising and
| attacking encryption, and there were security researchers
| at NIST and other places that raised concerns even before
| it was implemented as a standard. Oh, and the NSA paid
| the RSA $10 million to implement it.
|
| Heck, even the chairman of the RSA implies they got used
| by the NSA:
|
| In an impassioned speech, Coveillo said RSA, like many in
| industry, has worked with the NSA on projects. But in the
| case of the NSA-developed algorithm which he didn't
| directly name, Coviello told conference attendees that
| RSA feels NSA exploited its position of trust. In its
| job, NSA plays two roles, he pointed out. In the
| information assurance directorate (IAD) arm of NSA, it
| decides on security technologies that might find use in
| the government, especially the military. The other side
| of the NSA is tasked with vacuuming up data for cyber-
| espionage purposes and now is prepared to take an
| offensive role in cyber-attacks and cyberwar.
|
| "We can't be sure which part of the NSA we're working
| with," said Coviello with a tone of anguish. He implied
| that if the NSA induced RSA to include a secret backdoor
| in any RSA product, it happened without RSA's consent or
| awareness.
|
| https://www.networkworld.com/article/687628/security-rsa-
| chi...
| fragmede wrote:
| What about the time it was shown they did the reverse
| (hardened security using math only they knew at the time)
| for DSA
| AimHere wrote:
| Hardly ridiculous.
|
| You say that as if members of US government agencies
| didn't plot terror attacks on Americans (Operation
| Northwood), steal the medical records of American
| whistleblowers (Ellsberg), had to be prevented from
| assassinating American journalists (Gordon Liddy, on Jack
| Anderson), collude to assassinate American political
| activists (Fred Hampton), spy on presidential candidates
| (Watergate), sell weapons to countries who'd allegedly
| supported groups who'd launched suicide bombing attacks
| on American soldiers (Iran-Contra), allow drug smugglers
| to flood the USA with cocaine so that they could supply
| illegal guns to terrorists abroad on their return trip
| (Iran-Contra again) and get caught conducting illegal
| mass-surveillance on American people as a whole
| (Snowden). Among others.
|
| It's super-naive to suggest that government agencies
| wouldn't act against the interest of American citizens
| and companies because there might be consequences if they
| were caught. Most of the instances above actually were
| instances where the perpetrators did get caught, which is
| why we know about them.
| 2OEH8eoCRo0 wrote:
| I love being called naive.
| QuantumG wrote:
| Whisper it to me lover.
| mrb wrote:
| Have you forgotten about the Snowden leaks exposing the
| surveillance on Americans by the American govt?
| occamsrazorwit wrote:
| Given the details from another comment [1], it sounds like
| both maintainers are suspicious. Lasse's behavior has
| changed recently, and he's been pushing to get Jia Tan's
| changes into the Linux kernel. It's possible both accounts
| aren't even run by the original Lasse Collin and Jia Tan
| anymore.
|
| [1] https://news.ycombinator.com/item?id=39865810#39866275
| jpalomaki wrote:
| Seems to be a perfect project to hijack. Not too much
| happening, widely used, long history, single maintainer who
| no longer has time to manage the project and wants to pass it
| over.
| PedroBatista wrote:
| Given the recent ( not so recent ) attacks/"bugs" I feel there
| is a need to do more than the already hard task of
| investigating and detecting attacks but also to bring IRL
| consequences to these people.
|
| My understanding is that right now it's pretty much a name and
| shame of people who most of the time aren't even real "people"
| but hostile agents either working for governments or criminal
| groups ( or both )
|
| Getting punched in the face is actually a necessary human
| condition for a healthy civilization.
| buildbot wrote:
| In the article it says CISA was notified - that sounds like
| it's going to be a federal investigation if nothing else. If
| I was this person, I wouldn't be in the USA (or any US
| friendly nation) ASAP.
| graemep wrote:
| One of Jia Tan's recent contributions is "Speed up CRC32
| calculation on LoongArch" I would guess the odds are that
| this is not someone in the US.
| buildbot wrote:
| Yeah I saw that - I wouldn't bet on them being in the US
| but who knows. Maybe they just really love CRC32 ;) And
| introducing backdoors (if it that was them not an account
| takeover).
| kevin_b_er wrote:
| Those tarballs are PGP signed, too..
| rrix2 wrote:
| That was a review of someone else's work?
| https://github.com/tukaani-project/xz/pull/86
| hangonhn wrote:
| It's also very possible that the account was compromised
| and taken over. A two years long con with real useful
| work is a lot of patience and effort vs. just stealing a
| weakly protected account. I wonder if MFA shouldn't be a
| requirement for accounts that contribute to important OSS
| projects.
| dist-epoch wrote:
| This is most likely not his first backdoor, but the first
| which was detected.
|
| So most likely he didn't wait two years to benefit.
| dralley wrote:
| >A two years long con with real useful work is a lot of
| patience and effort vs. just stealing a weakly protected
| account.
|
| The long-con theory seems a bit more plausible at the
| moment
|
| https://github.com/google/oss-fuzz/pull/10667
| IncreasePosts wrote:
| It might not even be a long time. He might have just been
| approached exactly because of his history to insert the
| back door. And offered either money, or blackmailed or
| threatened
| hangonhn wrote:
| Oh man. The was a scenario that didn't cross my mind. I
| was too narrowly focused on the technical aspects rather
| than the social aspects of security. Great point.
| wyldberry wrote:
| If you really step back and think about it, this type of
| behavior is perfectly aligned with any number of well
| resourced criminal groups and state actors. Two years of
| contributing in less visible software with the goal of
| gaining trust and then slowly pushing your broken fix in.
|
| To me that's way more plausible than losing control of
| your account and the person who compromised it then
| having someone over a long time insert the backdoor that
| took a long time to develop and then obfuscate it.
|
| Likely someone at GH is talking to some government
| agencies right now about the behavior of the private
| repos of that user and their associated users.
| sonicanatidae wrote:
| This would be the smarter attack vector, but I've noticed
| over time that these people are just assholes. They
| aren't patient. They are in for the smash/grab.
|
| I would not be surprised if there was a group using this
| approach, but I doubt most of them are/would. If they
| were that dedicated, they'd just have a fucking job,
| instead of being dicks on the internet for a living.
| Phenylacetyl wrote:
| But what if being a dick on the internet _is_ the job
| they do for a living?
| cjbprime wrote:
| I think you are confusing non-state e.g. ransomware
| groups, which are usually not part of a government
| (although some exceptions like North Korea likely exist)
| with state-sponsored hackers who are often directly
| working under military command. Soldiers are not "dicks
| on the internet".
| willdr wrote:
| As someone who has been in a fair few discord chats with
| soldiers, I'd beg to differ...
| wyldberry wrote:
| For some groups they certainly are.
|
| However at this point: every developed nation has a
| professional offensive security group that have varying
| degrees of potency. All are more resourced than 99.9% of
| organizations defending and enjoy legal autonomy in their
| country and allied countries for their work.
|
| If you're getting salaried comfortably, and you have near
| infinite resources, a two year timeline is trivial. As an
| American, I always like to point to things we know our
| own services have done first[0].
|
| Each actor group have their own motivations and
| tactics[1]. As someone who spent a lot of time dealing
| with a few state actors, you learn your adversaries
| tricks of the trade and they are patient for the long-con
| because they can afford to be.
|
| [0] -
| https://www.npr.org/2020/03/05/812499752/uncovering-the-
| cias... [1] - https://learn.microsoft.com/en-
| us/microsoft-365/security/def...
| jnxx wrote:
| I am thinking more in so-called rubberhose
| cryptoanalysis.
|
| https://xkcd.com/538/
| ranger_danger wrote:
| > It's also very possible that the account was
| compromised and taken over
|
| Or they WERE legit and simply went rogue, perhaps due to
| external factors.
| fmajid wrote:
| 2 years of one engineer's time is very cheap, compared to
| e.g. the NSA's CryptoAG scam. I'd say most likely a
| Chinese intelligence plant, _kindly_ offering to relieve
| the burden of the original author of xz.
| rdtsc wrote:
| I got the same idea. On XZ dev mailing list there were a
| few discussions about "is there a maintainer?" 2-3 years
| ago. It's not hard to find these types discussions and
| then dedicate a few years of effort to start "helping
| out" and eventually be the one signing releases for the
| project. That's peanuts for a state actor.
| dreamingincode wrote:
| The full name "Jia Cheong Tan" doesn't sound like
| Mainland China. The name and actions could be
| intentionally misleading though.
|
| https://news.ycombinator.com/item?id=39867737
| viraptor wrote:
| We're way too global now for this to be more than a tiny
| extra signal. People move around, families preserve
| names.
|
| Also nobody checked that person's id, so "Jia" is only
| slightly more meaningful than "ghrssbitrvii".
| graemep wrote:
| Names can be faked, and even real names are not a great
| indicator.
|
| Unless you have some very specific cultural knowledge you
| could not make even vaguely useful deductions about my
| location, nationality, culture, ethnicity etc. from my
| name. I get a lot of wrong guesses though!
| sneak wrote:
| What law do you think is being broken here?
| AlexCoventry wrote:
| Maybe https://www.law.cornell.edu/uscode/text/18/1030#a_5
| ?
|
| > knowingly causes the transmission of a program,
| information, code, or command, and as a result of such
| conduct, intentionally causes damage without
| authorization, to a protected computer;
| jethro_tell wrote:
| How does posting an exploit POC differ here?
| schlauerfox wrote:
| Intent. It's a big part of law and prosecution.
| cjbprime wrote:
| No, freedom of speech (as far as I know) protects even
| exploit code. The statutes being linked would cover
| _using_ the backdoor to gain unauthorized entry to a
| system. I think the question of whether anything illegal
| has occurred from the public facts is unclear, at least
| to me, and interesting.
| jethro_tell wrote:
| I see a dev on the project has just posted that it has
| been seen in the wild, so I guess you'd have standing
| there.
|
| https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee
| 78b...
| wtallis wrote:
| The first amendment might overrule the cited law if that
| law didn't already include a requirement for intentional
| harm. But since the law _does_ already have that
| requirement, there 's not really an opportunity for a
| freedom of speech justification to be what protects a
| non-malicious publication of a proof of concept. The law
| isn't trying to infringe on freedom of speech.
| computerfriend wrote:
| From their Git commits, they're in China's time zone.
| stephenr wrote:
| I assume you mean UTC+8... that covers about 20% of the
| earth's population, besides China it includes parts of
| Russia, a bunch of SEA and Western Australia.
| fmajid wrote:
| China _is_ 20 & of the world's population...
| jcgrillo wrote:
| My git commits are sometimes in UTC, depending on which
| computer I make them from. Sometimes my laptop just
| switches timezones depending on whether I'm using wifi or
| LTE. I wouldn't put much weight on the timezone.
| jnxx wrote:
| The time stamp of a git commit depends on the system
| clock of the computer the commit was checked in. This
| cannot be checked by github & co (except that they could
| reject commits which have time stamps in the future).
| rubymamis wrote:
| We shouldn't rule out the probability that this account is
| from a U.S. agency as well.
| wannacboatmovie wrote:
| Just so I understand, you're alleging that a U.S. agency
| was, among other things, submitting patches for a
| mainland Chinese home-grown CPU architecture (Loongson)?
| yorwba wrote:
| Aren't you confusing JiaT75 and xry111?
|
| And if someone wanted to attack a target running on
| Loongson, they would certainly have to make sure the code
| can actually run there in the first place.
| boutique wrote:
| No, they're not. They are saying that due to the
| extraordinary circumstances with this case US agencies
| cannot be excluded from suspicion. At this time no actor
| seems to be a more likely perpetrator than the next.
| (Keep in mind that false-flag operations are a very
| common occurrence in cyber warfare and this cannot be
| ruled out yet.)
| FLT8 wrote:
| It doesn't seem out of the question that the U.S. or
| allied nations might want to be involved in the
| development effort around these CPUs. Even if initially
| it's just to build some credibility for this account so
| future adversarial patches are accepted with less
| suspicion? If you think that's implausible, I'm
| interested why?
| EthanHeilman wrote:
| We shouldn't rule it out, but it seems unlikely to me.
|
| This is more reckless than any backdoor I can think of by
| a US agency . NSA backdoored Dual EC DRBG, which was
| extremely reckless, but this makes that look careful and
| that was the Zenith of NSA recklessness. The attackers
| here straight up just cowboy'd the joint. I can't think
| of any instance in which US intelligence used sock
| puppets on public forums and mailinglists to encourage
| deployment of the backdoored software and I maintain a
| list of NSA backdoors:
| https://www.ethanheilman.com/x/12/index.html
|
| It just doesn't seem like their style.
| offmycloud wrote:
| CISA Advisory: https://www.cisa.gov/news-
| events/alerts/2024/03/29/reported-...
|
| Note that it say "Fedora 41" in the CISA page link to Red
| Hat, but Red Hat changed the blog title to "Fedora 40" and
| left the HTML page title as "Fedora 41".
| weinberg wrote:
| And I bet if it ended up on a NATO system, things escalate
| quickly for the person / nation states being scrutinized
| (https://www.nato.int/cps/en/natohq/topics_78170.htm)
| oceanplexian wrote:
| A federal investigation into what, itself? The primary
| actors doing this type of thing are the US Government.
| progbits wrote:
| > Getting punched in the face is actually a necessary human
| condition for a healthy civilization.
|
| Aside from signed commits, we need to bring back GPG key
| parties and web of trust. When using a project you would know
| how many punches away from the committers you are.
| woodruffw wrote:
| PGP is more famous for "web of trust" topologies, not
| chains of trust.
|
| For all of their nerd cred, key parties didn't accomplish
| very much (as evidenced by the fact that nothing on the
| Internet really broke when the WoT imploded a few years
| ago[1]). The "real" solution here is mostly cultural:
| treating third-party software like the risky thing it
| actually is, rather than a free source of pre-screened
| labor.
|
| [1]: https://inversegravity.net/2019/web-of-trust-dead/
| progbits wrote:
| Chain/web was typo, corrected, thanks.
|
| I know of the key party issues. But there is some value
| to knowing how far removed from me and people I trust the
| project authors are.
| woodruffw wrote:
| > But there is some value to knowing how far removed from
| me and people I trust the project authors are
|
| That's true!
| weinzierl wrote:
| Yes, but there was also little pressure to really build
| the WOT. People, like myself, did it because it was fun,
| but no one really relied on it. This _could_ change, but
| it is still far from certain if it 'd work given enough
| pressure.
| EthanHeilman wrote:
| The web of punches?
| move-on-by wrote:
| Fascinating. Just yesterday the author added a `SECURITY.md`
| file to the `xz-java` project.
|
| > If you discover a security vulnerability in this project
| please report it privately. *Do not disclose it as a public
| issue.* This gives us time to work with you to fix the issue
| before public exposure, reducing the chance that the exploit
| will be used before a patch is released.
|
| Reading that in a different light, it says give me time to
| adjust my exploits and capitalize on any targets. Makes me
| wonder what other vulns might exist in the author's other
| projects.
| xyst wrote:
| 90 day dark window for maintainers is SOP though. Then after
| 90 days, it's free game for public disclosure
| szundi wrote:
| How many of people like this one exist?
| ldayley wrote:
| If this question had a reliable (and public) answer then
| the world would be a very different place!
|
| That said, this is an important question. We, particularly
| those us who work on critical infrastructure or software,
| should be asking ourselves this regularly to help prevent
| this type of thing.
|
| Note that it's also easy (and similarly catastrophic) to
| swing too far the other way and approach all unknowns with
| automatic paranoia. We live in a world where we have to
| trust strangers every day, and if we lose that option
| completely then our civilization grinds to a halt.
|
| But-- vigilance is warranted. I applaud these engineers who
| followed their instincts and dug into this. They all did us
| a huge service!
|
| EDIT: wording, spelling
| josephg wrote:
| Yeah thanks for saying this; I agree. And as cliche as it
| is to look for a technical solution to a social problem,
| I also think better tools could help a lot here.
|
| The current situation is ridiculous - if I pull in a
| compression library from npm, cargo or Python, why can
| that package interact with my network, make syscalls (as
| me) and read and write files on my computer? Package
| managers should include capability based security. I want
| to say "use this package from cargo, but refuse to
| compile or link into my binary any function which makes
| any syscall except for _read_ and _write_. No _open_ - if
| I want to compress or decompress a file, I'll open the
| file myself and pass it in." No messing with my
| filesystem. No network access. That's all you get.
|
| The capability should be transitive. All dependencies of
| the package should be brought in under the same
| restriction.
|
| In dynamic languages like (server side) JavaScript, I
| think this would have to be handled at runtime. We could
| add a capability parameter to all functions which issue
| syscalls (or do anything else that's security sensitive).
| When the program starts, it gets an "everything"
| capability. That capability can be cloned and reduced to
| just the capabilities needed. (Think, _pledge_ ). If I
| want to talk to redis using a 3rd party library, I pass
| the redis package a capability which only allows it to
| open network connections. And only to this specific host
| on this specific port.
| ncr100 wrote:
| _Security Researchers_ : Is this request-for-private-
| disclosure + "90-days before public" reasonable?
|
| It's a SEVERE issue, to my mind, and 90 days seems too long
| to me.
| bawolff wrote:
| Whether its reasonable is debatable, but that type of time
| frame is pretty normal for things that aren't being
| actively exploited.
|
| This situation is perhaps a little different as its not an
| accidental bug waiting to be discovered but an
| intentionally placed exploit. We know that a malicious
| person already knows about it.
| londons_explore wrote:
| If you were following Google Project Zero's policy (which
| many researchers do), any in-the-wild exploits would
| trigger an immediate reveal.
| larschdk wrote:
| Detecting a security issue is one thing. Detecting a
| malicious payload is something completely different. The
| latter has intent to exploit and must be addressed
| immediately. The former has at least some chance of noone
| knowing about it.
| cjbprime wrote:
| In this particular case, there is a strong reason to expect
| exploitation in the wild to already be occurring (because
| it's an intentional backdoor) and this would change the
| risk calculus around disclosure timelines.
|
| But in the general case, it's normal for 90 days to be
| given for the coordinated patching of even very severe
| vulnerabilities -- you are giving time not just to the
| project maintainers, but to the users of the software to
| finish updating their systems to a new fixed release,
| before enough detail to easily weaponize the vulnerability
| is shared. Google Project Zero is an example of a team with
| many critical impact findings using a 90-day timeline.
| ang_cire wrote:
| As someone in security who doesn't work at a major place
| that get invited to the nice pre-notification
| notifications, I hate this practice.
|
| My customers and business are not any less important or
| valuable than anyone else's, and I should not be left
| being potentially exploited, and my customers harmed, for
| 90 more days while the big guys get to patch their
| systems (thinking of e.g. Log4J, where Amazon, Meta,
| Google, and others were told privately how to fix their
| systems, before others were even though the fix was
| simple).
|
| Likewise, as a customer I should get to know as soon as
| someone's software is found vulnerable, so I can then
| make the choice whether to continue to subject myself to
| the risk of continuing to use it until it gets patched.
| cjbprime wrote:
| OpenSSL's "notification of an upcoming critical release"
| is public, not private.
|
| You do get to know that the vulnerability exists quickly,
| and you could choose to stop using OpenSSL altogether
| (among other mitigations) once that email goes out.
| wyldberry wrote:
| I empathize with this as I've been in the same boat, but
| all entities are not equal when performing triage.
| freedomben wrote:
| Being in a similar boat, I heartily agree.
|
| But I don't want anyone else to get notified immediately
| because the odds that somebody will start exploiting
| people before a patch is available is pretty high. Since
| I can't have both, I will choose the 90 days for the
| project to get patches done and all the packagers to
| include them and make them available, so that by the time
| it's public knowledge I'm already patched.
|
| I think this is a Tragedy of the Commons type of problem.
|
| Caveat: This assume the vuln is found by a white hat. If
| it's being exploited already or is known to others, then
| I fully agree the disclosure time should be eliminated
| and it's BS for the big companies to get more time than
| us.
| oceanplexian wrote:
| Yeah I worked in FAANG when we got the advance notice of
| a number of CVEs. Personally I think it's shady, I don't
| care how big Amazon or Google is, they shouldn't get
| special privileges because they are a large corporation.
| sterlind wrote:
| I think you have to take the credibility of the maintainer
| into account.
|
| If it's a large company, made of people with names and
| faces, with a lot to lose by hacking its users, they're
| unlikely to abuse private disclosure. If it's some tiny
| library, the maintainers might be in on it.
|
| Also, if there's evidence of exploitation in the wild, the
| embargo is a gift to the attacker. The existence of a
| vulnerability in that case should be announced, even if the
| specifics have to be kept under embargo.
| fmajid wrote:
| In this case the maintainer is the one who deliberately
| introduced the backdoor. As Andres Freund puts it
| deadpan, "Given the apparent upstream involvement I have
| not reported an upstream bug."
| decoy78 wrote:
| imho it depends on the vuln. I've given a vendor over a
| year, because it was a very low risk vuln. This isn't a
| vuln though - this is an attack.
| tw04 wrote:
| Honestly it seems like a state-based actor hoping to get
| whatever high value target compromised before it's made
| public. Reporting privately buys them more time, and allows
| them to let handlers know when the jig is up.
| gigatexal wrote:
| Name and shame this author. They should never be allowed
| anywhere near any open projects ever again.
| Lichtso wrote:
| They might have burnt the reputation built for this
| particular pseudonym but what is stopping them from doing it
| again? They were clearly in it for the long run.
| jethro_tell wrote:
| You're assuming that it's even a single person, it's just a
| gmail address and an avatar with a j icon from a clip art
| thing.
| Lichtso wrote:
| I literally said "they", I know, I know, in English that
| can also be interpreted as a gender unspecific singular.
|
| Anyways, yes it is an interesting question whether he/she
| is alone or they are a group. Conway's law probably
| applies here as well. And my hunch in general is that
| these criminal mad minds operate individually / alone.
| Maybe they are hired by an agency but I don't count that
| as a group effort.
| 0xbadcafebee wrote:
| Please don't?
|
| 1. You don't actually know what has been done by whom or why.
| You don't know if the author intended all of this, or if
| their account was compromised. You don't know if someone is
| pretending to be someone else. You don't know if this person
| was being blackmailed, forced against their will, etc. You
| don't really know much of anything, except a backdoor was
| introduced by somebody.
|
| 2. Assuming the author did do something maliciously, relying
| on personal reputation is bad security practice. The majority
| of successful security attacks come from insiders. You have
| to trust insiders, because _someone_ has to get work done,
| and you don 't know who's an insider attacker until they are
| found out. It's therefore a best security practice to limit
| access, provide audit logs, sign artifacts, etc, so you can
| trace back where an incursion happened, identify poisoned
| artifacts, remove them, etc. Just saying "let's ostracize
| Phil and hope this never happens again" doesn't work.
|
| 3. A lot of today's famous and important security researchers
| were, at one time or another, absolute dirtbags who did bad
| things. Human beings are fallible. But human beings can also
| grow and change. Nobody wants to listen to reason or
| compassion when their blood is up, so nobody wants to hear
| this right now. But that's why it needs to be said now. If
| someone is found guilty beyond a reasonable doubt (that's
| really the important part...), then name and shame, sure,
| shame can work wonders. But at some point people need to be
| given another chance.
| gigatexal wrote:
| 100% fair -- we don't know if their account was compromised
| or if they meant to do this intentionally.
|
| If it were me I'd be doing damage control to clear my name
| if my account was hacked and abused in this manner.
|
| Otherwise if I was doing this knowing full well what would
| happen then full, complete defederation of me and my
| ability to contribute to anything ever again should
| commence -- the open source world is too open to such
| attacks where things are developed by people who assume
| good faith actors.
| gigatexal wrote:
| upon further reflection all 3 of your points are cogent
| and fair and valid. my original point was a knee-jerk
| reaction to this. :/
| Biganon wrote:
| Your being able to reflect upon it and analyze your own
| reaction is rare, valuable and appreciated
| yieldcrv wrote:
| I wonder who the target was!
| formerly_proven wrote:
| I think this has been in the making for almost a year. The
| whole ifunc infrastructure was added in June 2023 by Hans
| Jansen and Jia Tan. The initial patch is "authored by" Lasse
| Collin in the git metadata, but the code actually came from
| Hans Jansen: https://github.com/tukaani-
| project/xz/commit/ee44863ae88e377...
|
| > Thanks to Hans Jansen for the original patch.
|
| https://github.com/tukaani-project/xz/pull/53
|
| There were a ton of patches by these two subsequently because
| the ifunc code was breaking with all sorts of build options and
| obviously caused many problems with various sanitizers.
| Subsequently the configure script was modified multiple times
| to detect the use of sanitizers and abort the build unless
| either the sanitizer was disabled or the use of ifuncs was
| disabled. That would've masked the payload in many testing and
| debugging environments.
|
| The hansjans162 Github account was created in 2023 and the only
| thing it did was add this code to liblzma. The same name later
| applied to do a NMU at Debian for the vulnerable version.
| Another "<name><number>" account (which only appears here,
| once) then pops up and asks for the vulnerable version to be
| imported: https://www.mail-archive.com/search?l=debian-bugs-
| dist@lists...
| zb3 wrote:
| Also I saw this hans jansen user pushing for merging the
| 5.6.1 update in debian: https://bugs.debian.org/cgi-
| bin/bugreport.cgi?bug=1067708
| hxelk1 wrote:
| From: krygorin4545 <krygorin4545@proton.me> To:
| "1067708@bugs.debian.org" <1067708@bugs.debian.org> Cc:
| "sebastian@breakpoint.cc" <sebastian@breakpoint.cc>,
| "bage@debian.org" <bage@debian.org> Subject: Re: RFS: xz-
| utils/5.6.1-0.1 [NMU] -- XZ-format compression utilities
| Date: Tue, 26 Mar 2024 19:27:47 +0000
|
| Also seeing this bug. Extra valgrind output causes some
| failed tests for me. Looks like the new version will
| resolve it. Would like this new version so I can continue
| work.
|
| --
|
| Wow.
|
| (Edited for clarity.)
| zb3 wrote:
| Also I see this PR: https://github.com/tukaani-
| project/xz/pull/64
| bluecheese33 wrote:
| > because the ifunc code was breaking with all sorts of build
| options and obviously caused many problems with various
| sanitizers
|
| for example, https://github.com/google/oss-fuzz/pull/10667
| bed99 wrote:
| 1 week ago "Hans Jansen" user "hjansen" was created in debian
| and opened 8 PRs including the upgrade to 5.6.1 to xz-utils
|
| From https://salsa.debian.org/users/hjansen/activity
|
| Author: Hans Jansen <hansjansen162@outlook.com>
|
| - [Debian Games / empire](https://salsa.debian.org/games-
| team/empire): opened merge request "!2 New upstream version
| 1.17" - March 17, 2024
|
| - [Debian Games / empire](https://salsa.debian.org/games-
| team/empire): opened merge request "!1 Update to upstream
| 1.17" - March 17, 2024
|
| - [Debian Games / libretro / libretro-core-
| info](https://salsa.debian.org/games-team/libretro/libretro-
| core-i...): opened merge request "!2 New upstream version
| 1.17.0" - March 17, 2024
|
| - [Debian Games / libretro / libretro-core-
| info](https://salsa.debian.org/games-team/libretro/libretro-
| core-i...): opened merge request "!1 Update to upstream
| 1.17.0" - March 17, 2024
|
| - [Debian Games / endless-
| sky](https://salsa.debian.org/games-team/endless-sky): opened
| merge request "!6 Update upstream branch to 0.10.6" - March
| 17, 2024
|
| - [Debian Games / endless-
| sky](https://salsa.debian.org/games-team/endless-sky): opened
| merge request "!5 Update to upstream 0.10.6" - March 17, 2024
|
| - [Debian / Xz Utils](https://salsa.debian.org/debian/xz-
| utils): opened merge request "!1 Update to upstream 5.6.1" -
| March 17, 2024
| bombcar wrote:
| That looks exactly like what you'd want to see to disguise
| the actual request you want, a number of pointless upstream
| updates in things that are mostly ignored, and then the one
| you want.
| bed99 wrote:
| agree
| amluto wrote:
| Wow, what a big pile of infrastructure for a non-
| optimization.
|
| An internal call via ifunc is not magic -- it's just a call
| via the GOT or PLT, which boils down to function pointers. An
| internal call through a hidden visibility function pointer
| (the right way to do this) is also a function pointer.
|
| The _even better_ solution is a plain old if statement, which
| implements the very very fancy "devirtualization"
| optimization, and the result will be effectively predicted on
| most CPUs and is not subject to the whole pile of issue that
| retpolines are needed to work around.
| jonathanspw wrote:
| Yesterday sure was fun wasn't it :p Thanks for all your
| help/working with me on getting this cleaned up in Fedora.
| pfortuny wrote:
| Unfortunately, this is how _good_ bad actors work: with a very
| long-term point of view. There is no "harmless" project any
| more.
| ametrau wrote:
| Probably a state actor. You can look far into the future when
| you're working for the party.
| DrewRWx wrote:
| And that long term perspective could be used constructively
| instead!
| hangonhn wrote:
| I imagine it might be easier to just compromise a weakly
| protected account than to actual put in a 2 years long effort
| with real contributions. If we mandated MFA for all
| contributors who contribute to these really important
| projects then we can know with greater certainty if it was
| really a long con vs. a recently compromised account.
| __s wrote:
| This seems like a great way to invest in supporting open
| source projects in meantime if these projects are being
| used by these actors. Just have to maintain an internal
| fork without the backdoors
|
| Maybe someone can disrupt the open source funding problem
| by brokering exploit bounties /s
| the8472 wrote:
| github already mandates MFA for members of important
| projects
| rvense wrote:
| Doesn't it mandate it for everyone? I don't use it
| anymore and haven't logged in since forever, but I think
| I got a series of e-mails that it was being made
| mandatory.
| ryukoposting wrote:
| It will soon. I think I have to sort it out before April
| 4. My passwords are already >20 random characters, so I
| wasn't going to do it until they told me to.
| guinea-unicorn wrote:
| I find it funny how MFA is treated as if it would make
| account takeover suddenly impossible. It's just a bit more
| work, isn't it? And a big loss in convenience.
|
| I'd much rather see passwords entirely replaced by key-
| based authentication. _That_ would improve security. Adding
| 2FA to my password is just patching a fundamentally broken
| system.
| hangonhn wrote:
| yeah someone replied to one of my comments about adding
| MFA that an attacker can get around all that simply by
| buying the account from the author. I was way too
| narrowly focused on the technical aspects and was
| completely blind to other avenues like social
| engineering, etc.
|
| All very fair points.
| ryukoposting wrote:
| Customer service at one of my banks has an official
| policy of sending me a verification code via email that I
| then read to them over the phone, and that's not even
| close to the most "wrong" 2FA implementation I've ever
| seen. Somehow that institution knows what a YubiKey is,
| but several major banks don't.
| port443 wrote:
| I think its much more likely this was not a bad actor, given
| their long history of commits.
|
| It's a known fact that China will "recruit" people to operate
| them. A quote:
|
| > They talk to them, say my friend, I see you like our
| special menu. Are you from China? Are you here on a VISA? Do
| you have family back there? Would you like your family to
| stay alive? Is your loyalty to this temporary employer or is
| your loyalty to your motherland? You know, a whole bunch of
| stuff like that. That's how Chinese intelligence operations
| acts...
|
| This just gives feelings of less "compromised account" and
| more "Your account is now our account"
| Johnny555 wrote:
| Isn't that still a "bad actor" even if they are coerced
| into it?
| foobiekr wrote:
| Yes.
| Terr_ wrote:
| For the purposes of security discussions, I would say
| yes.
|
| However if we were critiquing characters in a book--
| especially ones where narrative voice tells us exactly
| their true motivations--then maybe not, and they get
| framed as a "dupe" or "manipulated" etc.
| Almondsetat wrote:
| "bad actor" doesn't mean "bad faith", it's not a value
| judgement
| zeroCalories wrote:
| I think we should seriously consider something like a ts
| clearance as mandatory for work on core technologies. Many
| other projects, both open and closed, are probably
| compromised by foreign agents.
| Meetvelde wrote:
| That's hard to do when the development of these libraries
| is so international. Not to mention that it's already so
| hard to find maintainers for some of these projects.
| Given that getting a TS clearance is such a long and
| difficult process, it would almost guarantee more
| difficulty in finding people to do this thankless job.
| zeroCalories wrote:
| It doesn't need to be TS for open source(but for closed,
| I'm leaning yes). But all code for these core
| technologies need to be tied to a real person that can be
| charged in western nations. Yes, it will make it harder
| to get people, but with how important these technologies
| are, we really should not be using some random guys code
| in the kernel.
| guinea-unicorn wrote:
| Don't forget that the NSA bribed RSA (the company) to
| insert a backdoor into their RNG. Being in western
| jurisdiction doesn't mean you won't insert backdoors into
| code. It just changes whom you will target with these
| backdoors. But they all equally make our technology less
| trustworthy so they are all equally despicable.
| zeroCalories wrote:
| It will significantly cut down on Russian and Chinese
| back doors, which is still an improvement, Mr. Just Made
| an Account.
| rwmj wrote:
| That just means the bad actors will all have clearance
| while putting in a bunch of hurdles for amateur
| contributors. The only answer is the hard one, constant
| improvement in methods to detect and mitigate bugs.
| zeroCalories wrote:
| "Constant improvement" sounds like "constantly playing
| catch-up". Besides that, someone with TS can be arrested
| and charged, and I don't want amateur contributors.
| cesarb wrote:
| > I think we should seriously consider something like a
| ts clearance as mandatory for work on core technologies.
|
| Was xz/lzma a core technology when it was created? Is my
| tiny "constant time equality" Rust crate a core
| technology? Even though it's used by the BLAKE3 crate? By
| the way, is the BLAKE3 crate a core technology? Will it
| ever become a core technology?
|
| With free software in general, things do not start a
| "core technology"; they become a "core technology" over
| time due to usage. At which point would a maintainer have
| to get a TS clearance? Would the equivalent of a TS
| clearance from my Latin America country be acceptable?
| And how would I obtain it? Is it even available to people
| outside the military and government (legit question, I
| never looked)?
| zeroCalories wrote:
| We probably shouldn't use your code at all, is the real
| answer. You can get TS, it just costs a lot of money.
| stackskipton wrote:
| In United States, you cannot apply for a clearance. You
| must get a job that requires a clearance, then start
| application process and wait.
| csdreamer7 wrote:
| The Linux kernel is complaining about a lack of funding
| for CI-one of the highest visibility projects out there.
| Where will the money come from for this?
|
| Corps? Aside from Intel most of them barely pay to
| upstream their drivers.
|
| The govt? The US federal government cut so much of it's
| support since the 70s and 80s.
| zeroCalories wrote:
| You're right, but accepting code from random Gmail
| accounts can't be the solution. Honestly the Linux kernel
| is a bloated mess, and will probably never be secured.
| isbvhodnvemrwvn wrote:
| This only ensures the backdoors are coming from
| governments that issued the clearances, nothing more. I
| prefer more competition, at least there is incentive to
| detect those issues.
| zeroCalories wrote:
| It will ensure that my OS doesn't have code from random
| Gmail accounts. If someone with U.S clearance submits a
| backdoor, they should either be charged in the U.S, or
| extradited to somewhere that will charge them. We have no
| idea who this person is, and even if we did we probably
| could not hold them accountable.
| vmladenov wrote:
| This seems infeasible for projects like LLVM that depend
| on international collaboration.
| AnonymousPlanet wrote:
| That's a very US centric view and would practically split
| the open source community along the atlantic at best and
| fracture it globally at worst. Be careful what you wish
| for.
| colinsane wrote:
| how many people in PRISM had such clearance? and how many
| of them would i trust? precisely zero.
| okasaki wrote:
| A quote from... your arse?
| threeseed wrote:
| It's also a known fact that China will coerce people by
| threatening family and friends.
|
| Seen this happen to friends here in Australia who were
| attending pro-Taiwan protests.
| jnxx wrote:
| And, Joey Hess has counted at least 750 commits to xz from
| that handle.
|
| https://hachyderm.io/@joeyh/112180715824680521
|
| This does not look trust-inspiring. If the code is complex,
| there could be many more exploits hiding.
| KingOfCoders wrote:
| Sleeper.
| gmnon wrote:
| Funny how Lasse Collin started to ccing himself and Jia Tan
| from 2024-03-20 (that was a day of tons of xz kernel patches),
| he never did that before. :)
|
| https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...
| ncr100 wrote:
| Also interesting, to me, how the GMail account for the
| backdoor contributor ONLY appears in the context of "XZ"
| discussions. Google their email address. Suggests a kind of
| focus, to me, and a lack of reality / genuineness.
| fullstop wrote:
| This also means that Google might know who they are, unless
| they were careful to hide behind VPN or other such means.
| bombcar wrote:
| This is extremely suspicious.
|
| It looks like someone may have noticed a unmaintained or
| lightly maintained project related to various things, and
| moved to take control of it.
|
| Otherwhere in the discussion here someone mentions the domain
| details changed; if you have control of the domain you have
| control of all emails associated with it.
| ui2RjUen875bfFA wrote:
| those pipe usages are quite suspicious
|
| https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n.
| ..
|
| https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n.
| ..
|
| pipeing into this shell script which now uses "eval"
|
| https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n.
| ..
|
| i guess this will be revisited and removed soon
| Hackbraten wrote:
| > pipeing into this shell script which now uses "eval"
|
| I don't actually see an issue with that `eval`. Why would
| one consider running `xz` followed by `eval`-ing its output
| more insecure than just running `xz`? If `xz` wants to do
| shenanigans with the privileges it already has, then it
| wouldn't need `eval`'s help for that.
| smeehee wrote:
| Debian have reverted xz-utils (in unstable) to 5.4.5 - actual
| version string is "5.6.1+really5.4.5-1". So presumably that
| version's safe; we shall see...
| Jommi wrote:
| the account was either sold or stolen
| userbinator wrote:
| _because of it 's "great new features"_
|
| "great" for whom? I've seen enough of the industry to
| immediately feel suspicious when someone uses that sort of
| phrasing in an attempt to persuade me. It's no different from
| claiming a "better experience" or similar.
| kapouer wrote:
| Github accounts of both xz maintainers have been suspended.
| miduil wrote:
| Not true, the original author wasn't suspended:
| https://github.com/Larhzu
|
| https://github.com/JiaT75 was suspended for a moment, but
| isn't anymore?
| boutique wrote:
| Both are suspended for me. Check followers on both
| accounts, both have a suspended pill right next to their
| names.
| miduil wrote:
| Ah, thanks for correcting me there - really weird that
| this isn't visible from the profile itself. Not even from
| the organization.
|
| The following page for each other show both accounts
| suspended indeed.
|
| https://github.com/Larhzu?tab=following
|
| https://github.com/JiaT75?tab=following
| mongol wrote:
| Interesting that one of the commits commented on update of the
| test file that it was for better reproducibility for having
| been generated by a fixed random seed (although how goes
| unmentioned). For the future, random test data better be
| generated as part of the build, rather than being committed as
| opaque blobs...
| api wrote:
| I'm surprised there isn't way more of this stuff. The supply
| chain is so huge and therefore represents so much surface area.
| thesnide wrote:
| The discussion to upload it to Debian is interesting on its own
| https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
| 0x0 wrote:
| Wow, that's a lot of anonymous accounts adding comments there
| urging for a fast merge!
|
| And this "Hans Jansen" guy is apparently running around
| salsa.debian.org pushing for more updates in other projects as
| well: https://salsa.debian.org/users/hjansen/activity
| matthews2 wrote:
| > running around salsa.debian.org pushing for more updates in
| other projects as well
|
| This is quite common in most (all?) distributions. People are
| going through lists of outdated packages, updating them,
| testing them, and pushing them.
| mardifoufs wrote:
| That account seems to be a contributor for xz though, you
| can see him interact a lot with the author of the backdoor
| on the GitHub repo. Some pull requests seem to be just the
| two of them discussing and merging stuff (which is normal
| but looks weird in this context)
| 2OEH8eoCRo0 wrote:
| And now we see why I don't trust anons, aliases, or anime
| characters to make contributions.
|
| My GitHub says exactly who I am!
| ValdikSS wrote:
| >that's a lot of anonymous accounts
|
| Just FYI, krygorin4545@proton.me (the latest message before
| the upload) was created Tue Mar 26 18:30:02 UTC 2024, about
| an hour earlier than the message was posted.
|
| Proton generates PGP key upon creating the account, with the
| real datetime of the key (but the key does not include the
| timezone).
| nammi wrote:
| That name jumped out at me, Hans Jansen is the name Dominic
| Monaghan used when posing as a German interviewer with Elijah
| Woods. Not that it can't be a real person
|
| https://youtu.be/IfhMILe8C84
| joeyh wrote:
| See comments about "Hans Janson" upthread, he appeared to
| collaborate on the exploit in other ways as well.
| rsync wrote:
| Hans Gruber would have Been a much more stylish choice...
| fullstop wrote:
| I get the feeling that a number of the comments are all the
| same person / group.
| oasisaimlessly wrote:
| For anyone else feeling some deja vu about ifunc / Valgrind
| errors, this Red Hat issue [1] was previously linked from HN 12
| days ago [2].
|
| [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598
|
| [2]: https://news.ycombinator.com/item?id=39733185
| formerly_proven wrote:
| Quite ironic: The most recent commit in the git repo is "Simplify
| SECURITY.md", committed by the same Github account which added
| the backdoor.
|
| https://github.com/tukaani-project/xz/commit/af071ef7702debe...
| rany_ wrote:
| It's not ironic, this change is really sinister IMO. They want
| you to waste more time after you've submitted the security
| report and maximize the amount of back and forth. Basically the
| hope is that they'd be able to pester you with requests for
| more info/details in order to "resolve the issue" which would
| give them more time to exploit their targets.
| arp242 wrote:
| I've long since said that if you want to hide something nefarious
| you'd do that in the GNU autoconf soup (and not in "curl | sh"
| scripts).
|
| Would be interesting to see what's going on here; the person who
| did the releases has done previous releases too (are they
| affected?) And has commits going back to 2022 - relatively
| recent, but not _that_ recent. Many are real commits with real
| changes, and they have commits on some related projects like
| libarchive. Seems like a lot of effort just to insert a backdoor.
|
| _Edit_ : anyone with access can add files to existing releases
| and it won't show that someone else added it (I just tested).
| However, the timestamp of the file will be to when you uploaded
| it, not that of the release. On xz all the timestamps of the
| files match with the timestamp of the release (usually the
| .tar.gz is a few minutes earlier, which makes sense). So looks
| like they were done by the same person who did the release. I
| suspected someone else might have added/altered the files briefly
| after the release before anyone noticed, but that doesn't seem to
| be the case.
| sslayer wrote:
| I would be curious if their commits could be analyzed for
| patterns that could then be used to detect commits from their
| other account
| carom wrote:
| There was a DARPA program on this topic called Social Cyber.
| [1]
|
| 1. https://www.darpa.mil/program/hybrid-ai-to-protect-
| integrity...
| bombcar wrote:
| One thing that is annoying is that many open source projects
| have been getting "garbage commits" apparently from people
| looking to "build cred" for resumes or such.
|
| Easier and easier to hide this junk in amongst them.
| bananapub wrote:
| annoying ... and convenient for some!
| bawolff wrote:
| I mean, a backdoor at this scale (particularly if it wasn't
| noticed for a while and got into stable distros) could be worth
| millions. Maybe hundreds of millions (think of the insider
| trading possibilities alone, not to mention espionage). 2 years
| doesn't seem like that much work relative to the potential pay
| off.
|
| This is the sort of case where america's over the top hacking
| laws make sense.
| jethro_tell wrote:
| And what law would you use to target someone who wrote some
| code and posted it for free on the internet that was
| willingly consumed?
| bawolff wrote:
| The computer abuse and fraud act? Seems like a pretty easy
| question to answer.
| jethro_tell wrote:
| Maybe I'm miss understanding things, but it seems like
| anyone can publish an exploit on the internet without
| being a crime. In the same way encryption is free speech.
|
| It would seem unlikely this guy would be also logging
| into peoples boxes after this.
|
| It seems a much tougher job to link something like this
| to an intentional unauthorized access.
|
| At this point, we have no confirmed access via
| compromise.
|
| Do you know of a specific case where the existence of a
| backdoor has been prosecuted without a compromise?
|
| Who would have standing to bring this case? Anyone with a
| vulnerable machine? Someone with a known unauthorized
| access. Other maintainers of the repo?
|
| IANAL but it is unclear that a provable crime has been
| committed here
| refulgentis wrote:
| > IANAL
|
| Best to leave it at that.
|
| It's not worth your time or the reader's time trying to
| come up with a technicality to make it perfectly legal to
| do something we know little about, other than it's
| extremely dangerous.
|
| Law isn't code, you gotta violate some pretty bedrock
| principles to pull off something like this and get away
| with it.
|
| Yes, if you were just a security researcher experimenting
| on GitHub, it's common sense you should get away with
| it*, and yes, it's hard to define a logical proof that
| ensnares this person, and not the researcher.
|
| * and yes, we can come up with another hypothetical where
| the security researcher shouldn't get away with it.
| Hypotheticals all the way down.
| bawolff wrote:
| > but it seems like anyone can publish an exploit on the
| internet without being a crime
|
| Of course. The mere publishing of the exploit is not the
| criminal part. Its the manner & intent in which it was
| published that is the problem.
|
| > At this point, we have no confirmed access via
| compromise.
|
| While i don't know the specifics for this particular law,
| generally it doesn't matter what you actually did. What
| is relavent is what you _tried_ to do. Lack of success
| doesn 't make you innocent.
|
| > Who would have standing to bring this case?
|
| The state obviously. This is a criminal matter not a
| civil one. You don't even need the victim's consent to
| bring a case.
|
| [IANAL]
| eigenvalue wrote:
| Every single commit this person ever did should immediately be
| rolled back in all projects.
| gopher_space wrote:
| It's weird and disturbing that this isn't the default
| perspective.
| freedomben wrote:
| Well, it is much easier said than done. Philosophically I
| agree, but in the real world where you have later commits
| that might break and downstream projects, etc, it isn't
| very practical. It strikes me as in a similar vein to high
| school students and beauty pageant constestants calling for
| world peace. Really great goal, not super easy to
| implement.
|
| I would definitely be looking at every single commit though
| and if it isn't obviously safe I'd be drilling in.
| maxcoder4 wrote:
| Imagine someone tried to revert all the commits you ever
| did. Doesn't sound easy.
| concordDance wrote:
| Some of those commits might fix genuine vulnerabilities. So
| you might trade a new backdoor for an old vulnerability
| that thousands of criminal orgs have bots for exploiting.
|
| Damage wise, most orgs aren't going to be hurt much by NSA
| or the Chinese equivalent getting access, but a Nigerian
| criminal gang? They're far more likely to encrypt all your
| files and demand a ransom.
| mysidia wrote:
| Still.. At this point the default assumption should be
| every commit is a vulnerability or facilitating a
| potential vulnerability.
|
| For example, change from safe_fprintf to fprintf. It
| would be appropriate that every commit should be reviewed
| and either tweaked or re-written to ensure the task is
| being done in the safest way and doesn't have anything
| that is "off" or introducing a deviation from the way
| that codebase standardly goes about tasks within
| functions.
| KeplerBoy wrote:
| Surely this is happening right now.
|
| A lot of eyes are on the code. From all sides. Folks
| trying to find old unpatched backdoors to exploit or
| patch.
| bananapub wrote:
| it's not weird at all?
|
| randomly reverting two years of things across dozens of
| repositories will break them, almost definitely make them
| unbuildable, but also make them unreleasable in case any
| other change needs to happen soon.
|
| all of their code needs to be audited to prove it shouldn't
| be deleted, of course, but that can't happen in the next
| ten minutes.
|
| I swear that HN has the least-thought-through hot takes of
| any media in the world.
| datascienced wrote:
| Yeah if you tried to revert stuff that was done weeks ago
| on a relatively small team you know how much painstaking
| work it can be.
| andruby wrote:
| Hoe will you do that practically though? That's probably
| thousands of commits upon which tens or hundred thousand
| commits from others were built. You can't just rollback
| everything two years and expect it not to break or bring back
| older vulnerabilities that were patched in those commits.
| kjs3 wrote:
| Likely part of what the attacker(s) are counting on. Anyone
| want to place odds this isn't the only thing that's going
| to be found?
| neurostimulant wrote:
| Rolling back two years worth of commits made by a major
| contributor is going to be hell. I'm looking forward to see
| how they'll do this.
| ptx wrote:
| Couldn't the autoconf soup be generated from simpler inputs by
| the CI/CD system to avoid this kind of problem?
| Incomprehensible soup as a build artifact (e.g. executables) is
| perfectly normal, but it seems to me that such things don't
| belong in the source code.
|
| (This means you too, gradle-wrapper! And your generated wrapper
| for your generated wrapper. That junk is not source code and
| doesn't belong in the repo.)
| bonzini wrote:
| Yes, it's usually regenerated already. However even the
| source is often pretty gnarly.
|
| And in general, the build system of a large project is doing
| a lot of work and is considered pretty uninteresting and
| obscure. Random CMake macros or shell scripts would be just
| as likely to host bad code.
|
| This is also why I like meson, because it's much more
| constrained than the others and the build system tends to be
| more modular and the complex parts split across multiple
| smaller, mostly independent scripts (written in Python or
| bash, 20-30 lines max). It's still complex, but I find it
| easier to organize.
| jnxx wrote:
| > And in general, the build system of a large project is
| doing a lot of work and is considered pretty uninteresting
| and obscure. Random CMake macros or shell scripts would be
| just as likely to host bad code.
|
| Build systems can even have undefined behaviour in the C++
| sense. For example Conan 2 has a whole page on that.
| mysidia wrote:
| The other thing besides the autoconf soup is the XZ project
| contains incomprehensible binaries as "test data"; the
| "bad-3-corrupt_lzma2.xz" part of the backdoor that they even
| put in the repo.
|
| It's entirely possible they could have got that injection
| through review, even if they had that framwork and instead
| put it in source files used to generate autoconf soup.
| AeroNotix wrote:
| Pure speculation but my guess is a specific state actor _ahem_
| is looking for developers innocently working with open source
| to then strongarm them into doing stuff like this.
| bodyfour wrote:
| > I've long since said that if you want to hide something
| nefarious you'd do that in the GNU autoconf soup (and not in
| "curl | sh" scripts).
|
| Yeah, I've been banging on that same drum for ages too... for
| example on this very site a decade ago:
| https://news.ycombinator.com/item?id=7213563
|
| I'm honestly surprised that this autoconf vector hasn't
| happened more often... or more often that we know of.
| dist-epoch wrote:
| > they have commits on some related projects like libarchive
|
| Windows started using libarchive to support .rar, .7z, ...
|
| https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...
| IshKebab wrote:
| Yeah this was my first thought too. Though I think the case
| against autoconf is already so overwhelming I think anyone
| still using it is just irredeemable; this isn't going to
| persuade them.
| stabbles wrote:
| How about wheels in the python ecosystem
| bonyt wrote:
| For those panicking, here are some key things to look for, based
| on the writeup:
|
| - A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was
| added in the last month or so. If you're not on a rolling release
| distro, your version is probably older.
|
| - A debian or RPM based distro of Linux on x86_64. In an apparent
| attempt to make reverse engineering harder, it does not seem to
| apply when built outside of deb or rpm packaging. It is also
| specific to Linux.
|
| - Running OpenSSH sshd from systemd. OpenSSH as patched by some
| distros only pulls in libsystemd for logging functionality, which
| pulls in the compromised liblzma5.
|
| Debian testing already has a version called '5.6.1+really5.4.5-1'
| that is really an older version 5.4, repackaged with a newer
| version to convince apt that it is in fact an upgrade.
|
| It is possible there are other flaws or backdoors in liblzma5,
| though.
| treffer wrote:
| Ubuntu still ships 5.4.5 on 24.03 (atm).
|
| I did a quick diff of the source (.orig file from
| packages.ubuntu.com) and the content mostly matched the 5.4.5
| github tag except for Changelog and some translation files. It
| does match the tarball content, though.
|
| So for 5.4.5 the tagged release and download on github differ.
|
| It does change format strings, e.g. +#:
| src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With
| --format=raw, --suffix=.SUF is required unless writing to
| stdout" +msgid "With --format=raw, --suffix=.SUF is
| required unless writing to stdout" +msgstr "%s: amb
| --format=raw, --suffix=.SUF es necessari si no s'escriu a la
| sortida estandard"
|
| There is no second argument to that printf for example. I think
| there is at least a format string injection in the older
| tarballs.
|
| [Edit] formatting
| fransje26 wrote:
| Thanks for the heads up.
| chasil wrote:
| RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.
| mort96 wrote:
| FYI, your formatting is broken. Hacker News doesn't support
| backtick code blocks, you have to indent code.
|
| Anyway, so... the xz project has been compromised for a long
| time, at least since 5.4.5. I see that this JiaT75 guy has
| been the primary guy in charge of at least the GitHub
| releases for years. Should we view all releases after he got
| involved as probably compromised?
| treffer wrote:
| Thank you, formatting fixed.
|
| My TLDR is that I would regard all commits by JiaT75 as
| potentially compromised.
|
| Given the ability to manipulate gitnhistory I am not sure
| if a simple time based revert is enough.
|
| It would be great to compare old copies of the repo with
| the current state. There is no guarantee that the history
| wasn't tampered with.
|
| Overall the only safe action would IMHO to establish a new
| upstream from an assumed good state, then fully audit it.
| At that point we should probably just abandon it and use
| zstd instead.
| tomrod wrote:
| Not just Jia. There are some other accounts of concern
| with associated activity or short term/bot-is names.
| jwilk wrote:
| "#, fuzzy" means the translation is out-of-date and it will
| be discarded at compile time.
| treffer wrote:
| I tried to get the translation to trigger by switching to
| french and it does not show. You are right.
|
| So it's just odd that the tags and release tarballs
| diverge.
| hostyle wrote:
| $ dpkg-query -W liblzma5
|
| liblzma5:amd64 5.4.1-0.2
| fransje26 wrote:
| I did notice that my debian-based system got noticeably slower
| and unresponsive at times the last two weeks, without obvious
| reasons. Could it be related?
|
| I read through the report, but what wasn't directly clear to me
| was: what does the exploit actually do?
|
| My normal internet connection has such an appalling upload that
| I don't think anything relevant could be uploaded. But I will
| change my ssh keys asap.
| anarazel wrote:
| > I did notice that my debian-based system got noticeably
| slower and unresponsive at times the last two weeks, without
| obvious reasons. Could it be related?
|
| Possible but unlikely.
|
| > I read through the report, but what wasn't directly clear
| to me was: what does the exploit actually do?
|
| It injects code that runs early during sshd connection
| establishment. Likely allowing remote code execution if you
| know the right magic to send to the server.
| fransje26 wrote:
| Thank you for the explanation.
| cpach wrote:
| Are you on stable/testing/unstable?
|
| With our current knowledge, stable shouldn't be affected by
| this.
| fransje26 wrote:
| Stable, luckily. Thank you for the information.
| idoubtit wrote:
| The article gives a link to a simple shell script that detects
| the signature of the compromised function.
|
| > Running OpenSSH sshd from systemd
|
| I think this is irrelevant.
|
| From the article: "Initially starting sshd outside of systemd
| did not show the slowdown, despite the backdoor briefly getting
| invoked." If I understand correctly the whole section, the
| behavior of OpenSSH may have differed when launched from
| systemd, but the backdoor was there in both cases.
|
| Maybe some distributions that don't use systemd strip the libxz
| code from the upstream OpenSSH release, but I wouldn't bet on
| it if a fix is available.
| bonyt wrote:
| I think the distributions that _do_ use systemd are the ones
| that add the libsystemd code, which in turn brings in the
| liblzma5 code. So, it may not be entirely relevant how it is
| run, but it needs to be a version of OpenSSH patched.
| anarazel wrote:
| > From the article: "Initially starting sshd outside of
| systemd did not show the slowdown, despite the backdoor
| briefly getting invoked." If I understand correctly the whole
| section, the behavior of OpenSSH may have differed when
| launched from systemd, but the backdoor was there in both
| cases.
|
| It looks like the backdoor "deactivates" itself when it
| detects being started interactively, as a security researcher
| might. I was eventually able to circumvent that, but unless
| you do so, it'll not be active when started interactively.
|
| However, the backdoor would also be active if you started it
| with an shell script (as the traditional sys-v rc scripts
| did) outside the context of an interactive shell, as TERM
| wouldn't be set either in that context.
|
| > Maybe some distributions that don't use systemd strip the
| libxz code from the upstream OpenSSH release, but I wouldn't
| bet on it if a fix is available.
|
| There's no xz code in openssh.
| blcknight wrote:
| > Debian testing already has a version called
| '5.6.1+really5.4.5-1' that is really an older version 5.4,
| repackaged with a newer version to convince apt that it is in
| fact an upgrade.
|
| I'm surprised .deb doesn't have a better approach. RPM has
| epoch for this purpose http://novosial.org/rpm/epoch/index.html
| pja wrote:
| Debian packages can have epochs too. I'm not sure why the
| maintainers haven't just bumped the epoch here.
|
| Maybe they're expecting a 5.6.x release shortly that fixes
| all these issues & don't want to add an epoch for a very
| short term packaging issue?
| nicolas_17 wrote:
| .deb has epochs too, but I _think_ Debian developers avoid it
| where possible because 1:5.4.5 is interpreted as newer than
| _anything_ without a colon, so it would break eg. packages
| that depend on liblzma >= 5.0, < 6. There may be more common
| cases that aren't coming to mind now.
| stefanor wrote:
| Debian has epochs, but it's a bad idea to use them for this
| purpose.
|
| Two reasons:
|
| 1. Once you bump the epoch, you have to use it forever. 2.
| The deb filename often doesn't contain the epoch (we use a
| colon which isn't valid on many filesystems), so an epoch-
| revert will give the same file name as pre-epoch, which
| breaks your repository.
|
| So, the current best practice is the +really+ thing.
| intel_brain wrote:
| Got this on OpenSUSE: `5.6.1.revertto5.4-3.2`
| o11c wrote:
| Honestly, the Gentoo-style global blacklist (package.mask)
| to force a downgrade is probably a better approach for
| cases like this. Epochs only make sense if your upstream is
| insane and does not follow a consistent numbering system.
| kzrdude wrote:
| Thanks for the info, the filename thing sounds like a
| problem, one aspect of the epoch system doesn't work for
| the purpose then.
| 5p4n911 wrote:
| I really like the XBPS way of the reverts keyword in the
| package template that forces a downgrade from said software
| version. It's simple but works without any of the troubles
| RPM epochs have with resolving dependencies as it's just
| literally a way to tell xbps-install that "yeah, this is a
| lower version number in the repository but you should update
| anyway".
| NotPractical wrote:
| > If you're not on a rolling release distro, your version is
| probably older.
|
| Ironic considering security is often advertised as a feature of
| rolling release distros. I suppose in most instances it does
| provide better security, but there are some advantages to
| Debian's approach (stable Debian, that is).
| pdw wrote:
| Focusing on sshd is the wrong approach. The backdoor was in
| liblzma5. It was discovered to attack sshd, but it very likely
| had other targets as well. The payload hasn't been analyzed
| yet, but _almost everything_ links to libzma5. Firefox and
| Chromium do. Keepassxc does. And it might have made arbitrary
| changes to your system, so installing the security update might
| not remove the backdoor.
| colanderman wrote:
| The latest commit from the user who committed those patches is
| weirdly a simplification of the security reporting process, to
| not request as much detail:
|
| https://github.com/tukaani-project/xz/commit/af071ef7702debe...
|
| Not sure what to make of this.
| caelum19 wrote:
| Potentially the purpose is that if someone goes to the effort
| to get those details together, they are more likely to send the
| same report to other trusted individuals. Maybe it was
| originally there to add legitimacy, then they got a report sent
| in, and removed it to slow the spread of awareness
| londons_explore wrote:
| > Affected versions of XZ Utils
|
| Most people, to find the affected versions, would either have
| to bisect or delve deep enough to find the offending commit.
| Either of which would reveal the attacker.
|
| By not asking for the version, there is a good chance you
| just report "It's acting oddly, plz investigate".
| rany_ wrote:
| I think the reason is pretty obvious. They want you to waste
| more time after you've submitted the security report and
| maximize the amount of back and forth. Basically the hope is
| that they'd be able to pester you with requests for more
| info/details in order to "resolve the issue" which would give
| them more time to exploit their targets.
| ParetoOptimal wrote:
| If you have a recently updated NixOS unstable it has the affected
| version: $ xz --version xz (XZ Utils)
| 5.6.1 liblzma 5.6.1
|
| EDIT: I've been informed on the NixOS matrix that they are 99%
| sure NixOS isn't affected, based on conversations in
| #security:nixos.org
| 20after4 wrote:
| > "Docs: Simplify SECURITY.md."
|
| https://github.com/tukaani-project/xz/commit/af071ef7702debe...
|
| Removes instructions about details relevant to security reports.
| Heh, nice one.
| bagels wrote:
| Is this a crime? Has anyone been prosecuted for adding a backdoor
| like this?
| pvg wrote:
| _Has anyone been prosecuted for adding a backdoor_
|
| Google up Randal Schwartz. Caution: clickhole.
| bagels wrote:
| Seems a little different. Based on a quick read, he gained
| unauthorized access to systems.
|
| In this case, backdoor code was offered to and accepted by xz
| maintainers.
| pvg wrote:
| _Seems a little different. Based on a quick read_
|
| It is a little different but a thing that you might have
| missed in the quick read is that one of the things he was
| accused of was installing and using a backdoor.
| bagels wrote:
| One involves making unauthorized access, the other does
| not.
| ptx wrote:
| Lots of things are crimes even though they're just offering
| something to a victim who willingly accepts it, e.g.
| phishing attacks, fraudulent investment schemes,
| contaminated food products.
| bagels wrote:
| Sure. I'm wondering if there is a specific law that was
| broken here. It seems to me that it might be beneficial
| if there were some legal protection against this sort of
| act.
| amiga386 wrote:
| As far as I remember, he added no backdoors.
|
| He was a consultant/sysadmin for Intel, and he did 3 things
| which he thought his employer would support, and was
| astonished to find that not only did his employer not
| support, but actively had him prosecuted for doing it. Ouch.
|
| 1. He ran a reverse-proxy on two machines so he could check
| in on them from home.
|
| 2. He used the crack program to find weak passwords.
|
| 3. He found a weak password, and used it to log into a
| system, which he copied the /etc/shadow file from to look for
| additional weak passwords.
|
| https://www.giac.org/paper/gsec/4039/intel-v-randal-l-
| schwar...
|
| https://web.archive.org/web/20160216204357/http://www.lightl.
| ..
|
| He didn't try and hide his activities, and didn't do anything
| else untoward, it was literally just these things which most
| people wouldn't bat an eyelid at. These days, it is
| completely normal for a company to provide VPNs for their
| employees, and completely normal to continually scan for
| unexpected user accounts or weak passwords. But... because he
| didn't explain this to higher-ups and get their buy-in, they
| prosecuted him instead of thanking him.
| uniformlyrandom wrote:
| To be fair, it is perfectly normal for a surgeon to cut
| people with a sharp knife with their permission while in
| the hospital.
|
| It is kinda sus when they do it at home without consent.
| returningfory2 wrote:
| A couple of years ago I wrote a Go library that wraps the xz C
| code and allows you to do xz compression in Go:
| https://github.com/jamespfennell/xz
|
| About a week ago I received the first PR on that repo, to upgrade
| to 5.6.1. I thought it was odd to get such a random PR...it's not
| the same GitHub account as upstream though.
| pinko wrote:
| > it's not the same GitHub account as upstream
|
| This is valuable information, and a sign that this may be the
| tip of an iceberg.
| Bromeo wrote:
| I don't want to read too much into it, but the person
| (supposedly) submitting the PR seems to work at 1Password since
| December last year, as per his Linkedin. (And his Linkedin page
| has a link to the Github profile that made the PR).
| returningfory2 wrote:
| Yeah the GitHub account looks really really legitimate. Maybe
| it was compromised though?
| jethro_tell wrote:
| What looks legit about a gmail address and some stock art
| for a profile?
| gpm wrote:
| [Deleted per below]
| TeMPOraL wrote:
| Can you stay in that org after leaving Google?
| bananapub wrote:
| whoever is in charge of removing people from the Google
| github org has the itchiest trigger finger in the whole
| exiting-the-company process tree.
| fooker wrote:
| No
| Jyaif wrote:
| You are not looking at the right profile. This is the
| profile that people are talking about:
| https://github.com/jaredallard
| gpm wrote:
| Oops, you're absolutely correct. Deleted (via edit) my
| comment above. Thanks.
| ncr100 wrote:
| He was just (50 minutes ago) removed from the oss fuzz
| repo.
|
| I hope this also (at least temporarily until verification
| of 'bad/good') remove him from the org?
| buildbot wrote:
| Plus the README.md that is just a rickroll
| ncr100 wrote:
| The 2 GMail accounts are 85% / mainly associated with XZ
| work, since 2021, per searching for them explicitly via
| Google.
| computerfriend wrote:
| The PR's two commits are signed by a key that was also used
| to sign previous commits belonging to that author.
| dralley wrote:
| Hold up, are you saying that
| https://github.com/jaredallard and the accounts
| affiliated with this XZ backdoor share a PGP key? Or
| something else?
| computerfriend wrote:
| No, this account made a PR and their commits were signed
| [1]. Take a look at their other repositories, e.g. they
| did AoC 2023 in Rust and published it, the commits in
| that repository are signed by the same key. So this is
| not (just) a GitHub account compromise.
|
| I find this aspect to be an outlier, the other attacker
| accounts were cutouts. So this doesn't quite make sense
| to me.
|
| [1] https://github.com/jamespfennell/xz/pull/2/commits
| bombcar wrote:
| If I were trying to compromise supply chains, getting into
| someplace like 1Password would be high up on the list.
|
| Poor guy, he's probably going to get the third degree now.
| switch007 wrote:
| As a 1Password user, I just got rather nervous.
| bombcar wrote:
| Yubikeys starting to look kinda yummy.
| wiml wrote:
| Hardware gets backdoored too, remember Crypto AG?
| lelandbatey wrote:
| They're definitely a real person. I know cause that
| "1Password employee since December" is a person I know IRL
| and worked with for years at their prior employer. They're
| not a no-name person or a fake identity just FYI. Please
| don't be witch hunting; this genuinely looks like an
| unfortunate case where Jared was merely proactively doing
| their job by trying to get an externally maintained golang
| bindings of XZ to the latest version of XZ. Jared's pretty
| fantastic to work with and is definitely the type of person
| to be filing PRs on external tools to get them to update
| dependencies. I think the timing is comically bad, but I can
| vouch for Jared.
|
| https://github.com/jamespfennell/xz/pull/2
| greatjack613 wrote:
| Something smells off here, for everyone else here please
| don't take the above comment seriously until the person
| brings in proof that they work with him
| arp242 wrote:
| As a bit of an aside, I would never accept a PR like this, and
| would always update $large_vendored_dependency myself. This is
| unreviewable, and trivial to insert any backdoor (unless you go
| through the motions of updating it yourself and diffing, at
| which point the PR becomes superfluous). I'd be wary even from
| a well-known author unless I knew them personally on some level
| (real-life or via internet). Not that I wouldn't trust them,
| but people's machines or accounts can get compromised, people
| can have psychotic episodes, things like that. At the very
| least I'd like to have some out-of-band "is this really you?"
| signal.
|
| This is how I once inserted a joke in one of our (private)
| repos that would randomly send cryptic messages to our chat
| channel. This was pretty harmless and just a joke (there's some
| context that made it funny), but it took them years to find it
| - and that was only because I told them after I quit.
|
| That said, looking at the GitHub account I'd be surprised if
| there's anything nefarious going on here. Probably just someone
| using your repo, seeing it's outdated, and updating it.
| LVB wrote:
| The (most?) popular SQLite driver for Go often gets PRs to
| update the SQLite C amalgamation, which the owner politely
| declines (and I appreciate him for that stance, and for
| taking on the maintenance burden it brings).
|
| e.g., https://github.com/mattn/go-
| sqlite3/pull/1042#issuecomment-1...
| 5kg wrote:
| The backdoor (test binary blob and autoconf) is not part of the
| pull request.
| cbmuser wrote:
| There was also a bug report in Debian which requested updating
| xz-utils to 5.6.1: https://bugs.debian.org/cgi-
| bin/bugreport.cgi?bug=1067708
| notyoutube wrote:
| Is the solution against such attacks in the future only to
| scrutinize more, or are there other reasonable options in terms
| of hardening?
| JanisErdmanis wrote:
| The lesson here seems to not depend on tools written in
| languages that have complex, obscure build systems and no one
| is either able or interested to read. Using tools rewritten in
| Rust, Go or any other languege which resolves dependencies
| within project seems the only way to do hardening here.
| Lichtso wrote:
| Once somebody actually does this people are gonna complain
| the same as always: "The sole purpose of your project is to
| rewrite perfectly fine stuff in Rust for the sake of it" or
| something along these lines.
| blcknight wrote:
| I agree there's safer languages than C, but nobody reads the
| 50,000 lines changed when you update the vendoring in a
| random golang project. It would be easy to introduce
| something there that nobody notices too.
| JanisErdmanis wrote:
| It is generally harder to introduce vulnerabilities in
| readable language even more when it is memory safe. Sure
| life is not perfect and bad actors would have found a ways
| to inject vulnerabilities also in Rust, Go codebase. The
| benefit of modern languages is that there is one way to
| build things and the source code is the only thing that
| needs to be auditted.
| ok123456 wrote:
| Wouldn't a supply chain attack like this be much worse with
| Rust and Cargo because of the fact it's not just a single
| dynamic library that needs to be reinstalled system-wise,
| but, instead, every binary would require a new release?
| gpm wrote:
| It would mean rebuilding more packages. I don't think
| that's meaningfully "much worse", package mangers are
| perfectly capable of rebuilding the world and the end-user
| fix is the same "pacman -Syu"/"apt-get update && apt-get
| upgrade"/...
|
| On the flip side the elegant/readable build system means
| that the place this exploit was hidden wouldn't exist.
| Though I wouldn't confidently say that 'no hiding places
| exist' (especially with the parts of the ecosystem that
| wrap dependencies in other languages).
| ok123456 wrote:
| It's much worse because it requires repackaging every
| affected system package instead of a single library.
| Knowing which packages are affected is difficult because
| that information isn't exposed to the larger system
| package manager. After all, it's all managed by the build
| system.
| packetlost wrote:
| In the era of modern CI and build infrastructure, I don't
| really think that's materially an issue.
| ok123456 wrote:
| Those CI and build infrastructures rely on the Debian and
| RedHat being able to build system packages.
|
| How would an automated CI or build infrastructure stop
| this attack? It was stopped because the competent package
| maintainer noticed a performance regression.
|
| In this case, this imagined build system would have to
| track every rust library used in every package to know
| which packages to perform an emergency release for.
| packetlost wrote:
| I... don't see your point. Tracking the dependencies a
| static binary is built with is already a feature for
| build systems, just maybe not the ones Debian and RH are
| using now, but I imagine they would if they were shipping
| static binaries.
|
| Rust isn't really the point here, it's the age old static
| vs dynamic linking argument. Rust (or rather, Cargo)
| already tracks which version of a dependency a library
| depends on (or a pattern to resolve one), but it's
| besides the point.
| ok123456 wrote:
| Rust is the issue here because it doesn't give you much
| of an option. And that option is the wrong one if you
| need to do an emergency upgrade of a particular library
| system-wide.
| packetlost wrote:
| It's really not, it's not hard to do a reverse search of
| [broken lib] <= depends on <= [rust application] and then
| rebuild everything that matches. You might have to
| rebuild more, but that's not really _hard_ with modern
| build infrastructure.
|
| Not to mention if you have a Rust application that
| depends on C libraries, it already dynamically links on
| most platforms. You only need to rebuild if a Rust crate
| needs to be updated.
| steveklabnik wrote:
| > imagined
|
| Cargo already has this information for every project it
| builds. That other systems do not is their issue, but
| it's not a theoretical design.
| ok123456 wrote:
| So, I know that librustxz has been compromised. I'm
| Debian. I must dive into each rust binary I distribute as
| part of my system and inspect their Cargo.toml files.
| Then what? Do I fork each one, bump the version, hope it
| doesn't break everything, and then push an emergency
| release!??!
| steveklabnik wrote:
| > I must dive into each rust binary I distribute as part
| of my system and inspect their Cargo.toml
|
| A few things:
|
| 1. It'd be Cargo.lock
|
| 2. Debian, in particular, processes Cargo's output here
| and makes individual debs. So they've taken advantage of
| this to already know via their regular package manager
| tooling.
|
| 3. You wouldn't dive into and look through these by hand,
| you'd have it as a first-class concept. "Which packages
| use this package" _should_ be table stakes for a package
| manager.
|
| > Then what? Do I fork each one, bump the version, hope
| it doesn't break everything, and then push an emergency
| release!??!
|
| The exact same thing you do in this current situation? It
| depends on what the issue is. Cargo isn't magic.
|
| The point is just that "which libraries does the binary
| depend on" isn't a problem with actual tooling.
|
| People already run tools like cargo-vet in CI to catch
| versions of packages that may have issues they care
| about.
| ok123456 wrote:
| > The exact same thing you do in this current situation?
| It depends on what the issue is. Cargo isn't magic.
|
| False. In the current situation, you just release a new
| shared library that is used system-wide.
| steveklabnik wrote:
| Okay, so the analogous situation here is that you release
| a new version of the library, and rebuild. Done.
| ok123456 wrote:
| Except that's not the case at all with Rust.
| packetlost wrote:
| Except it _is_. The system package maintainers release a
| new build of the package in question and then you install
| it. There 's not really anything else to do here. There's
| nothing special about Rust in this context, it would be
| exactly the same scenario on, for example, Musl libc
| based distros with any C application.
| ok123456 wrote:
| And Alpine Linux is largely a mistake.
| packetlost wrote:
| That's not an argument, nor is it productive. Nobody even
| mentioned Alpine. Go away.
| steveklabnik wrote:
| Ok well have a nice day I guess.
| JanisErdmanis wrote:
| This seems to be an orthogonal issue. Rust could build the
| same dynamic library with cargo which could then be
| distributed. The diference is that there would be a single
| way to build things.
| ok123456 wrote:
| Most Rust libraries are not dynamically linked; instead,
| versions are pinned and included statically during the
| build process. This is touted as a feature.
|
| Only a few projects are built as system-wide libraries
| that expose a C-compatible abi interface; rsvg comes to
| mind.
| timschmidt wrote:
| It's not touted as a feature by any Rust developers I
| know of. The Rust ABI is merely still stabilizing. See:
| https://github.com/rust-lang/rust/pull/105586
| YetAnotherNick wrote:
| I am not completely sure about this exploit, but seems like
| a binary needed to be modified for the exploit to work[1]
| which was later picked up by build system.
|
| https://github.com/tukaani-
| project/xz/commit/6e636819e8f0703...
| ok123456 wrote:
| The binary was an xz test file that contained a script
| that patched the c-code.
| klysm wrote:
| People are going to be upset with this perspective but I
| completely agree. The whole autoconf set of tools is a
| complete disaster.
| arp242 wrote:
| You don't need a complex obscure build system for most C
| code. There's a lot of historical baggage here, but many
| projects (including xz, I suspect) can get away with a fairly
| straight-forward Makefile. Double so when using some GNU make
| extensions.
| bonzini wrote:
| Rust is the worst in terms of build system transparency. Ever
| heard of build.rs? You can hide backdoors in any crate, or in
| any crate's build.rs, or the same recursively.
| timschmidt wrote:
| Most build systems are turing-complete. Rust, at least,
| drastically reduces the need for custom build scripts (most
| of my projects have empty build.rs files or lack one
| entirely), and build.rs being in the same language as the
| rest of the codebase aids transparency immensely.
| bonzini wrote:
| That doesn't make build.rs any less of a juicy target for
| a supply chain attack.
|
| Arbitrary code downloaded from the internet and run at
| build time? That's a nightmare scenario for auditing,
| much worse than anything Autotools or CMake can offer.
| timschmidt wrote:
| You're not wrong about arbitrary code execution. It's
| just that your statement applies to most of the packages
| on any linux distribution, Autotools and Cmake included,
| regardless of language. Many moreso than Rust due to the
| aforementioned features of Cargo and build.rs not
| requiring me to be an expert in a second language just to
| audit it.
| bonzini wrote:
| Packages in a Linux distro are not built on my machine,
| they are built by the distro in a sandbox. Every time I
| type "cargo build" I am potentially running arbitrary
| code downloaded from the internet. Every time I type
| "make" in an Autotools program only my code runs.
|
| > not requiring me to be an expert in another language
| just to audit it.
|
| Do you do that every time your Cargo.lock changes?
| timschmidt wrote:
| > Every time I type "make" in an Autotools program only
| my code runs.
|
| Says who? Make is just as good at calling arbitrary code
| as Cargo. Including code that reaches out over the
| network. Have you audited every single makefile to ensure
| that isn't the case?
| fragmede wrote:
| seems trivial for a configure script to call curl/wget
| somewhere in the depths of it, no?
| timschmidt wrote:
| Exactly. And at least Cargo will refuse to download a
| crate which has been yanked. So any crate which has been
| discovered to be compromised can be yanked, preventing
| further damage even when someone has already downloaded
| something it depends on.
|
| Building package with up-to-date dependencies is also
| vastly preferable to building against ancient copies of
| libraries vendored into a codebase at some point in the
| past, a situation I see far too often in C/C++ codebases.
| perihelions wrote:
| Imagine a more competent backdoor attempt on xz(1)--one that
| wouldn't have been noticed this quickly. xz is everywhere. They
| could pull off a "reflections on trusting trust": an xz which
| selectively modifies a tiny subset of the files it sees, like
| .tar.xz software tarballs underlying certain build processes. Not
| source code tarballs (someone might notice)--tarballs
| distributing pre-compiled binaries.
|
| edit to add: Arch Linux' entire package system used to run on
| .tar.xz binaries (they switched to Zstd a few years ago [0]).
|
| [0] https://news.ycombinator.com/item?id=19478171 ( _" Arch Linux
| propose changing compression method from xz to zstd
| (archlinux.org)"_)
| nolist_policy wrote:
| deb packages are xz compressed...
| joeyh wrote:
| A backdoored xz could also run payloads hidden inside other xz
| files, allowing targeted attacks.
| Phenylacetyl wrote:
| The same authors have also contributed to Zstd
| joeyh wrote:
| details please? I do not see any such contributions to
| https://github.com/facebook/zstd
| alright2565 wrote:
| https://github.com/tukaani-project/tukaani-project.github.io...
|
| > Note: GitHub automatically includes two archives Source code
| (zip) and Source code (tar.gz) in the releases. These archives
| cannot be disabled and should be ignored.
|
| The author was thinking ahead! Latest commit hash for this repo:
| 8a3b5f28d00ebc2c1619c87a8c8975718f12e271
| o11c wrote:
| For a long time, there was one legitimately annoying
| disadvantage to the git-generated tarballs though - they lost
| tagging information. However, since git 2.32 (released June
| 2021; presumably available on GitHub by August 2021 when they
| blogged about it) you can use `$Format:%(describe)$` ...
| limited to once per repository for performance reasons.
| rany_ wrote:
| Except this change was made in 2023, it is just scary how
| good this threat actor was.
| legobmw99 wrote:
| I believe they also do not include sub modules, which is a
| big disadvantage for some projects
| rom1v wrote:
| Btw, this is not the only project providing a source tarball
| different from the git repo, for example libusb also does this
| (and probably others):
|
| -
| https://github.com/libusb/libusb/issues/1468#issuecomment-19...
|
| - https://github.com/orgs/community/discussions/6003
| londons_explore wrote:
| I think the lesson here for packagers is that binary testdata
| should not be present while doing the build.
|
| It is too easy to hide things in testdata.
| wood_spirit wrote:
| A lot of eyes will be dissecting this specific exploit, and
| investigating this specific account, but how can we find the same
| kind of attack in a general way if it's being used in other
| projects and using other contributor names?
| londons_explore wrote:
| Note that the malicious binary is fairly long and complex.
|
| This attack can be stopped by disallowing any binary testdata
| or other non-source code to be on the build machines during a
| build.
|
| You could imagine a simple process which checks out the code,
| then runs some kind of entropy checker over the code to check
| it is all unminified and uncompressed source code, before
| finally kicking off the build process.
|
| autogenerated files would also not be allowed to be in the
| source repo - they're too long and could easily hide bad stuff.
| Instead the build process should generate the file during the
| build.
| trulyrandom wrote:
| This requires a more comprehensive redesign of the build
| process. Most Linux distributions also run the tests of the
| project they're building as part of the build process.
| uecker wrote:
| The code that runs during testing should not be allowed to
| affect the package though. If this is possible, this is
| misdesigned.
| treffer wrote:
| 1. Everything must be visible. A diff between the release
| tarball and tag should be unacceptable. It was hidden from the
| eyes to begin with.
|
| 2. Build systems should be simple and obvious. Potentially not
| even code. The inclusion was well hidden.
|
| 3. This was caught through runtime inspection. It should be
| possible to halt any Linux system at runtime, load debug
| symbols and map _everything_ back to the source code. If
| something can't map back then regard it as a potentially
| malicious blackbox.
|
| There has been a strong focus and joint effort to make
| distributions reproducible. What we haven't managed though is
| prove that the project compromises only of freshly compiled
| content. Sorta like a build time / runtime "libre" proof.
|
| This should exist for good debugging anyway.
|
| It wouldn't hinder source code based backdoors or malicious
| vulnerable code. But it would detect a backdoor like this one.
|
| Just an initial thought though, and probably hard to do, but
| not impossibly hard, especially for a default server
| environment.
| Avamander wrote:
| More reproducible builds, maybe even across distributions?
| Builds based on specific commits (no tarballs like in this
| case), possibly signed (just for attribution, not for security
| per se)? Allow fewer unsafe/runtime modifications The way oss-
| fuzz ASAN was disabled should've been a warning on its own, if
| these issues weren't so common.
|
| I'm not aware of any efforts towards it, but libraries should
| also probably be more confined to only provide intended
| functionality without being able to hook elsewhere?
| Tenobrus wrote:
| It looks like the person who added the backdoor is in fact the
| current co-maintainer of the project (and the more active of the
| two): https://tukaani.org/about.html
| kzrdude wrote:
| Makes me wonder if he's an owner of the github organization,
| and what happens with it now?
| Scaevolus wrote:
| It's wild that this could have laid dormant for far longer if the
| exploit was better written-- if it didn't spike slow down logins
| or disturb valgrind.
| jiripospisil wrote:
| Now consider that your average Linux distribution pulls in tens
| of thousands of packages, each of which can be similarly
| compromised. Pretty scary to think about.
| RGamma wrote:
| The terrible desktop software security model of
| weak/essentially non-existent security boundaries at run and
| compile time makes this all the more spicy.
|
| Computer security for billions runs on the simultaneous
| goodwill of many thousand contributors. Optimistically said
| it's actually a giant compliment to the programming community.
|
| And this is not even talking about hardware backdoors that are
| a million times worse and basically undetectable when done
| well. The myriad ways to betray user trust at any level of
| computation make me dizzy...
| Luker88 wrote:
| @people who write github scanners for updates and security issues
| (dependabot and the like)
|
| Can we start including a blacklist of emails and names of
| contributors (with reasons/links to discussions)?
|
| I can't track them and I don't want them in my projects.
|
| Might not be very helpful as it is easy to create new identities,
| but I see no reason to make it easier for them. Also, I might
| approach differently someone with lots of contributions to known
| projects than a new account, so it still helps.
| arp242 wrote:
| It takes a minute to create a new email address. And you can
| change or fake an email address on a git commit trivially. You,
| too, can writing code as anyone you want by just doing "git
| commit --author='Joe Biden <icecream@whitehouse.gov>'". On the
| internet nobody knows you're Joe Biden.
| nine_k wrote:
| You can write a rather simple GitHub action that would do that:
| look at a PR and reject / close it if you don't like it for
| some reason. AFAIK open-source projects have a free quota of
| actions.
|
| OTOH sticking to the same email for more than one exploit might
| be not as wise for a malicious agent.
| the8472 wrote:
| github already suspended the account
| n3uman wrote:
| https://github.com/tukaani-project/tukaani-project.github.io...
| Does this mean anything that it changed to a parameter??
| danielhlockard wrote:
| no. unlikely.
| Aissen wrote:
| Looks like one of the backdoor authors even went and disabled the
| feature the exploit relied on directly on oss-fuzz to prevent
| accidental discovery:
| https://social.treehouse.systems/@Aissen/112180302735030319
| https://github.com/google/oss-fuzz/pull/10667
|
| But luckily there was some serendipity: "I accidentally found a
| security issue while benchmarking postgres changes."
| https://mastodon.social/@AndresFreundTec/112180083704606941
| miduil wrote:
| This is getting addressed here: https://github.com/google/oss-
| fuzz/issues/11760
| nialv7 wrote:
| This in of itself can be legitimate. ifunc has real uses and it
| indeed does not work when sanitizer is enabled. Similar change
| in llvm: https://github.com/llvm/llvm-
| project/commit/1ef3de6b09f6b21a...
| mdip wrote:
| Anyone keeping current with OpenSUSE Tumbleweed got a
| update...downgrade. Prior to `zypper dup --no-allow-vendor-
| change` I had 5.6.0, now I'm at 5.4.6.
| intel_brain wrote:
| I see `5.6.1.revertto5.4-3.2`
| weinzierl wrote:
| The backdoor is not in the C source directly, but a build script
| uses data from files in the test dir to only create the backdoor
| in the release tars. Did I summarize that correctly?
| soneil wrote:
| That's how I understand it. A build script that's in the
| releases tarballs but not the git repo, checks to see if it's
| being run as part of the debian/build or rpm build processes,
| and then injects content from one of the "test" files.
| bombcar wrote:
| I could imagine another similar attack done against an image
| processing library, include some "test data" of corrupted
| images that should "clean up" (and have it actually work!)
| but the corruption data itself is code to be run elsewhere.
| sylware wrote:
| This is why the less the better... even if it means less
| comfortable... to a certain point obviously. And that includes
| SDKs...
| crispyambulance wrote:
| I am not embarrassed to say... is there anything in there that
| someone who runs a server with ssh needs to know?
|
| I literally can't make heads or tails of the risk here. All I see
| is the very alarming and scary words "backdoor" and "ssh server"
| in the same sentence.
|
| If I am keeping stuff up to date, is there anything at all to
| worry about?
| pxx wrote:
| You should probably not be running your own publicly-accessible
| ssh servers if this email is not sufficient to at least start
| figuring out what your next actions are.
|
| The email itself comes with an evaluation script to figure out
| if anything is currently vulnerable to specifically this
| discovery. For affected distributions, openssh servers may have
| been backdoored for at least the past month.
| crispyambulance wrote:
| Yet here I am, getting up every morning and getting dressed
| and tying my shoes all by myself, and then maintaining a
| small number of servers that have openssh on them!
|
| Thanks, though, for pointing out the little script at the
| very end of that technical gauntlet of an email intended for
| specialists. I had gotten through the first 3 or 4 paragraphs
| and had given up.
|
| What I should have done is just googled CVE-2024-3094,
| whatever, still glad I asked.
| sschueller wrote:
| So much for a quiet Easter holiday. Fuck
| multimoon wrote:
| It seems like based on the (very well written) analysis that this
| is a way to bypass ssh auth, not something that phones out which
| would've been even scarier.
|
| My server runs arch w/ a LTS kernel (which sounds dumb on the
| surface, but was by far the easiest way to do ZFS on Linux that
| wasn't Ubuntu) and it seems that since I don't have SSH exposed
| to the outside internet for good reason, and my understanding is
| Arch never patched shhd to begin with that I and most people who
| would be in similar situations to me are unaffected.
|
| Still insane that this happened to begin with, and I feel bad for
| the Archlinux maintainers who are now going to feel more pressure
| to try to catch things like this.
| LeoPanthera wrote:
| xz is just a horribly designed format, and always has been. If
| you use it, please switch to Lzip. Same compression level, but
| designed by someone competent.
|
| https://www.nongnu.org/lzip/
|
| https://www.nongnu.org/lzip/xz_inadequate.html
| xyst wrote:
| Time for another OS wipe. Glad I keep bleeding edge versions VMd
| pushedx wrote:
| Mirror of the report, since the Openwall servers appear to be
| down.
|
| https://web.archive.org/web/20240329182300/https://www.openw...
| markus_zhang wrote:
| Keeps one wonder how many similar backdoors are there in the
| wild. What is the best way to execute such a move? This is
| sophisticated enough, but not good enough to stay unnoticed for a
| long while. If I were a state actor I'd think about at least 6-12
| months.
| wannacboatmovie wrote:
| Really disappointed in the number of posters here who are playing
| down rushing to judgement and suggesting perhaps a legitimate
| developer was compromised, when it's very clear this is
| sophisticated and not the work of a single person.
|
| I'm recalling bad memories of the Juniper backdoor years ago.
|
| Whoever did this, was playing the long game. As the top post
| pointed out, there was an effort to get this into Fedora....
| which eventually makes its way into RHEL (read: high value
| targets). This was not for short term payoffs by some rogue
| developer trying to mine crypto or other such nonsense. What you
| are seeing here is the planting of seeds for something months or
| a year down the road.
| jcalvinowens wrote:
| Oof, this is on my Sid laptop:
| {0}[calvinow@mozart ~] dpkg-query -W liblzma5
| liblzma5:amd64 5.6.0-0.2 {0}[calvinow@mozart ~] hexdump
| -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30
| f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
| 1
|
| Glad I stopped running sshd on my laptop a long time ago... still
| probably going to reinstall :/
| msm_ wrote:
| No obvious need to reinstall if you didn't use ssh _and_ expose
| it publicly _and_ are not a politically important person. All
| signs suggest that it was a nation state attack, and you are
| likely not a target.
| jcalvinowens wrote:
| We'll see... given that sshd is just one of many possible
| argv[0] it may have choosen to act on, I'm going to be a
| little paranoid until it's been fully analyzed. It just takes
| half an hour to reinstall, I have some shows to catch up on
| anyway :)
| liveoneggs wrote:
| The best part is everyone disabling security tests that started
| failing
| c_rrodriguez wrote:
| Everybody here In jumping into the pure malice bandwagon, I have
| a better hypothesis.
|
| Abandonment and inaction, the actual developers of these tools
| are elsewhere, oblivious to this drama, trying to make living
| because most of the time you are not compensated nor any
| corporation cares about making things sustainable at all. This is
| the default status of everything your fancy cloud depends on
| underneath.
|
| An attacker took over of the project slowly and stayed dormant
| until recently.
| johnklos wrote:
| Except that doesn't match reality.
|
| Someone has worked on xz for several years. Are you saying that
| this somewhat active contributor was likely actively
| contributing, then all of a sudden stopped, also stopped paying
| attention, and also allowed their account to be compromised or
| otherwise handed it over to a nefarious party?
|
| That fails the sniff test.
| c_rrodriguez wrote:
| See, people drop dead from OSS projects pretty frecuently,
| usually because they take on other life responsabilities and
| there is no cushion or guard against a bus factor. Then it is
| very easy to get credentials compromised or have your project
| took over by someone else.
| elchief wrote:
| "Amazon Linux customers are not affected by this issue, and no
| action is required. AWS infrastructure and services do not
| utilize the affected software and are not impacted. Users of
| Bottlerocket are not affected."
|
| https://aws.amazon.com/security/security-bulletins/AWS-2024-...
| ikekkdcjkfke wrote:
| Github should probably remove the dopamine hits of green
| checkmarks etc. like in serious stock broker apps
| elintknower wrote:
| Candidly how would someone protect against a vulnerability like
| this?
| anononaut wrote:
| Compile all your packages from source would be a start.
| kosolam wrote:
| Jesus! Does anyone know if Debian stable is affected?
| ValdikSS wrote:
| It's not. Neither Ubuntu.
| anononaut wrote:
| Do you have a source my friend? I thought Ubuntu was built
| off of Debian testing or unstable
| bhaak wrote:
| I looked at the differences between the GitHub repository and
| released packages. About 60 files are in a release package that
| are not in the repo (most are generated files for building) but
| also some of the .po files have changes.
|
| That's devastating.
|
| If you don't build your release packages from feeding "git ls-
| files" into tar, you are doing it wrong.
| oxymoron290 wrote:
| Jai Tan's commit history on his github profile suggests he took
| off for Christmas, new years, and spring break. I smell an
| American.
| bloak wrote:
| Interesting. Is there also a pattern in the times of day? (I
| don't so much mean the times in commits done by the developer
| because they can be fake. I'd be more interested in authentic
| times recorded by GitHub, if any such times are publicly
| accessible.)
|
| Another thing would be to examine everything ever written by
| the user for linguistic clues. This might point towards
| particular native languages or a particular variant of English
| or towards there being several different authors.
| bombcar wrote:
| Someone said commits lined up with Beijing time, but I've not
| verified that.
|
| But that wouldn't count for much, someone employed by anyone
| could work any hours.
| rany_ wrote:
| Also git actually stores the timezone information. You
| could see it is consistently China time (GMT+8).
|
| P.S. could be Taiwanese as China and Taiwan share the same
| timezone.
|
| Below are links to the git mailbox files where you could
| see the timezone.
|
| From 2022:
|
| - https://github.com/tukaani-
| project/xz/commit/c6977e740008817...
|
| - https://github.com/tukaani-
| project/xz/commit/7c16e312cb2f40b...
|
| From 2024:
|
| - https://github.com/tukaani-
| project/xz/commit/af071ef7702debe...
|
| - https://github.com/tukaani-
| project/xz/commit/a4f2e20d8466369...
| rdtsc wrote:
| Sometimes you smell an American because someone wanted you to
| smell an American.
|
| Operating on a target region schedule doesn't seem particularly
| sophisticated, at least compared to the all the efforts put
| into this exploit.
| returningfory2 wrote:
| Another interesting data point: about 2 years ago there was a
| clear pressure campaign to name a new maintainer:
| https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...
|
| At the time I thought it was just rude, but maybe this is when it
| all started.
| leosanchez wrote:
| How many people are involved in this ?
| masklinn wrote:
| Could be just a single person with a bunch of identities.
| IshKebab wrote:
| I would put money on government hackers. They're the sort
| of people that have the time to pull something like this
| off. Frankly I'm really surprised it isn't more common,
| though maybe it is and these guys were just super blatant.
| I would have expected more plausible deniability.
| JaDogg wrote:
| Good cop bad cop play maybe.
| jamespo wrote:
| "Jigar Kumar" seems to have disappeared
| Nathanba wrote:
| true, that is suspicious as well. A person that hasn't even
| created any bugs or issues suddenly has a big problem with
| the speed of development? Especially the way this was
| phrased: "You ignore the many patches bit rotting away on
| this mailing list. Right now you choke your repo. Why wait
| until 5.4.0 to change maintainer? Why delay what your repo
| needs?"
|
| "Why delay what your repo needs?" This sounds like scammer
| lingo
| matsemann wrote:
| Wow, people suck. I almost hope it's fake profiles urging the
| maintainer to take on a new member as a long con. Because I
| sincerely hope Jigar Kumar is not a real person behaving like
| that towards volunteers working for free.
| MaximilianEmel wrote:
| Has this affected OpenBSD at all?
| ikmckenz wrote:
| Seems the backdoor relied on Debian and others patching their
| copies of openssh to support systemd notifications, and this
| would obviously not be the case on OpenBSD.
|
| To be sure the current ports version of xz is 5.4.5:
| https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/ports/a...
|
| Although the maintainer was working on updating to 5.6.1, but
| this news broke before the diff was landed:
| https://marc.info/?l=openbsd-ports&m=171174441521894&w=2
| k8svet wrote:
| Wait, I'm on mobile. Did this partially slip by because of the
| ABSURD PRACTICE of publishing release.tarballs that do not 1:1
| correspond with source?
|
| Let me guess, autotools? I want to rage shit post but I guess
| I'll wait for confirmation first.
|
| EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.
| 5p4n911 wrote:
| The author (Jia Tan) also changed the xz.tukaani.org (actually
| the github.io, where the main contributor is, surprise, also
| them) release description to state all new releases are signed by
| their OpenPGP key. I'd guess that was one of the first steps to a
| complete project takeover.
|
| I hope Lasse Collin still has control of his accounts, though the
| CC on the kernel mailing list looks kind of suspicious to me.
| tree24005 wrote:
| What an insightful article! Thanks for sharing.
| 0xthr0w4 wrote:
| Out of curiosity I looked at the list of followers of the account
| who committed the backdoor.
|
| Randomly picked https://github.com/Neustradamus and looked at all
| their contributions.
|
| Interestingly enough, they got Microsoft to upgrade ([0],[1])
| `vcpkg` to liblzma 5.6.0 3 weeks ago.
|
| [0] https://github.com/microsoft/vcpkg/issues/37197
|
| [1] https://github.com/microsoft/vcpkg/pull/37199
| sroussey wrote:
| OMG: look at the other contributions. He is trying to take over
| projects and pushing some change to sha256 in a hundred
| projects.
|
| Example: https://github.com/duesee/imap-flow/issues/96
| masklinn wrote:
| This guy's interactions seem weird but it might just be
| because of the non-native english or a strange attitude, or
| he's very good at covering his track e.g. found a cpython
| issue where he got reprimanded for serially opening issues: h
| ttps://github.com/python/cpython/issues/115195#issuecomment..
| .
|
| But clicking around he seems to mostly be interacting with
| interest around these bits e.g. https://github.com/python/cpy
| thon/issues/95341#issuecomment-... or pinging the entire
| python team to link to the PR... of a core python developer:
| https://github.com/python/cpython/issues/95341#issuecomment-.
| ..
|
| If I saw that on a $dayjob project I'd pit him as an
| innocuous pain in the ass (overly excited, noisy,
| dickriding).
|
| Here's a PR from 2020 where he recommends / requests the
| addition of SCRAM to an SMTP client:
| https://github.com/marlam/msmtp/issues/36 which is basically
| the same thing as the PR you found. The linked documents seem
| genuine, and SCRAM is an actual challenge/response
| authentication method for a variety of protocols (in this
| case mostly SMTP, IMAP, and XMPP): https://en.wikipedia.org/w
| iki/Salted_Challenge_Response_Auth...
|
| Although, and that's a bit creepy, he shows up in the edition
| history for the SCRAM page, the edit mostly seem innocent
| though he does plug his "state of play" github repository.
| gouggoug wrote:
| List of pull request requesting the updating to liblzma 5.6.0 [0]
|
| I wonder what amount of scrutiny all the accounts that proposed
| the upgrade should be put under.
|
| [0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests
| mikolajw wrote:
| Tukaani website states "jiatan" as the nickname of the malicious
| code committer on Libera Chat.
|
| WHOWAS jiatan provided me the following information:
|
| jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163
| :actually using host jiatan jiatan :was logged in as jiatan
| tungsten.libera.chat :Fri Mar 14:47:40 2024
|
| WHOIS yields nothing, the user is not present on the network at
| the moment.
|
| Given that 185.128.24.163 is covered with a range-block on the
| English Wikipedia, it appears this is a proxy.
| chrononaut wrote:
| > it appears this is a proxy.
|
| Yes, that IP address appears associated with witopia[.]net,
| specifically vpn.singapore.witopia[.]net points to that IP
| address.
| circusfly wrote:
| Waiting for the new YouTube videos on this. "Woah! Linux has a
| back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022
| isn't affected.
| autoexecbat wrote:
| I'm really curious about if the act of injecting a backdoor into
| OSS software is legal/illegal ?
|
| Are they somehow in the clear unless we can show they actively
| exploited it?
| mnau wrote:
| Probably depends on criminal code a country. Mine does (EU
| country):
|
| > Section 231 Obtaining and Possession of Access Device and
| Computer System Passwords and other such Data
|
| > (1) Whoever with the intent to commit a criminal offence of
| Breach of secrecy of correspondence [...] or a criminal offence
| of Unauthorised access to computer systems and information
| media [...] _produces_ , _puts into circulation_ , imports,
| exports, transits, offers, provides, sells, or otherwise makes
| available, obtains for him/herself or for another, or handles
|
| > a) a device or _its component_ , process, instrument or any
| other means, including _a computer programme designed or
| adapted for unauthorised access to electronic communications
| networks, computer system_ or a part thereof, or
|
| > b) a computer password, access code, data, process or any
| other similar means by which it is possible to gain access to a
| computer system or a part thereof,
|
| shall be sentenced .. (1 year as an individual, 3 years as a
| member of a organized group)
| vhiremath4 wrote:
| My favorite part was the analysis of "I'm not really a security
| researcher or reverse engineer but here's a complete breakdown of
| exactly how the behavior changes."
|
| You only get this kind of humility when you're working with
| absolute wizards on a consistent basis.
| rpigab wrote:
| I'd love to be at Microsoft right now and have the power to
| review this user's connection history to Github, even though VPN
| exists, many things can be learned from connection habits, links
| to ISPs, maybe even guess if VPNs were used, roundtrip time on
| connections can give hints.
|
| I really don't think some random guy wants to weaken ssh just to
| extract some petty ransomware cash from a couple targets.
| bananapub wrote:
| people are mis-reading the Debian bug report:
| https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
|
| it wasn't the apparently newly-created identity "Hans Jansen"
| just _asking_ for a new version to be uploaded, it was "Hans
| Jansen" _providing_ a new version to be uploaded as a non-
| maintainer-upload - Debian-speak for "the maintainer is AWOL,
| someone else is uploading their package". if "Hans Jansen" is
| another attacker then they did this cleverly, providing the new -
| compromised - upstream tarballs in an innocent-looking way and
| avoiding anyone examining the upstream diff.
| vasili111 wrote:
| Could anyone please tell me if current stable version of Debian
| has that backdoor or not?
| yabones wrote:
| Debian stable has 5.4.1, the backdoored versions are
| 5.6.0-5.6.1
|
| https://packages.debian.org/bookworm/xz-utils
|
| https://packages.debian.org/bookworm/liblzma5
| anononaut wrote:
| Debian Stable seems to be in the clear.
|
| https://lists.debian.org/debian-security-announce/2024/msg00...
| teddyh wrote:
| It does _not_ contain the backdoor: <https://security-
| tracker.debian.org/tracker/CVE-2024-3094<
| kazinator wrote:
| Doesn't this call for criminal charges?
| mnau wrote:
| Good luck finding him/her.
|
| GitHub probably already gave feds all logs and IPs, but I would
| bet 100:1 that it's all going to be a VPN or something like
| that.
| Unfrozen0688 wrote:
| Time for the west to take Linux back.
| ozgune wrote:
| I read through the entire report and it gradually got more
| interesting. Then, I got to the very end, saw Andres Freund's
| name, and it put a smile on my face. :)
|
| Who else would have run a PostgreSQL performance benchmark and
| discover a major security issue in the process?
| mik1998 wrote:
| Personally, I use lzip ever since I read
| https://www.nongnu.org/lzip/xz_inadequate.html Seems like the
| complexity of XZ has backfired severely, as expected.
| lacoolj wrote:
| What a disappointment.
|
| It's something always in the back of our minds as developers
| using public libraries, but when something like this happens,
| non-developers that hear about it start to associate it with the
| rest of the open-source community.
|
| It's essentially a terrorist attack on developer experience.
| Thankfully, management doesn't follow the same approach as the
| TSA.
| Zigurd wrote:
| "Lasse Collin," as other posters here have found, does not seem
| to exist as an experienced coder. Oddly, there is a Swedish jazz
| musician named Lasse Collin, which would otherwise be one of
| those names, especially the last name, that would stick out.
| Instead it is buried under a lot of mentions of a musician.
| rany_ wrote:
| Searching for my real name on Google doesn't return anything
| either, I don't think this means anything.
| Zigurd wrote:
| Lasse Collin the contributor is findable, especially if you
| add "tukaani" to the search. But not in any other context,
| unless that's what old jazz musicians do in their retirement.
| rany_ wrote:
| I don't think that's what they meant. The idea is to find
| information about their personal life, not OSS
| contributions. Something that proves they're a real person.
| akyuu wrote:
| Lasse Collin has been working on xz for decades:
| https://sourceforge.net/p/sevenzip/discussion/45797/thread/0...
|
| Now, whether his GitHub account is currently being controlled
| by him is another question.
|
| Also, for some more context: In 2022, Lasse said he was
| struggling to work on xz and was looking for maintainers, and
| mentioned Jia Tan: https://www.mail-archive.com/xz-
| devel@tukaani.org/msg00567.h...
| nateskulic wrote:
| Fairly deep bugs for a Bazaar.
| alathers wrote:
| Thank the gods I didn't plan on having a life this weekend
| userbinator wrote:
| Looking at how many requests to update to the backdoored version
| have been made, I wonder if the fact that many people (including
| developers) have been conditioned to essentially accept updates
| as "always-good" is a huge contributing factor in how easy it is
| to spread something like this.
|
| The known unknowns can be better than the unknown unknowns.
| kapouer wrote:
| Both https://github.com/tukaani-project members accounts have
| been suspended. (to see that, you can list the followers of each
| account).
| dmarto wrote:
| Kinda relevant, as I saw few comments about how safer languages
| are the solution.
|
| Here[0] is a very simple example, that shows how easy such supply
| chain attacks are in Rust; and lets not forget that there was a
| very large python attack just a few days ago[1].
|
| [0] - https://github.com/c-skills/rust1
|
| [1] - https://checkmarx.com/blog/over-170k-users-affected-by-
| attac...
| mrcus wrote:
| I am very concerned about Rust.
|
| Rust's "decision" to have a very slim standard library has
| advantages, but it severely amplifies some other issues. In Go,
| I have to pull in zero dependencies to make an HTTP request. In
| Rust, pulling reqwest pulls in at least 30 distinct packages
| (https://lib.rs/crates/reqwest). Date/time, "basic" base64,
| common hashing or checksums, etc, they all become supply chain
| vectors.
|
| The Rust ecosystem's collective refusal to land stable major
| versions is one of the amplifying issues. "Upgrade fatigue"
| hits me, at least. "Sure, upgrade ring to 0.17" (which is
| effectively the 16th major version). And because v0.X versions
| are usually incompatible, it's not really possible to opt not
| to upgrade, because it only takes a short while before some
| other transitive dependency breaks because you are slow to
| upgrade. I recently spent a while writing my code to support
| running multiple versions of the `http` library, for example
| (which, to be fair, did just land version 1.0). My NATS library
| (https://lib.rs/crates/async-nats) is at version 34. My
| transitive base64 dependency is at version 22
| (https://lib.rs/crates/base64).
|
| This makes it nearly impossible for me to review these
| libraries and pin them, because if I pin foo@0.41.7, and bar
| needs foo@0.42.1, I just get both. bar can't do =>0.41, because
| the point of the 0.X series is that it is not backwards
| compatible. It makes this process so time consuming that I
| expect people will either just stop (as if they did) reviewing
| their dependencies, or accept that they might have to reinvent
| everything from URL parsing to constructing http headers or
| doing CRC checks.
|
| Combine this with a build- and compile-time system that allows
| completely arbitrary code execution, which is routinely just a
| wrapper for stuff like in the zx attack (look at a lot of the
| low-level libs you inevitably pull in). Sure, the build scripts
| and the macro system enables stuff like the amazing sqlx
| library, but said build and macro code is already so hard to
| read, it really takes proper wizardry to properly understand.
| mrcoffee4u wrote:
| can someone ELI5 ?
| jchoksi wrote:
| The two active maintainers seem to be: Lasse Collin
| <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>
|
| Searching DDG for "jiat0218" I came across a blog post which I
| found weird. Seems to be dated: 2006-05-03
|
| Blog post: "KusoPai Mai .You Ling Qi De Kuai Zi - Que Xiao Hao "
| <https://char.tw/blog/post/24397301>
|
| Internet Archive link:
| <https://web.archive.org/web/20240329182713/https://char.tw/b...>
|
| The contents of the page when translated seems to be about
| jiat0218 auctioning a pair of spiritual chopsticks as a prank.
|
| The blog entry is basically a QA between jiat0218 and various
| other people about these chopsticks.
|
| If Jia Tan does turn out to be a compromised maintainer working
| for a state actor then some of the content on the blog page can
| be viewed in a more sinister way (i.e. spycraft / hacks for sale
| etc.).
|
| Example question 38: Question 38
| accounta066 (3): Are these chopsticks really that good? I kind of
| want to buy them! But I recently sent money for online
| shopping but didn't receive anything. It's very risky;
| currently jiat0218 you don't have any reviews, you can
| interview me. Do you want to hand it over?! ... A sincere buyer
| will keep it. Reply to jiat0218 (4):
| First of all, I would like to express my condolences to you for
| your unfortunate experience! What can I say about this kind of
| thing...My little sister has always been trustworthy.
| What's more, this is a pair of spiritual chopsticks, so
| I hope to have a good one. It's the beginning! As you can see,
| my little sister is very careful and takes her time when
| answering your questions. Except for the two messages
| that were accidentally deleted by her, she always answers
| your questions. If this still doesn't reassure you, then I
| can only say that I still have room to work hard. You are still
| welcome to bid... ^_^
|
| Note however, it could all just be what it purports to be which
| is a prank auction of spiritual chopsticks.
| fragmede wrote:
| Chopsticks could also be a codeword for something. Maybe some
| sort a backdoor into a system somewhere.
| dboreham wrote:
| Something about this I found surprising is that Linux distros are
| pulling and packaging pre-built binaries from upstream projects.
| I'd have expected them to build from source.
| richardwhiuk wrote:
| They were pulling a tarball from upstream and building it - the
| tarball was compromised.
| haolez wrote:
| I'm not trying to troll, but I'm wondering if a distro like
| Gentoo is less susceptible to such attacks, since the source code
| feels more transparent with their approach. But then again, it
| seems that upstream was infected in this case, so I'm not sure if
| a culture of compiling from source locally would help.
| stephc_int13 wrote:
| I guess that rewriting liblzma in Rust would _not_ have prevented
| this backdoor. But would have likely increased the confidence in
| its safety.
|
| Using the build system (and potentially the compiler) to insert
| malicious backdoors is far from a new idea, and I don't see why
| this example would the only case.
| anonymous-panda wrote:
| Don't know all the details and rust isn't immune to a build
| attack, but stuff like that tends to stand out a lot more I
| think in a build.rs than it would in some m4 automake soup.
___________________________________________________________________
(page generated 2024-03-29 23:00 UTC)