[HN Gopher] The xz attack shell script
___________________________________________________________________
The xz attack shell script
Author : todsacerdoti
Score : 231 points
Date : 2024-04-02 09:05 UTC (13 hours ago)
(HTM) web link (research.swtch.com)
(TXT) w3m dump (research.swtch.com)
| mzs wrote:
| I haven't seen Thomas Roccia's infographic mentioned here yet:
| https://twitter.com/fr0gger_/status/1774342248437813525
| xomodo wrote:
| Thx. Timeline shows attack begun by adding ignore entry in
| .gitignore file. That is hard to detect nowadays.
| moopitydoop wrote:
| This just gives a partial high-level look at how the exploit
| gets planted in liblzma, it doesn't cover how the exploit works
| or its contents at all.
| ihsoy wrote:
| That seems to be largely grasped at straws and connecting dots
| without reason.
|
| For example oss-fuzz was building xz by cloning the github repo
| directly. There was never a chance for oss-fuzz to discover the
| backdoor, because only the tarball had it, not the repo itself.
| So that oss-fuzz PR might genuinely just be a genuine thing
| unrelated to the backdoor.
| sp332 wrote:
| The ifunc part looked more legitimate, it was used to switch
| between optimized implementations of CRC. So that part was in
| the git repo. https://github.com/google/oss-fuzz/pull/10667
| _trampeltier wrote:
| Jia Tan asked distros to update quickly just before it became
| public. How possible is it, there is another account / person
| who learned earlyer from people around Andreas Freund, the
| backdoor would become public. How possible is it, there is
| still another insider around?
| OneLeggedCat wrote:
| Right. Way too much coincidence. Jia Tan found out that it
| was about to become public and threw a Hail Mary. How did he
| find out?
| freedomben wrote:
| If the stakes weren't so high, this would be a damn fun
| game of murder mystery.
| ptx wrote:
| There were also changes to systemd happening around that time
| which would have prevented the backdoor from working. See the
| timeline article by the same author linked in this one.
| benlivengood wrote:
| The only upside of finding this attack (aside from preventing it
| from being rolled out more widely) is that it gives a public
| example of a complex APT supply-chain attack. Rest assured there
| are more out there, and the patterns used will probably be
| repeated elsewhere and easier to spot.
|
| Obfuscated autoconf changes, multi-stage deployment, binary blobs
| (hello lib/firmware and friends, various other encoders/decoders,
| boot logos, etc), repo ownership changes, new-ish prolific
| committers in understaffed dependency libraries, magic values
| introduced without explanation.
| cassianoleal wrote:
| > a complex APT supply-chain attack
|
| What do you mean by APT? If you mean Debian's package manager,
| that's not what this attack was. This was done upstream and
| affected non-apt distros just as much.
|
| It's true that upstream is part of apt's supply chain but
| focussing on apt is misleading.
|
| edit: why the downvotes? I get from the responses that I was
| wrong but given how the exploit was initially found in a Debian
| system and a lot of people very quickly jumped on the "Debian
| patched a thing and broke security" bandwagon, I don't think it
| was much of a leap to wonder if that's what was meant.
|
| Acronyms and initialisms are not the best way to convey
| specific information.
| mttpgn wrote:
| Not apt the package manager-- it's an acronym for Advanced
| Persistent Threat
| supriyo-biswas wrote:
| > What do you mean by APT
|
| Advanced, persistent threat.
| haimez wrote:
| It stands for "Advanced Persistent Threat" -
| https://en.m.wikipedia.org/wiki/Advanced_persistent_threat
| danieldk wrote:
| I think Advanced Persistent Threat.
| nequo wrote:
| Others have answered your question about APT but FWIW I don't
| understand the downvotes. You were respectful and simply
| sought to clear up a misunderstanding.
| macintux wrote:
| I was heavily downvoted for supplying a link to a previous
| discussion on the same topic recently. I wouldn't worry much
| about it.
| MuffinFlavored wrote:
| > The changes to build-to-host.m4 weren't in the source repo,
| so there was no commit. > > The attacker had permissions to
| create GitHub releases, so they simply added it to the GitHub
| release tarball.
|
| What are some simple tweaks the "Debians" of the world can do
| to mitigate this kind of stuff?
|
| Not trust "hand-curated-by-possibly-malicious-maintainers"
| GitHub release tarballs, only trust git commits?
|
| Whitelist functions that are allowed to do IFUNC/ELF hooking to
| core OpenSSH functions?
| MarkSweep wrote:
| Add to that list suppression of warnings from valgrind and
| address sanitizer without any justification. And no tracking
| issue to follow up on fixing the problem so the suppression can
| be removed.
|
| Committing binary files to source control rather than including
| build commands to generate the files is a bit of a red flag.
| sanxiyn wrote:
| > The first difference is that the script makes sure (very sure!)
| to exit if not being run on Linux.
|
| The repeated check is indeed mysterious. My only hypothesis is
| that the attacker may have thought that it should look plausible
| as a test input to a compression library, hence repetition.
| TheBlight wrote:
| Is it though? The attacker probably has Linux x86 target(s) in
| mind and IFUNC support isn't guaranteed to work with other
| platforms.
| sanxiyn wrote:
| Checking for Linux makes sense. Doing the exact same check
| for Linux five times in a row is mysterious.
| tamimio wrote:
| It would be an interesting plot twist if the whole thing
| was an AI hallucination.
| neffy wrote:
| ...or a stalking horse by somebody in Microsoft's
| marketing division.
| ddalex wrote:
| How about an AI trying to make itself some spare CPU
| cycles available ?
| mercurialuser wrote:
| It can be to make space for script changes: you may overwrite
| the first bytes of the script.
|
| Or just add some laziness.
| xyst wrote:
| Whoever hired these people to infiltrate this project spent a
| shit ton of hours building it in such a way to avoid detection
| this long. Fortunately, it was so complicated that they couldn't
| account for all of the factors.
|
| This is why open source will always outperform closed source in
| terms of security. Sure it pointed a massive flaw in the supply
| chain. Sure it highlights how under appreciated the foundational
| elements of FOSS, leaving maintainers subject to manipulation.
|
| But the same attack within a private company? Shit, probably
| wouldn't even need advanced obfuscation. With a large enough PR
| and looming deadlines, could easily sneak something like this
| with a minimal amount of effort into production systems. By the
| time company even realizes what happens, you are already flying
| off to a non-extradition country and selling the exfiltrated data
| on Tor (or dark web).
| skrtskrt wrote:
| off topic, but how many actual non-extradition countries are
| there these days (for the US)?
|
| Even countries we have strained relationships with will
| extradite as part of a negotiation when it's convenient for
| them politically.
|
| Russia probably wouldn't have even kept Snowden if it wasn't
| state secrets he revealed. If it was just some random data
| breach they would have prisoner-swapped him for an oligarch
| caught money laundering elsewhere.
| moopitydoop wrote:
| Our adversaries aren't going to extradite their citizens to
| the west. And obviously if it's a state-level actor they
| aren't going to extradite their own actors.
|
| As someone old enough to remember the tail end of the early
| hacker eras (e.g. Mitnick), I don't think anyone SHOULD be
| extradited over this, in particular if they're not being
| charged with actually using the exploit. Prosecute them where
| they live. Should they be prosecuted 193 times over in every
| state on Earth? What's the nexus? Github? Every server that
| installed the compromised xz utils?
|
| But you are right they will _deport_ (not extradite)
| foreigners who are inconvenient to them or when it is
| politically expedient to do so, if the foreigners are a
| nuisance, or as part of a political negotiation or prisoner
| exchange.
|
| The whole "extradition treaties" meme is a misconception. You
| will only get extradited if you flee to a country where you
| are a citizen (even dual citizen), or the ability to assert
| citizenship/nationality there. A fugitive fleeing to a
| country without an extradition treaty is subject to
| deportation. Every country on earth reserves the right to
| deny entry to or deport foreign fugitives. They might choose
| not to if someone is found to be a refugee, subject to the
| death penalty in a non-death-penalty state, etc.
| EasyMark wrote:
| Russia and China would never extradite for this, they would
| hire the people involved if they aren't already in their
| employ, and I wouldn't blame them. I'm not even sure if they
| could be charged with more than misdemeanor charges anyway.
| bsza wrote:
| I'm gonna cry survivorship bias here. How do we know how many
| similar attempts succeeded? How many of the ones discovered
| have been written off as honest mistakes? How can we know for
| sure that e.g. Heartbleed wasn't put there by someone on
| purpose (and that that someone isn't filthy rich now)?
|
| When you get hired to a private company, they know who you are.
| That's an immediate deterrent against trying anything funny. On
| Github, no one knows who you are. It might be harder to
| backdoor a project without getting noticed, but there is no
| risk to getting noticed. You can try as many times as you like.
| Jia Tan is still at large, and didn't even have to plan their
| whole life around living in a non-extraditing country (if they
| aren't in one already).
| xyst wrote:
| https://en.wikipedia.org/wiki/Industrial_espionage (aka
| corporate espionage)
|
| Happens all the time. Maybe it's a state actor. Maybe it's a
| disgruntled employee. It's just not in the same lens as you
| expect (software supply chain attack).
|
| Apple has trouble keeping the lid on top secret projects.
| Leaks about designs happen all the time prior to scheduled
| debut at WWDC.
|
| MS has had trouble in the past as well when it came to
| developing the Xbox (One?).
|
| "Owners of China-Based Company Charged With Conspiracy to
| Send Trade Secrets Belonging to Leading U.S.-Based Electric
| Vehicle Company" - https://www.justice.gov/usao-
| edny/pr/owners-china-based-comp...
|
| "Ex-Google engineer charged with stealing AI trade secrets
| while working with Chinese companies" -
| https://www.latimes.com/world-nation/story/2024-03-07/ex-
| goo...
|
| "The US Hits Huawei With New Charges of Trade Secret Theft" -
| https://www.wired.com/story/us-hits-huawei-new-charges-
| trade...
|
| "U.S. charges China-controlled company in trade secrets
| theft" - https://www.pbs.org/newshour/economy/u-s-charges-
| china-contr...
|
| "Ex-Google and Uber engineer Anthony Levandowski charged with
| trade secret theft" -
| https://www.theverge.com/2019/8/27/20835368/google-uber-
| engi...
|
| In the case of Levandowski, the dude didn't even bother with
| covering his tracks. Straight up just downloads trade secrets
| from source control and transfers them to personal computer -
| https://www.justice.gov/usao-ndca/press-
| release/file/1197991...
|
| In this small sample of cases, were the exfiltration attempts
| as elaborate as the "xz attack"? Probably not, but all of
| these people were vetted by internal procedures and that did
| nothing to stop them from acting maliciously.
|
| Forget back dooring the project when getting through the
| front door is so much easier! People are very relaxed in
| their walled off garden and cubicle :)
| yongjik wrote:
| I don't buy it. Most private companies would require a face-to-
| face meeting when you start. Even if you're fully remote, the
| expectation would be that at some point you would meet your
| coworkers at meetspace, most likely before you get to commit
| anything substantial. The ones that are worth penetrating will
| almost certainly require a background check.
|
| And then, once you're in, you cannot just commit to your target
| project willy-nilly, as your manager and your manager's manager
| will have other priorities. A for-profit company's frequently
| dysfunctional management would actually work as deterrent here:
| you don't just need to justify your code, you will have to
| justify why you were working on it in the first place.
| ThePowerOfFuet wrote:
| https://rigor-
| mortis.nmrc.org/@simplenomad/11218486968142017...
| xyst wrote:
| face to face, background checks. they are all superficial.
|
| A smooth talker can get you to relax your guard.
|
| Identity can be faked, especially if you have a nation state
| backing you.
| EasyMark wrote:
| Could be interesting as a class project to investigate other
| small relatively innocuous but near ubiquitous projects that
| could be treated the same way and investigate whether something
| similar could be done or has been done already. Just making a
| list of them would be useful if nothing else.
| bandrami wrote:
| As a longtime autotools-hater I would say this justifies my
| position, but any build system complex enough to be multiplatform
| is going to be inscrutable enough to let somebody slip something
| in like this. But it really is a problem that so much software is
| built with what's essentially become a giant cargo cult-style
| shell script whose pieces no one person understands all of.
| echoangle wrote:
| I think the common usage of bash and other languages with a
| dense and complicated syntax is the root problem here. If build
| scripts were written in python, you would have a hard time
| obfuscating backdoors, because everyone would see that it is
| weird code. Bash code is just assumed to be normal when you
| can't read it.
| kimixa wrote:
| I think the issue is that build systems are often build on a
| completely different language - python with some weird build
| framework is likely as inscrutable as bash and autotools to
| someone who doesn't use python.
|
| You _can_ write pretty straightforward readable bash, just as
| I 'm sure you can write pretty gnarly python. Especially if
| you're intentionally trying to obfuscate.
| echoangle wrote:
| The problem is that obfuscated bash is considered normal.
| If unreadable bash was not allowed to be committed, it
| would be much harder to hide stuff like this. But
| unreadable bash code is not suspicious, because it is kind
| of expected. That's the main problem in my opinion.
| kimixa wrote:
| Lots of autogenerated code appears "obfuscated" -
| certainly less clear than if a programmer would have
| written it directly.
|
| But all this relies on one specific thing about the
| autotools ecosystem - that shipping the generated code is
| considered normal.
|
| I know of no other build system that does this? It feels
| weird, like shipping cmake-generated makefiles instead of
| just generating them yourself, or something like scons or
| meson being packaged _with_ the tarball instead of
| requiring an eternal installation.
|
| That's a _lot_ of extra code to review, before you even
| get to any kind of language differences.
| owlbite wrote:
| I think the main issue is auto tools tries to support so
| many different shells/versions all with their own
| foibles, so the resulting cross-compatible code looks
| obfuscated to a modern user.
|
| Something built on python won't cover quite as wide a
| range of (obsolete?) hardware.
| freedomben wrote:
| This probably varies widely, because unreadable bash is
| absolutely _not_ considered normal, nor would pass code
| review in any of my projects.
|
| On a slightly different note, unless the application is
| written in python, it grosses me out to think of writing
| scripts in python. IMHO, if the script is more complex
| that what bash is good at (my general rule of thumb is do
| you need a data structure like an array or hash? then
| don't use bash), then use the same language that the
| application is written in. It really grosses me out to
| think of a rails application with scripts written in
| python. Same with most languages/platforms.
| sunshowers wrote:
| It's rather easy to monkeypatch Python into doing spooky
| things. For something like this you really want a language
| that can't be monkeypatched, like I think Starlark.
| echoangle wrote:
| Where are you going to hide your monkey patching though? As
| long as your code is public, stuff like this is always
| going to stand out, because no one writes weird magic one
| liners in python.
| sunshowers wrote:
| I'm not a security-oriented professional, but to me a
| place I could hide this logic is by secretly evaling the
| contents of some file (like the "corrupt archive" used in
| xz) somewhere in the build process, hiding it behind a
| decorator or similar.
| echoangle wrote:
| I'm not a security professional either, but that doesn't
| sound very plausible to me. If you assume a maintainer
| who checks every commit added to the codebase, he's
| hopefully blocking you the second he sees an eval call in
| your build script. And even a code audit should find
| weird stuff like that, if the code is pythonic and simple
| to read. And if it's not, it should not be trusted and
| should be treated as malicious.
| sunshowers wrote:
| Well, the threat model here is that a maintainer
| themselves is the saboteur.
| lijok wrote:
| Would stick out like a sore thumb
| cyanydeez wrote:
| This code was essentially monkey patched from a test
| script. Python automatically runs any code in a imported
| module, so not hard to see a chain of module imports that
| progressively modifies and deploys a similar structure.
| zmmmmm wrote:
| The way python's run time literally executes code when
| you import a module makes it seem pretty easy to taint
| things from afar. You only need to control a single
| import anywhere in the dependency hierarchy and you can
| reach over and override any code somewhere else.
| sunshowers wrote:
| Oh yeah, that's a fantastic point.
| matheusmoreira wrote:
| Any language can become a turing tarpit if you try hard
| enough.
| JonChesterfield wrote:
| Right there with you. It's really tempting to blame this entire
| thing existing on m4 but that's the trauma talking.
| rgmerk wrote:
| Yeah.
|
| I haven't used it for some time but autoconf always seemed like
| a horrible hack that was impossible to debug if it didn't work
| properly.
|
| That was bad enough back in the days where one was mostly
| concerned with accidents, but in more modern times things that
| are impossible to debug are such tempting targets for mischief.
| shp0ngle wrote:
| Every time someone explains to me autotools, the individual
| pieces sort of make sense, yet the result is always this
| inscrutable unreadable mess.
|
| I don't know why.
| cyanydeez wrote:
| Just the fact that you have multiple platforms suggests few
| people will fully understand the entire complex.
| jongjong wrote:
| The devil is in the detail and nothing obscures details like
| complexity.
|
| Same reason why I don't like TypeScript in its current form.
| It's not worth the extra complexity it brings.
___________________________________________________________________
(page generated 2024-04-02 23:00 UTC)