[HN Gopher] Fifty Years of Open Source Software Supply Chain Sec...
___________________________________________________________________
Fifty Years of Open Source Software Supply Chain Security
Author : yarapavan
Score : 91 points
Date : 2025-04-07 18:04 UTC (4 hours ago)
(HTM) web link (queue.acm.org)
(TXT) w3m dump (queue.acm.org)
| aadhavans wrote:
| A very well-written piece. The section on funding open source is
| as relevant as it's ever been, and I don't think we've learnt
| much since last year.
|
| As the proportion of younger engineers contributing to open-
| source decreases (a reasonable choice, given the state of the
| economy), I see only two future possibilities:
|
| 1. Big corporations take ownership of key open-source libraries
| in an effort to continue their development.
|
| 2. Said key open-source libraries die, and corporations develop
| proprietary replacements for their own use. The open source scene
| remains alive, but with a much smaller influence.
| bluGill wrote:
| Unfortunately I have no clue how to get a company to put money
| into the open source we use. Not just my current company, but
| any company. I've sometimes been able to get permission to
| contribute something I build on company time, but often what I
| really want is someone on the project to spend a year or two
| maintaining it. Do the boring effort of creating a release.
| Write that complex feature everyone (including me) wants.
|
| In decades past companies you to pay for my license for Visual
| Studio (I think of a MSDN subscription), clear case, a dozen
| different issue/work trackers. However as soon as an open
| source alternative is used I don't know how to get the money
| that would have been spent to them.
|
| Come to think of it I'm maintainer of a couple open source
| projects that I don't use anymore and I don't normally bother
| even looking at the project either. Either someone needs to pay
| me to continue maintaining it (remember I don't find them
| useful myself so I'm not doing it to scratch an itch), or
| someone needs to take them over from me - but given xz attacks
| I'm no longer sure how to hand maintenance over.
| johngossman wrote:
| In my prior career I talked to many companies about open
| source usage. If you tell them they are running an
| unsupported database or operating system in production, they
| will often see the value of buying support. But it is much
| harder to get them to pay for non-production stuff,
| especially development tools. And even if you find an
| enlightened manager, getting budget to pay a maintainer for a
| project is very difficult to even explain.
|
| "We're paying for contract development? But it's not one of
| our products and we'll have no rights to the software?
| They'll fix all the bugs we find, right? Right?" This a hard
| conversation at most companies, even tech companies.
| ghaff wrote:
| Development tools was almost always a tough standalone
| business even before open source became so prevalent.
| ndiddy wrote:
| At companies where I've worked, all of the money we've put
| into open source has been in contracting the developer(s) to
| add a feature we needed to the upstream version. Of course,
| this means that we didn't really fund ongoing maintenance on
| anything we used that had all the features we needed.
| lrvick wrote:
| My company, Distrust, exists to produce, support, and fund our
| open source security tools.
|
| So far our core full time team of 3 gets to spend about half
| our time consulting/auditing and half our time contributing to
| our open projects that most of our clients use and depend on.
|
| The key is for companies to have visibility into the current
| funding status of the software they depend on, and
| relationships with maintainers, so they can offer to fund
| features or fixes they need instead of being blocked.
|
| https://distrust.co
| throw83848484 wrote:
| Sadly, as OS dev, I see third way: development behind closed
| doors.
|
| With AI and CV reference hunting, number of contributions is
| higher than ever. Open-source projects are basically spammed,
| with low quality contributions.
|
| Public page is just a liability. I am considering to close
| public bugzilla, git repo and discussions. I would just take
| bug reports and patches from very small circle of customers and
| power users. Everything except release source tarball, and
| short changelog would be private!
|
| Open-source means you get a source code, not free customer and
| dev support!
| ozim wrote:
| I think big corporations will take ownership - well not
| directly but via paying to foundations and it already is the
| case.
|
| Second thing is there are bunch of things corporations need to
| use but don't want to develop on their own like SSH.
|
| There is already too much internal tooling inside of big
| corporations that is rotting there and a lot of times it would
| be much better if they give it out to a foundation - like
| Apache foundation where projects go to die or limp through.
| mlinksva wrote:
| Good article for what it covers, but sadly does not cover
| isolation/sandboxing/least privilege.
| Alive-in-2025 wrote:
| Yes. The crucial issue to me is the increasing frequency of
| attacks where some piece of open source gets an update -
| leading to endless hidden supply chain attacks.
|
| I don't see anything that is going to block this from getting
| worse and worse. It became a pretty common issue that I first
| heard about with npm or node.js and their variants, maybe
| because people update software so much there and have lots of
| dependencies. I don't see a solution. A single program can have
| huge numbers of dependencies, even c++ or java programs now.
|
| It's not new, here's one from 6 years ago on c++ -
| https://www.trendmicro.com/en_us/research/19/d/analyzing-c-c...
|
| Don't forget log4j -
| https://www.infoworld.com/article/3850718/developers-
| apply-t..., points to this recent paper
| https://arxiv.org/pdf/2503.12192
| bitwize wrote:
| Indeed. In 2020s, if you're not sandboxing each thing, and then
| sandboxing each _library_ the thing depends on, you 're running
| with way too many opportunities for vulnerability.
| LtWorf wrote:
| Well said! How?
| bitwize wrote:
| I have no freaking idea. Needless to say I don't think our
| current operating systems are up to the task of _actually_
| being secure. You have to be able to somehow dynamic-link
| in a library whilst only giving calls into that library
| certain permissions /capabilities... which I don't think
| even Windows can do.
| huijzer wrote:
| Didn't Jess Frazelle have most of her dependencies
| running inside lots of Docker containers for a while? She
| went pretty far and also kept it up for a long time.
| E.g., https://blog.jessfraz.com/post/docker-containers-
| on-the-desk...
| GuinansEyebrows wrote:
| Ignorant reply here, but would openbsd's `pledge` and
| `unveil` sorta cover what you're talking about?
| LtWorf wrote:
| At the library level? Not as far as I know...
| gizmo686 wrote:
| Forget OS support, is that something that modern CPUs can
| support efficiently? As far as I can tell, enforcing a
| security boundary across libraries would require changing
| the page table twice for every library call, which seems
| like a big performance hit.
| ndiddy wrote:
| > The OpenSSH project is careful about not taking on unnecessary
| dependencies, but Debian was not as careful. That distribution
| patched sshd to link against libsystemd, which in turn linked
| against a variety of compression packages, including xz's
| liblzma. Debian's relaxing of sshd's dependency posture was a key
| enabler for the attack, as well as the reason its impact was
| limited to Debian-based systems such as Debian, Ubuntu, and
| Fedora, avoiding other distributions such as Arch, Gentoo, and
| NixOS.
|
| Does Fedora use Debian's patch set for sshd, or a similar patch
| set that adds libsystemd?
|
| Edit: It looks like Fedora wasn't affected because the backdoor
| triggered a valgrind test failure, so they shipped it with a flag
| that disabled the functionality that was backdoored. Seems like
| they lucked out.
| https://lists.fedoraproject.org/archives/list/devel@lists.fe...
| lrvick wrote:
| Great coverage, however it failed to mention code review and
| artifact signing as well as full source bootstrapping which are
| fundamental defenses most distros skip.
|
| In our distro, Stagex, our threat model assumes at least one
| maintainer, sysadmin, or computer is compromised at all times.
|
| This has resulted in some specific design choices and practices:
|
| - 100% deterministic, hermetic, reproducible
|
| - full source bootstrapped from 180 bytes of human-auditable
| machine code
|
| - all commits signed by authors
|
| - all reviews signed by reviewers
|
| - all released artifacts are multi-party reproduced and signed
|
| - fully OCI (container) native all the way down "FROM scratch"
|
| - All packages easily hash-locked to give downstream software
| easy determinism as well
|
| This all goes well beyond the tactics used in Nix and Guix.
|
| As far as we know, Stagex is the only distro designed to strictly
| distrust maintainers.
|
| https://stagex.tools
| AstralStorm wrote:
| Good step.
|
| It doesn't distrust the developers of the software though, so
| does not fix the biggest hole. Multiparty reproduction does not
| fix it either, that only distrusts the build system.
|
| The bigger the project, the higher the chance something slips
| through, if even an exploitable bug. Maybe it's the developer
| themselves being compromised, or their maintainer.
|
| Reviews are done on what, you have someone reviewing clang
| code? Binutils?
| amiga386 wrote:
| A lovely article, but one section definitely needs a [citation
| needed]
|
| > (OpenSSL is written in C, so this mistake was incredibly easy
| to make and miss; in a memory-safe language with proper bounds
| checking, it would have been nearly impossible.)
| package main import "fmt" type CmdType
| int const ( WriteMsg CmdType = iota
| ReadMsg ) type Cmd struct { t
| CmdType d []byte l int }
| var buffer [256]byte var cmds = []Cmd{
| Cmd{WriteMsg, []byte("Rain. And a little ice. It's a damn good
| thing he doesn't know how much I hate his guts."), 88},
| Cmd{WriteMsg, []byte("Rain. And a little ice."), 23},
| Cmd{ReadMsg, nil, 23}, Cmd{ReadMsg, nil, 88}, //
| oops! } func main() { for c :=
| range cmds { if cmds[c].t == WriteMsg {
| copy(buffer[:], cmds[c].d[:cmds[c].l]) } else if
| cmds[c].t == ReadMsg {
| fmt.Println(string(buffer[:cmds[c].l])) }
| } }
|
| The heartbleed problem was that user-controlled input could say
| how long it was, separate from how long it _actually_ was.
| OpenSSL then copied the (short) thing into a buffer, but returned
| the (long) thing, thus revealing all sorts of other data it was
| keeping in the same buffer.
|
| It wasn't caught because OpenSSL had built its own buffer/memory
| management routines _on top_ of the actual ones provided by the
| language (malloc, memcpy, realloc, free), and all sorts of unsafe
| manipulations were happening inside one big buffer. That buffer
| could be in a language with perfect memory safety, the same flaw
| would still be there.
| neuroelectron wrote:
| Very suspicious article. Sounds like the "nothing to see here
| folks, move along" school of security.
|
| Reproducibility is more like a security smell; a symptom you're
| doing things right. Determinism is the correct target and subtly
| different.
|
| The focus on supply chain is a distraction, a variant of The
| "trusting trust" attack Ken Thompson described in 1984 is still
| among the most elegant and devastating. Infected development
| toolchains can spread horizontally to "secure" builds.
|
| Just because it's open doesn't mean anyone's been watching
| closely. "50 years of security"? Important pillars of OSS have
| been touched by thousands of contributors with varying levels of
| oversight. Many commits predate strong code-signing or provenance
| tracking. If a compiler was compromised at any point, everything
| it compiled--including future versions of itself--could carry
| that compromise forward invisibly. This includes even "cleanroom"
| rebuilds.
| lrvick wrote:
| The best defense we have against the Trusting Trust attack is
| full source bootstrapping, now done by two distros: Guix and
| Stagex.
| AstralStorm wrote:
| No you do not. If you have not actually validated each and
| every source package your trust is only related to the
| generated binaries corresponding to the sources you had. The
| trusting trust attack was deployed against the _source code_
| of the compiler, poisoning specific binaries. Do you know if
| GCC 6.99 or 7.0 doesn 't put a backdoor in some specific
| condition?
|
| There's no static or dynamic analysis deployed to enhance
| this level of trust.
|
| The initial attempts are simulated execution like in
| valgrind, all the sanitizer work, perhaps difference on the
| functional level beyond the text of the source code where
| it's too easy to smuggle things through... (Like on an
| abstracted conditional graph.)
|
| We cannot even compare binaries or executables right given
| differing compiler revisions.
| neuroelectron wrote:
| Besides full source boostrapping which could adopt
| progressive verification of hardware features and
| assumption of untrusted _hardware_ , integration of Formal
| Verification into the lowest levels of boostrapping is a
| must. Bootstap _security_ with the compiler.
|
| This won't protect against more complex attacks like RoP or
| unverified state. For that we need to implement simple
| artifacts that are verifiable and mapped. Return to more
| simple return states (pass/error). Do error handling
| external to the compiled binaries. Automate state mapping
| and combine with targeted fuzzing. Systemd is a perfect
| example of this kind of thing, what not to do: internal
| logs and error states being handled by a web of
| interdependent systems.
| AstralStorm wrote:
| RoP and unverified state would at least be highlighted by
| such an analysis. Generally it's a lot of work and we
| cannot quite trust fully automated systems to keyword it
| to us... Especially when some optimizer changes between
| versions of the compiler. Even a single compile flag can
| throw the abstract language upside down, much less the
| execution graph...
|
| Fuzzing is good but probabilistic. It is unlikely to hit
| on a deliberate backdoor. Solid for finding bugs though.
| edoceo wrote:
| One of my struggles is to get docker to lockdown which images it
| loads. I'd like to only pull from my own blessed registry and it
| seems Docker wants to always go back to theirs.
|
| For other "package" managers (eg: CPAN, Debian) I can point to my
| own archive and be sure everything I manage down stream gets the
| blessed bits.
|
| I basically have a huge archive/mirror for the supply chain for
| my perl, PHP, JavaScript, etc.
|
| If anyone has pro tips on how to "lock" docker to one registry
| that would be cool.
___________________________________________________________________
(page generated 2025-04-07 23:00 UTC)