[HN Gopher] The modern packager's security nightmare
       ___________________________________________________________________
        
       The modern packager's security nightmare
        
       Author : summm
       Score  : 155 points
       Date   : 2021-02-20 13:12 UTC (1 days ago)
        
 (HTM) web link (blogs.gentoo.org)
 (TXT) w3m dump (blogs.gentoo.org)
        
       | paultopia wrote:
       | I feel like this confuses a lot of things by assuming an
       | extremely sophisticated end user. Sure, if you're a double-threat
       | dev/sysadmin using linux, then when some vulnerability gets
       | discovered in some dynamically linked library on your system, you
       | have the capacity to (a) receive information about that fact, and
       | (b) update it.
       | 
       | But now suppose you're an ordinary person. You use software.
       | Maybe you even have a windows machine.[1] Which is more likely to
       | actually get a security update to you?
       | 
       | (a) You have to update a single piece of software, which you know
       | you've installed, though a recognized distributional channel like
       | an app store or something, and all its dependencies come with it.
       | 
       | (b) You have to either learn what a DLL is and learn how to
       | update it and then hope that nothing you rely on breaks in some
       | mysterious way because of some dependency on a dependency on a
       | dependency on a dependency. Or you have to accept a whole
       | operating system update, assuming that the operating system comes
       | with a DLL update---and hence accepting all of the other crap
       | that comes with operating system updates from Microsoft (and
       | Apple), such as bugginess from complex updates, incompatibilities
       | between new versions of operating systems and software (or
       | hardware) that you rely on, new obnoxious security rules that you
       | might not agree to (looking at you, Cupertino), and massive
       | disruptions to your ability to actually use your computer to do
       | your work.
       | 
       | No thanks.
       | 
       | [1] Maybe this article is specifically targeted against the linux
       | ecosystem? If so, perhaps this issue is ameliorated somewhat, but
       | it still seems to put a fairly substantial burden on end users,
       | that seems to be inconsistent with actually letting non-experts
       | use linux OSes.
        
         | regularfry wrote:
         | Alternatively, realise that the problems in (b) are largely
         | solved for reasonable OSes where the vendor takes
         | responsibility both for automatically getting you security
         | patches and for keeping you working without major disruptions.
         | 
         | I'm not sure where you've got the idea that updates are all-or-
         | nothing, but some of us have been living a life you seem to
         | think can't exist for _decades_ at this point.
        
         | deadbunny wrote:
         | That sounds like a good argument for using package managers
         | with automatic security updates. The user doesn't need to be an
         | expert, just reboot when the system tells them to.
        
       | mixedCase wrote:
       | Yeah well, who's hungry?
       | 
       | No really, neither users nor developers care, nor should they.
       | I've been using Linux on the desktop for well over a decade and
       | I'm tired of seeing this plea for everything to behave exactly
       | like C or scripting languages because every distro wants to be
       | its own special snowflake and it would be too hard to adapt.
       | 
       | The world has changed and distros have to stop pretending it
       | hasn't. Flatpak, Nix, those are better models that reflect our
       | reality where developers don't want to worry about ten thousand
       | different distros with wildly different downstream configurations
       | producing different kinds of bugs, as well as the desire of users
       | of just getting the darned app.
       | 
       | If you're worried about security you must always, first, work
       | with upstream. Upstream should be the party responsible for its
       | users, not you. Upstream not fast enough and you want to go the
       | extra mile? Well then your packaging tool-belt should have
       | support for patching a series of libaries in an automated fashion
       | and rebuilding the dependency tree; and make sure that you return
       | to upstream as soon as possible.
       | 
       | If you want your downstream to diverge from upstream because you
       | want to act as a barrier as the old distros do, then you'll have
       | to accept the fact that you're _maintaining a fork_ and not
       | pretend to upstream that you 're distributing their work or that
       | your bug reports are compatible. Otherwise, again, just limit
       | yourself on distributing the latest upstream's stable release
       | with the bare minimum patching necessary for getting things to
       | work with the distro's chosen config.
       | 
       | With some luck, after the Linux community comes to terms with the
       | fact that the distro model must shift, we can begin to finally
       | share some packaging efforts between all distros and we can leave
       | much of the incompatibility bullshit in the past.
       | 
       | Imagine one day everyone just building from shared Nix
       | derivations? Very ironically, it would look a lot like Portage
       | and Gentoo's USE flags but with everyone building off those
       | generic derivations and offering binary caches for them.
        
       | kenniskrag wrote:
       | Do I need to recompile the shared library and the clients if one
       | of them needs a newer compiler or if the client is provided as
       | binary by the vendor?
        
       | ChrisMarshallNY wrote:
       | For an OS like Linux, the thing about static linking is 100%
       | spot-on.
       | 
       | However, for vetted systems, like app stores, this makes no
       | difference, as the only dylibs allowed to be patched at runtime,
       | are the OS-supplied (ABI) ones. The whole app needs to be
       | rebuilt, re-vetted, and rereleased.
       | 
       | Frankly, I would be worried about dylibs, but for a different
       | reason, as more than one hack has been done by hijacking dynamic
       | links. I'm trying to remember how it went, but there was once a
       | common practice for Mac programs to deliberately take advantage
       | of this, in order to add functionality.
        
       | bscphil wrote:
       | Over the last several weeks I was working on an essay about this
       | exact problem, including the connection between static linking
       | and bundling. This one is so well done that I probably won't even
       | publish it.
       | 
       | But I'll add this, for people who may not immediately see why
       | this is important. I think that the real danger these new
       | technologies represent is not inherently bad technology, but the
       | possibility of ecosystem damage. The distribution / maintainer /
       | package manager approach has proven to be an extremely reliable
       | way to get trustworthy software. Many of us love it and want to
       | see it stick around. And it's been possible because "upstream"
       | developers in the open source ecosystem have been willing (or
       | forced) to work with distributions to include their software. But
       | this seems to be changing. A highlight from my essay:
       | 
       | 'Many software projects are not good citizens in the open source
       | ecosystem. They are working with a model of development and
       | distribution that does not mesh well with how open source
       | software is predominantly produced. ... These [new] languages are
       | now developing _their own_ ecosystems, with their own
       | expectations for how software should be created and distributed,
       | how dependencies should be handled, and what a  "good piece of
       | software" looks like. Regardless, an increasing amount of
       | software is being built in these ecosystems, _including_ open
       | source software.
       | 
       | There might come a day in which open source is fractured. Two
       | different communities create two very different kinds of
       | software, which run on the same systems, but are created and
       | distributed in very different ways. I begin to worry that the
       | community I care about might not survive. Even if my ecosystem
       | continues on its way, I don't want this split to take place. I
       | think being part of the open source ecosystem is good for
       | software, and I think having all the software you could want
       | available within that ecosystem is good for users. If anything,
       | this essay is a call to those who agree to be more careful about
       | the software they create. Make sure it's something that Debian
       | maintainers could be proud to ship. If you're working on a Rust
       | program, for example, ask questions like "how can I make this
       | program as easy for maintainers to distribute as possible?" If
       | you have the ability to work on projects like Rust or Go, do what
       | you can to give their applications the ability to easily split
       | dependencies and support system provided libraries. Let's try to
       | make sure the software ecosystem we love is around for the next
       | generation.'
       | 
       | I'd also strongly recommend the essay "Maintainers Matter" for
       | another take on why this is so important.
       | http://kmkeen.com/maintainers-matter/ Or read about Debian's
       | recent problems with vendoring: https://lwn.net/Articles/842319/
        
         | hctaw wrote:
         | > The distribution / maintainer / package manager approach has
         | proven to be an extremely reliable way to get trustworthy
         | software. Many of us love it and want to see it stick around.
         | 
         | I disagree, it's proven to be inadequate for modern software
         | development and that's why these new languages/ecosystems are
         | springing up. The least reliable way to package and distribute
         | software is by relying on traditional package managers.
         | 
         | > Do what you can to give their applications the ability to
         | easily split dependencies and support system provided libraries
         | 
         | This is unrealistic. I do not trust system provided libraries
         | to function with my applications because I've been burned so
         | many times in the past.
         | 
         | > how can I make this program as easy for maintainers to
         | distribute as possible
         | 
         | By statically linking everything as much as possible and
         | shipping everything else in a self contained bundle with a
         | launcher that overrides any symbols that might inadvertently be
         | pulled in from the system.
         | 
         | The universe I'd like to live in is where the only use case for
         | dynamic linking are OS vendor APIs and cryptographically secure
         | functions like TLS. My dream package manager would whitelist
         | those system librarie and forbid distribution for any bundle
         | that does contain the shared objects with the symbols it needs.
        
           | bscphil wrote:
           | > I disagree, it's proven to be inadequate for modern
           | software development
           | 
           | Well, that's exactly why the OP (and my essay) are "anti"
           | modern software development in many ways. The view is that
           | we're moving away from the traditional open source ecosystem
           | and methods of software development with these new
           | technologies, which (to be clear) are _good_ technologies,
           | but were created mostly to solve problems that some large
           | corporations have, not to solve the problems that the open
           | source ecosystem has.
           | 
           | > The least reliable way to package and distribute software
           | is by relying on traditional package managers.
           | 
           | Not sure what you mean by this, but it's entirely untrue in
           | my experience. Anything I install with a package manager just
           | works, 100% of the time. Stuff I try to get any other way is
           | a shitshow, and the lack of "quality control" provided by
           | maintainers speaks for itself. I mean, just look at Android
           | apps or the Chrome extension store. Heaven forbid we go back
           | to the days of curl | bash off someone's website.
           | 
           | > By statically linking everything as much as possible and
           | shipping everything else in a self contained bundle with a
           | launcher that overrides any symbols that might inadvertently
           | be pulled in from the system.
           | 
           | I know you know this, but just to be clear, that's not a
           | solution to the problem of "making things easier for
           | maintainers to distribute", that's cutting maintainers out of
           | the loop. The whole point of my focus on ecosystems is that
           | this is something that I, as a user, don't want to happen.
        
             | hctaw wrote:
             | I'd disagree that the problem of large organizations are
             | different from the problems of the FOSS ecosystem.
             | Organizations just have a financial incentive to fix them,
             | the FOSS ecosystem does not. If mutually incompatible
             | dependencies and security updates breaking software weren't
             | problems for both corporate and FOSS ecoystems, these new
             | technologies wouldn't have needed to exist. They'd just use
             | the existing platforms.
             | 
             | And mind you, this is not a corporate/open source split.
             | The burgeoning ecosystems are also full of FOSS
             | technologies doing new and exciting things, they just don't
             | break when a dependency updates!
             | 
             | >Anything I install with a package manager just works, 100%
             | of the time
             | 
             | I run into issues with packages weekly. So much so I've
             | spent engineer days purging references to system packages.
             | It's universal too - yum, apt, pacman, brew, macports, I
             | have to make sure nothing tries to reference packages
             | installed outside a local working directory for an
             | application because of mutual incompatibilities. Maybe it's
             | because I'm trying to write software that runs on multiple
             | targets and not use software where someone else has already
             | spent the time and money to resolve these issues.
             | 
             | > I know you know this, but just to be clear, that's not a
             | solution to the problem of "making things easier for
             | maintainers to distribute", that's cutting maintainers out
             | of the loop. The whole point of my focus on ecosystems is
             | that this is something that I, as a user, don't want to
             | happen.
             | 
             | They should be cut out of the loop. Maintainers don't have
             | a right to dictate what design decisions I put into my
             | applications because they don't think it adds value (the
             | value is, it doesn't just run on their distro!). Another
             | comment in this thread put it better, maintainers shouldn't
             | place themselves in the development process.
        
               | ticviking wrote:
               | > maintainers shouldn't place themselves in the
               | development process.
               | 
               | Then perhaps we as developers should spend more time
               | thinking about and participating in the maintenance
               | process?
        
               | bscphil wrote:
               | > They should be cut out of the loop.
               | 
               | I disagree. This is how you get the Google Play store or
               | the "freeware" app marketplace. It sucks. As a user, I'm
               | quite happy to continue using a traditional distribution
               | even if it means I don't get to use a handful of flashy
               | programs by developers that disagree with the concept of
               | maintainers. So far that choice has been much more
               | positive than negative for me, and I'm doing my best (by
               | promoting the open source ecosystem) to keep it that way.
               | 
               | > Maintainers don't have a right
               | 
               | If you're creating open source software, they quite
               | literally do!
               | https://www.gnu.org/licenses/gpl-3.0.en.html
               | 
               | If you put in a bunch of crap that doesn't belong in an
               | application (ads, for example), I'm glad I have a
               | maintainer that can either strip this out thanks to the
               | GPL (or BSD / MIT etc), or else choose not to include
               | your app in the distribution at all.
        
               | sanxiyn wrote:
               | > a handful of flashy programs
               | 
               | This only works if "core" refuses to use Rust, so it's
               | not sustainable. We already can't build Firefox and GNOME
               | without Rust. Maybe Apache and curl next. And then?
        
             | alfiedotwtf wrote:
             | I've been on Debian since Potato, so I totally see what
             | you're saying. But...
             | 
             | > that's cutting maintainers out of the loop
             | 
             | Is this necessarily a bad thing? The market has seen the
             | need to fill a hole, and it seems to be working.
             | 
             | I first started with Slackware, and dependency nightmares
             | is what got me into Debian in the first place. Although
             | Debian is nice _because_ of its slow and stable base (which
             | makes me happy for production), I've recently moved to Arch
             | and have been so happy as it's brought back Slackware's
             | idea of getting as close to upstream as possible _and_ it
             | handles dependencies! And to be honest, I'm loving it. And
             | as an even added bonus, I'm getting more and more surprised
             | how a tonne of packages that I've installed _are_ Rust
             | apps.
             | 
             | So, coming back to your comment:
             | 
             | > that's cutting maintainers out of the loop
             | 
             | With systems like Arch that get us closer and closer to
             | upstream, are maintainers the unnecessary middlemen? Of
             | course they're not entirely redundant, but maybe a new
             | model of distros like Arch will be more commonplace in the
             | future
        
               | bscphil wrote:
               | Thoughtful comment, thanks. I'm an Arch user as well and
               | agree broadly with its approach to a desktop operating
               | system. (I.e. stick as closely to upstream as possible.)
               | 
               | That said, I disagree that this means maintainers are
               | unnecessary middlemen, even though their role on a
               | distribution like Debian is obviously more prominent. The
               | essay I linked to in my top level comment is actually by
               | an Arch maintainer, explaining why they still see
               | maintainers as playing an important role.
               | http://kmkeen.com/maintainers-matter/
        
             | rsj_hn wrote:
             | The problem is that the modern practices are being adopted
             | to meet the changing needs of this code, which weren't an
             | issue in the enthusiast linux and BSD communities, but are
             | an issue when open source code is being put in mission
             | critical professionally managed production environments on
             | AWS, or into cars and appliances.
             | 
             | The userland is much more complex now, so freezing
             | dependencies and bundling them in order to have a smaller
             | number of test cases may not have been necessary when the
             | environment was smaller and simpler. The reliability
             | expectations were lower, and there weren't big money
             | professional support contracts which required you to
             | validate your application against a set of well defined
             | environments. All that has changed.
        
             | oblio wrote:
             | The open source solutions are primarily for C/C++. A bit
             | for Perl, a bit for Python. But they haven't really moved
             | on. Java has been tacked on since forever. Same for .NET,
             | JavaScript, whatever.
             | 
             | And if you wanted your program propagated to all major
             | distros you'd have to wait a decade.
             | 
             | Nobody has time for that. Not corporations, not mom and pop
             | stores, and I doubt many hobbyists.
        
               | ticviking wrote:
               | Then you always have the option to package your software
               | for the ones you use and not worry about the rest.
               | 
               | If you want to make your program popular then packaging
               | is part of the process needed.
        
           | lucideer wrote:
           | > _The universe I 'd like to live in is where the only use
           | case for dynamic linking are OS vendor APIs and
           | cryptographically secure functions like TLS. My dream package
           | manager would whitelist those system librarie and forbid
           | distribution for any bundle that does contain the shared
           | objects with the symbols it needs._
           | 
           | This idea seems to be predicated on a belief that OS vendor
           | APIs and cryptographic libraries are the only attack surface
           | for serious user-affecting software exploits.
        
             | hctaw wrote:
             | They're obviously not, but they're the ones that package
             | managers can help mitigate automatically. The rest is going
             | to be up to the developers to patch.
        
           | nextos wrote:
           | Some of the things you mention are not incompatible with
           | package managers. Have you considered Nix?
           | 
           | I actually agree with the parent post in terms of strongly
           | preferring package managers to other means of distributing
           | software. I've always found Linux much easier to admin than
           | other OSes simply because of package managers.
        
             | kuratkull wrote:
             | Was really pumped after playing with Nix for a bit. But got
             | bitten twice in the first day. I tried to run my work
             | project using Nix. It needs Pythons python-prctl library -
             | that doesn't work in Nix. So that's a dud. Next I try to
             | use it as a Nim environment - Nim is unable to compile
             | anything in Nix due to some missing (bundled) Glibc symbol.
             | (Nim works nicely in my standard installation). The cool
             | "export full container" thing mentioned in the tutorial
             | failed for all, even the simplest cases. So I am kinda
             | disillusioned by Nix.
        
               | nextos wrote:
               | You can always emulate FHS environments for things that
               | are hard to turn into a Nix package without significant
               | effort.
               | 
               | Aside from that, Guix is an alternative implementation
               | that might have what you need.
        
         | [deleted]
        
         | edoceo wrote:
         | Based on this writing, I'm keen to read your full essay
        
         | jauer wrote:
         | > Two different communities create two very different kinds of
         | software, which run on the same systems, but are created and
         | distributed in very different ways.
         | 
         | This is where the disconnect is coming from. The distro
         | maintainers are coming from a world of multi-user systems where
         | backwards compatibility and updating deps without disturbing a
         | user's workload / forcing them to recompile is paramount.
         | 
         | Go (and a fair amount of rust/python work) come from the land
         | of CI/CD and, to a lesser extent, monorepos. When you are
         | rebuilding the world above a bare minimum of the OS literally
         | on every commit (or at least several times per day), it's
         | easier to reason about code that is running if you can look at
         | the commit that a bin was built from and know exactly what's
         | inside (including all deps).
        
           | bscphil wrote:
           | I agree. I think the difference has been that until recently
           | "the land of CI/CD" and so on has been certain segments of
           | the corporate world, and not how typical open source
           | developers did things*. So when the former developed new
           | technologies and new languages, they created build tools for
           | them that anticipated being used in the ways that _they_
           | usually produce software.
           | 
           | The "problem", in the sense that it's a problem, is that
           | these languages and related technologies are all pretty good!
           | And so it's understandable that many developers who would
           | traditionally be in the open source ecosystem want to use
           | them. As a result they end up creating software that can't
           | easily be shipped in traditional distributions. Ecosystem
           | fragmentation is the unavoidable result.
           | 
           | * By typical open source developers, I mean the sort of
           | developers (and their development practices) that produced
           | most of the software on my computer. I don't mean Firefox:
           | Mozilla and Google have much more standard corporate
           | development practices despite both producing quite a bit of
           | open source software.
        
       | choeger wrote:
       | Would it be feasible for rust to ship "re-link" scripts? That is,
       | when a cargo dependency gets updated, the distribution can check
       | whether packages code needs to be re-linked? Similarly, when
       | distributed code (say libssl) changes, the rust packages can be
       | re-linked by the distribution on-site?
        
         | charliesome wrote:
         | You can achieve this by recompiling the software. Cargo will
         | not recompile dependencies that have not changed.
        
       | Ericson2314 wrote:
       | This is a moronic by saying everything is equally bad.
       | 
       | 1. The linking strategy doesn't matter. Just rebuild everything.
       | The reason Rust and Haskell don't do dynamic linking is pervasive
       | inlining. This is required for many abstractions to work well,
       | and asking optimization boundaries to coincide with software
       | origin boundaries is stupid anyways.
       | 
       | The ABI arguement is stupid because replacing all the dynamic
       | libraries with the ABI changes is no worse than rebuilding
       | everything cause static linking. Therefor ignore that part and
       | just think about inlining.
       | 
       | 2. Lock files are fine as long as one can in fact bump the minor
       | versions. Yes, we should make tools to enforce minor versions are
       | non-breaking. And it's crazy we didn't have those from day one.
       | But until we do lock files are fine.
       | 
       | 3. Vendoring is pure evil.
        
         | Ericson2314 wrote:
         | The tl;dr is we always need reproducible and composible builds.
         | You can derive basically everything else from that.
        
       | nomercy400 wrote:
       | It's a risk calculation: do I want to risk being vulnerable to
       | 0days, or do I want to risk my application not running for any
       | user because a dependency changed its headers/api?
       | 
       | As software engineers we want to be in control of as much as
       | possible when running our application, to make it as
       | deterministic as possible. For that we select versions of our
       | dependencies, build it, and then test it, and if our tests pass,
       | we release it.
       | 
       | If we would let the OS determine the dependency versions, without
       | testing, what guarantees can we give that our code will work? Do
       | you write tests for future, unknown changes in libraries? Do you
       | write tests with Random, because that's the kind of
       | unpredictability you're introducing?
        
         | regularfry wrote:
         | Keeping the application working in the face of dependency
         | updates is the distro's job, if they're supported in doing it.
         | They won't be pushing dependency updates at random except to
         | channels explicitly marked as unstable (at least, assuming a
         | minimally sane distro).
         | 
         | I'm less familiar with Gentoo, but Debian-based distros ought
         | to be safe from that threat (and if anything, you ought to be
         | worries about the reverse problem). So your question becomes:
         | do I want to risk being vulnerable to zero-days?
        
       | hctaw wrote:
       | The day I stop static linking is the day I can compile on one
       | distro and ship to many without worrying about users reporting
       | loader errors. That day is not today.
       | 
       | Until then, I'll keep doing it, because it saves me time and
       | money.
       | 
       | I don't really buy that dynamic linking all the things is such a
       | boon to security. But I'll link to this:
       | https://drewdevault.com/dynlib
       | 
       | > Not including libc, the only libraries which had "critical" or
       | "high" severity vulnerabilities in 2019 which affected over 100
       | binaries on my system were dbus, gnutls, cairo, libssh2, and
       | curl. 265 binaries were affected by the rest.
       | 
       | > The total download cost to upgrade all binaries on my system
       | which were affected by CVEs in 2019 is 3.8 GiB. This is reduced
       | to 1.0 GiB if you eliminate glibc.
       | 
       | > It is also unknown if any of these vulnerabilities would have
       | been introduced after the last build date for a given statically
       | linked binary; if so that binary would not need to be updated.
       | Many vulnerabilities are also limited to a specific code path or
       | use-case, and binaries which do not invoke that code path in
       | their dependencies will not be affected. A process to ascertain
       | this information in the wake of a vulnerability could be
       | automated.
        
         | sigjuice wrote:
         | _The day I stop static linking is the day I can compile on one
         | distro and ship to many without worrying about users reporting
         | loader errors. That day is not today._
         | 
         | Maybe do the trendy thing and link your whole distro together
         | with your app (a.k.a containers)?
        
           | forrestthewoods wrote:
           | The fact that containers exist and are prevalent is a damning
           | indictment on the Linux model imho.
           | 
           | Producing software that can simply launch is so wildly
           | complex that the solution is... to snapshot an entire
           | operating system install to run the application.
           | 
           | That's INSANE. But it's unfortunately required.
           | https://xkcd.com/1987/
           | 
           | > Now, for the worst of all -- one that combines all the
           | aforementioned issues, and adds even more. Bundling (often
           | called vendoring in newspeak) means including the
           | dependencies of your program along with it. The exact
           | consequences of bundling vary depending on the method used.
           | 
           | Let's consider this statement at face value. The WORST thing
           | a program can do is... include the things the program needs
           | to run? So programs should... NOT include the things they
           | REQUIRE to run?
           | 
           | I get it. I understand the argument. But if you take just a
           | half step back it should be clear how utterly broken the
           | whole situation is.
        
         | orra wrote:
         | > I don't really buy that dynamic linking all the things is
         | such a boon to security.
         | 
         | Agreed. It's really not a panacea. When you upgrade a library,
         | you probably want to restart the running applications that
         | depend upon this. Dynamic linking won't save you.
         | 
         | The solution is having a graph of your dependencies! IIRC,
         | NixOS gets this right. I don't think Debian's apt did?
        
         | AshamedCaptain wrote:
         | > But I'll link to this: https://drewdevault.com/dynlib
         | 
         | We already discussed this in HN
         | https://news.ycombinator.com/item?id=23654353 . Both data and
         | conclusions are utterly suspect.
        
       | [deleted]
        
       | anotherhue wrote:
       | Correct me if I'm wrong (really) but isn't static linking mostly
       | a problem for packagers? The less a package maintainer modifies
       | the application the better IMO.
       | 
       | If application A and B rely on dependency D, which turns out to
       | have a vulnerability, fixed in D', Then _why_ do we think it is
       | anyone but A & B's developers responsibility to update to D' and
       | distribute patched versions? If the packager tries to do it, the
       | chances of diverging A and B due to some other incompatibility
       | are too high.
       | 
       | Either, you're running well maintained software (OSS or
       | commercial), and get the update in a timely manner, or you're
       | running unmaintained software and don't get the update.
       | 
       | Only in the second case does it make sense to patch in a
       | dynamically linked dependency. And I'm sure there's plenty of
       | examples of this, but the real issue is running un-maintained
       | software!
       | 
       | I am much happier keeping in close sync with timely releases from
       | Go/Rust projects than I am with Debian et. al's style of freezing
       | the world.
        
         | bombcar wrote:
         | The sad truth is without the package maintaining too many
         | people would be running horribly out of date and insecure
         | software - as there is zero incentive to upgrade a working
         | system.
        
           | jay_kyburz wrote:
           | I think if the software was labeled "insecure - requires
           | update" people would update. If you don't know that something
           | is insecure then there is zero incentive.
        
         | evmar wrote:
         | The OP is talking about technical details like static linking
         | and language ecosystems, but if you zoom one level out, this
         | comment correctly pins where the underlying problem actually
         | lies: that the distribution maintainers inject themselves into
         | the development process of all the software they ship.
         | 
         | When A/B are not well-maintained upstream, distributions can
         | assist by updating its dependency D, but even when A/B are
         | well-maintained upstream, distributions still might monkey with
         | them when D changes, even when that change to D actually breaks
         | the software. To me, the root cause is that the distributions
         | are effectively attempting to participate the software they
         | ship, but do so by getting in the middle, rather than
         | participating upstream.
         | 
         | As a former author of some well-maintained upstream software I
         | found their involvement made the software overall worse, but as
         | a user of a distribution I find their maintenance of otherwise
         | unmantained software sometimes helpful. In other words, I think
         | the goal is admirable but the mechanism is wrong, and doing
         | things like adding more dynamic linking only further enable the
         | bad behavior.
        
           | choeger wrote:
           | I think it would help if distributions could snatch a piece
           | of real estate in upstream software. Something like, "in
           | every project, the debian/ root folder belongs to the debian
           | project and follows their rules". The packaging could then
           | verify this folder and put their patches, build scripts, etc.
           | there. This would help upstream communication a lot, I guess.
        
             | CJefferson wrote:
             | The problem is there isn't just Debian, there is Debian and
             | arch and Gentoo and nix and guix and freebsd and cygwin (at
             | least).
             | 
             | Then, how do I check who should have merge access to all
             | these directories?
        
         | choeger wrote:
         | The distribution provides support for much longer than upstream
         | does. Also, packaging is actual work. Just because some
         | hipsters in some company consider it cool to release new
         | features every 4 weeks, it does not mean that the volunteers of
         | some linux distribution can keep up with thag. So if you want
         | your distribution to work well, you should focus on workable
         | and transparent interfaces.
         | 
         | If you are happy with the support cycles of your upstream and
         | can live with a black box, on the other hand you don't need a
         | distribution in the first place.
         | 
         | Besides, the whole "vendor everything, link statically" idea
         | works well only for the leafs of the tree. Guess what Rust
         | would do if llvm was _not_ a readily usable library but a bunch
         | of C++ files directly used inside clang? Thanks to the Rust
         | mode of operation, it is impossible to do with the rust
         | compiler what they did with llvm.
        
           | anotherhue wrote:
           | > on the other hand you don't need a distribution in the
           | first place.
           | 
           | I think you're right about that, I personally want as little
           | distribution as possible. FreeBSD ports or Arch AUR work well
           | for me.
           | 
           | The idea of solidifying other people's software into a bundle
           | and then maintaining it (with necessarily limited expertise)
           | seems like a loosing battle.
        
             | beojan wrote:
             | As someone who uses the AUR a lot, it's really the perfect
             | example of how you just won't update software if it isn't
             | done automatically by the distro package manager,
             | particularly when you use -git packages.
        
       | zxcvbn4038 wrote:
       | If your package management is worth it's salt then you don't care
       | about static linking, you just update every package that links to
       | that package. IBM's AIX had the best package management ever IMO,
       | you could roll forward or back and things would just work.
       | Completely deterministic. On the backend whenever a package was
       | updated they would regression test it and everything that
       | depended on it to ensure the update didn't break anything - when
       | a bug did get through they would fix it and add another
       | regression test so it didn't happen again. All of that kinda
       | broke when they adopted RPM as a secondary format because the
       | RPMs didn't make the same guarantees.
       | 
       | One of the best features of Golang is being able to cut a single
       | static linked Linux binary that works on Alpine, Red Hat, CentOS,
       | etc. With Go your also not linking to OpenSSL or Libxml, and
       | those two packages are responsible for every emergency roll out
       | I've ever had to do.
       | 
       | Time and again end result of version pinning is that when forced
       | to update developers are required to advance through multiple
       | major version updates and what should be a patch becomes a
       | rewrite. I have to constantly deal with a bunch of web sites on
       | jquery 2 and the developers boo-hooing every day because jquery
       | 3.5 is completely different and all I can tell them is that
       | jquery 2 is no longer maintained so they need to stop using it
       | for new projects and they need to update or retire their existing
       | projects using it.
       | 
       | One of the things I liked about Golang was that it didn't have
       | any versions to pin so it avoided all of that nonsense
       | altogether. However then they added "modules" and it's getting
       | harder to not use those, so that killed that magic. Though I did
       | make sure the CI/CD pipelines I use do a go get -u so hopefully
       | that will force the local Go devs to keep their code fresh.
        
       | dataflow wrote:
       | Why do only *nix folks pull their hair out about this? On Windows
       | programs bundle their dependencies all the time (sometimes even
       | as shared libraries! but without the independent update benefits)
       | and hardly anybody loses sleep over it. Heck, users actually like
       | the fact that it minimizes friction. Nobody claims it's rock-
       | solid security, but does it need to be?
       | 
       | Actually, now that I wrote it above, I think I might have found
       | an answer to my own question: while of course infosec experts
       | will freak out about any vulnerability existing anywhere for even
       | a shred of a nanosecond, in the real world this is really only a
       | big deal for servers, not clients. And I'm guessing this issue
       | affects Linux folks more because servers tend to run on Linux,
       | with security-sensitive software exposed to the open internet all
       | the time. Maybe we need to realize & embrace this trade-off
       | instead of fighting it till the end of time?
       | 
       | (I suppose one could even argue servers don't need this either if
       | all the packages are kept up-to-date anyway; I guess that might
       | also be debatable, but it's beside my point.)
        
         | staticassertion wrote:
         | > while of course infosec experts will freak out about any
         | vulnerability existing anywhere for even a shred of a
         | nanosecond, in the real world this is really only a big deal
         | for servers, not clients
         | 
         | I don't think that the linked article really reflects a
         | consensus among the security community at all. I don't really
         | find the "dynamic libraries are more secure" argument overly
         | compelling, certainly not stated so broadly.
        
         | GartzenDeHaes wrote:
         | The unix culture comes from a shared multi-user perspective,
         | whereas the windows point of view tends to be towards a single
         | user desktop. In a shared environment, it's not acceptable for
         | applications to change shared system components. Although this
         | is _maybe_ less important today, the culture still persists.
        
           | dataflow wrote:
           | > In a shared environment, it's not acceptable for
           | applications to change shared system components.
           | 
           | That era is _long_ gone on Windows (literally since XP I
           | think). In fact I 've had to do this far more on Linux than
           | on Windows in recent memory... the latest instance ironically
           | being the glibc CLOCK_REALTIME mess on WSL. Right now
           | programs bundle whatever they need, and these go into the app
           | directory (like \Program Files). e.g., I have maybe ~20
           | sqlite3 DLLs on my system besides the system ones.
           | 
           | Your larger point about the culture might still be correct
           | though, I don't know.
        
             | oblio wrote:
             | You're misreading his comment. He's saying Windows is share
             | nothing.
        
       | IshKebab wrote:
       | This article doesn't make a good case against static linking, and
       | the author doesn't seem to understand what vendoring is either:
       | 
       | > Bundling (often called vendoring in newspeak) means including
       | the dependencies of your program along with it.
       | 
       | No, vendoring means including a copy of the _source code_ of
       | dependencies in your repo. You can bundle dependencies without
       | vendoring them.
       | 
       | The only argument presented against static linking is that when a
       | library is updated rebuilding dependants takes longer (and people
       | will have to download bigger updates but I doubt many people care
       | about that).
       | 
       | That may be true, but is it really that big of an issue? I
       | somewhat doubt it (for sane distros that don't make users build
       | everything from source themselves anyway).
       | 
       | The author clearly doesn't like how people do development these
       | days but hasn't stopped to think _why_ they do it like that.
        
         | UncleEntity wrote:
         | > The only argument presented against static linking is that
         | when a library is updated rebuilding dependants takes longer
         | (and people will have to download bigger updates but I doubt
         | many people care about that).
         | 
         | Not really, the argument is that instead of rebuilding _one_
         | library you now have to rebuild _hundreds_ of applications to
         | see the benefits for whatever the library update was
         | for...assuming upstream even cares enough to bump the version
         | number for the _who knows how many libs they are vendoring_
         | because they have other stuff to do which is way more
         | important.
        
       | young_unixer wrote:
       | Centralized repositories are not a good, general solution to the
       | software distribution problem.
       | 
       | First: They don't have all the packages and versions the user
       | could ever need, which means that you'll always have a mixture of
       | software installed through the packagr manager and software
       | installed through tarballs or curl | bash
       | 
       | Second: They distort the relationship between application
       | developer and application user.
       | 
       | Third: they neglect offline use cases.
       | 
       | The only sane, generalizable solution is for the OS to be a
       | platform upon which bundles can be installed. The OS should just
       | define a sane format for said bundles and the user gets their
       | applications directly from the application developer.
        
       | posix_me_less wrote:
       | Here is the solution to this Debian/Gentoo/<other true free
       | software distribution> packager's dilemma. Packagers realize that
       | their numbers are small and they can't keep up fixing all the
       | modern big software (BS) projects that go against their
       | philosophy.
       | 
       | They define the core of the OS that they can keep managing in
       | this traditional way (kernel, basic userland, basic desktop,
       | basic server services, basic libraries, security support for all
       | that).
       | 
       | "But people want Firefox, Rust, Kubernetes, modern OS has to
       | provide them".
       | 
       | No it doesn't. Let the new complicated juggernaut software be
       | deployed and configured by the users, according to developers'
       | instructions, using whatever modern mechanism they prefer,
       | containers/chroots/light-weight VMs, whatever. Packagers can then
       | forget about the nasty big software and focus on quality of the
       | core OS. Users will have all the newest versions of BS they need
       | from the developers directly.
       | 
       | There is no reason to keep fast-pace, ever-changing and very
       | complicated programs/projects in the main OS distribution. It
       | only leads to lots of OS-specific packaging work with mediocre
       | and obsolete results, and that leads to discontent users,
       | developers and packagers.
        
         | sanxiyn wrote:
         | > basic desktop
         | 
         | I actually wish this is feasible, but it isn't. GNOME is basic
         | desktop, GNOME includes librsvg, librsvg build-depends on Rust.
         | So you need to package Rust (hence LLVM) to have GNOME. Your
         | proposal only works if "core OS" refuses to use Rust.
        
         | wheybags wrote:
         | This is why I think the snap store is going the right
         | direction. My web browser and my kernel don't need the same
         | level of attention.
        
         | tgbugs wrote:
         | I think that there is a good reason to keep those projects in
         | the main distribution: it makes them more robust and prevents
         | the build systems and bootstrapping process from becoming a
         | massive pile of technical debt.
         | 
         | For example, in order to get clojure 10 running on Gentoo I had
         | to go dig through the git history to figure out where spec-
         | alpha and core-specs-alpha and clojure itself did not form a
         | circular dependency in maven. Because clojure was not being
         | packaged in a variety of ways, they now are at risk of making
         | maven central a hard dependency, which makes the whole project
         | less robust.
        
           | posix_me_less wrote:
           | Do you mean "robust for people using Gentoo" or "robust for
           | everybody"? Isn't the latter responsibility of the Clojure
           | developers rather than packagers?
        
             | tgbugs wrote:
             | Robust for everyone. In theory it is the responsibility of
             | the Clojure devs, but if there isn't a packaging workflow
             | that they are aware of, how can they be expected to
             | integrate with it and make design and development decisions
             | that it would reveal to them? The point is more that, the
             | greater variety of different environments a piece of
             | software runs in the more likely it is that systematic
             | weaknesses will be revealed, sort of like chaos
             | engineering, except for whole environments instead of
             | sporadic failures.
        
         | pvorb wrote:
         | Exactly. When you want to install one of those fast-paced
         | pieces of software on your system, you're often dissatisfied
         | with the old version coming with your OS anyway.
        
         | delroth wrote:
         | You're making a distinction between "packagers" and "users"
         | which does not exist. Packagers are advanced users that take
         | the initiative to improve their distro when they find software
         | they want to use and that isn't integrated in their distro.
         | 
         | > Let the new complicated juggernaut software be deployed and
         | configured by the users, according to developers' instructions,
         | using whatever modern mechanism they prefer
         | 
         | I'm not sure what "they" refers to here (developers or users).
         | The existence of packagers is proof that some subset of users
         | "prefer" that software "be deployed and configured" via distro
         | package managers.
        
       | phkahler wrote:
       | >> Why do people pin dependencies? The primary reason is that
       | they don't want dependency updates to suddenly break their
       | packages for end users, or to have their CI results suddenly
       | broken by third-party changes.
       | 
       | Or because we dont want accidental or malicious security
       | vulnerabilities to get automatically incorporated into the
       | software.
       | 
       | This stuff works both ways. You dont automatically incorporate
       | fixes, nor new problems.
        
         | lucb1e wrote:
         | The vast, vast majority of updates fix security issues. It's
         | like not vaccinating in case you're one of the million people
         | that has an allergic reaction. Supply chain attacks are rare,
         | not the norm. We hear about such things (and only rarely at
         | that) because it's exceptional enough to make the news.
        
           | eecc wrote:
           | Nah, I don't buy it. If it's "just" bug fixes (for which I
           | might have implemented a hack that now depends on the bug) I
           | prefer Nightly builds with the latest (and re-pinned)
           | dependencies available. Releases are just a re-tag after
           | extra QA
        
           | 7sidedmarble wrote:
           | Yeah, but 'pinning' dependencies is useful so that you can
           | choose when you get those changes.
        
             | lucb1e wrote:
             | Which means extra maintenance work to check for every piece
             | of software that anyone uses whether it uses another
             | library that it needs to be recompiled against and, if it
             | fails, how to use the new version.
             | 
             | If there are automatic updates, at least it either works
             | and is more secure, or it breaks automatically and unsafe
             | software stops working.
             | 
             | Whether you prefer people to use MSIE6 because "it just
             | works" or whether you prefer old sites that only worked
             | with MSIE6 to break because it's no longer maintained,
             | that's the trade-off you have to choose between.
             | 
             | As a security person, I'm obviously biased, I can only
             | advise what I see from a professional perspective. All I
             | was saying above is that automatic updates being considered
             | a security risk is on the same scale of odds as considering
             | vaccines dangerous -- in regular cases, that is: of course
             | the advice is different if you're a special (sensitive)
             | organisation or a special (immunocompromised) person.
        
       | syllogism wrote:
       | This will be a somewhat intemperate response, because as a
       | developer of a significant library I found this quite irritating.
       | 
       | If you publish a Python library without pinned dependencies, your
       | code is broken. It happens to work today, but there will come a
       | day when the artifact you have published no longer works. It's
       | only a matter of time. The command the user had run before, like
       | "pip install spacy==2.3.5" will no longer work. The user will
       | have to then go to significant trouble to find the set of
       | versions that worked at the time.
       | 
       | In short unpinned dependencies mean hopeless bit-rot. It
       | guarantees that your system is a fleeting thing; that you will be
       | unable to today publish an end-to-end set of commands that will
       | work in 2025. This is completely intolerable for practical
       | engineering. In order to fix bugs you may need to go back to
       | prior states of a system and check behaviours. If you can't ever
       | go back and load up a previous version, you'll get into some
       | extremely difficult problems.
       | 
       | Of course the people who are doing the work to actually develop
       | these programs refuse to agree to this. No we will not fucking
       | unpin our dependencies. Yes we will tell you to get lost if you
       | ask us to. If you try to do it yourself, I guess we can't stop
       | you, but no we won't volunteer our help.
       | 
       | It's maddening to hear people say things like, "Oh if everyone
       | just used semantic versioning this wouldn't be a problem". Of
       | course this cannot work. _Think about it_. There are innumerable
       | ways two pieces of code can be incompatible. You might have a
       | change that alters the time-complexity for niche inputs, making
       | some call time-out that used to succeed. You might introduce a
       | new default keyword argument that throws off a *kwargs. If you
       | call these things "breaking" changes, you will constantly be
       | increasing the major version. But if you increase the major
       | version every release, what's the point of semver! You're not
       | actually conveying any information about whether the changes are
       | "breaking".
        
         | AshamedCaptain wrote:
         | > In short unpinned dependencies mean hopeless bit-rot.
         | 
         | No, this is not true, for the simple reason that there will
         | _always_ be unpinned dependencies (e.g. your compiler. your
         | hardware. your processor) and thus _those_ are the ones that
         | will guarantee bitrot.
         | 
         | Pinning a dependency only _guarantees you rot the same or even
         | faster_ because now it's less likely that you can use an
         | updated version of the dependency that supports more recent
         | hardware.
        
           | kuschku wrote:
           | > No, this is not true, for the simple reason that there will
           | _always_ be unpinned dependencies (e.g. your compiler. your
           | hardware. your processor) and thus _those_ are the ones that
           | will guarantee bitrot.
           | 
           | Docker with sha256 tags fixes that issue (and docker
           | container even specify a processor architecture).
        
           | gpm wrote:
           | > your compiler
           | 
           | Compilers of languages like C, C++, Rust, Go etc go above and
           | beyond to maintain backwards compatibility. It is extremely
           | likely that you will still be able to compile old code with a
           | modern compiler.
           | 
           | > your processor
           | 
           | Hardware is common enough that people go out of their way to
           | make backwards compatibility shims. Things like rosetta,
           | qemu, all the various emulators for various old gaming
           | systems, etc.
           | 
           | > your hardware
           | 
           | Apart from your CPU (see above) your hardware goes through
           | abstraction layers designed to maintain long term backwards
           | compatibility. Things like opengl, vulkan, metal, etc. The
           | abstraction layers are in widespread enough use that as older
           | ones stop being outdated people start implementing them on
           | top of the newer layers. E.g. here is OpenGL on top of
           | Vulkan: https://www.collabora.com/news-and-
           | blog/blog/2018/10/31/intr...
           | 
           | > [Your kernel]
           | 
           | Ok, you didn't say this part, but it's the other big unpinned
           | dependency. And it too goes above and beyond to maintain
           | backwards compatibility. In fact Linus has a good rant on
           | nearly this exact topic that I'd recommend watching:
           | https://www.youtube.com/watch?v=5PmHRSeA2c8&t=298s
           | 
           | > Pinning a dependency only _guarantees you rot the same or
           | even faster_ because now it's less likely that you can use an
           | updated version of the dependency that supports more recent
           | hardware.
           | 
           | Dependencies are _far_ more likely to rot because they change
           | in incompatible ways than the underlying hardware does, even
           | before considering emulators. It 's hard to take this
           | suggestion seriously at all.
        
             | AshamedCaptain wrote:
             | > Dependencies are far more likely to rot because they
             | change in incompatible ways than the underlying hardware
             | does
             | 
             | Yes, that is true. It is also very likely that you can more
             | easily go back to a previous version of a dependency than
             | you can go back to a previous hardware. The argument is
             | that, therefore, pinning can only speed up your rotting.
             | 
             | If you don't statically link your dependencies, and due to
             | an upgrade something breaks, you can always go back to the
             | previous version. If you statically link, and the hardware,
             | compiler, processor, operating system, whatever causes your
             | software to break, now you can't update the dependency that
             | is causing the breakage. And it is likely that your issue
             | is within that dependency.
             | 
             | Pinning can only make you rot faster.
        
         | foerbert wrote:
         | I don't have a particularly strong viewpoint on this, but I
         | find it noteworthy that in your example the user themselves is
         | asking for a specific version of the software. You don't seem
         | to be intending for users to ask for simply the latest version
         | and have that work, but a specific one, and you want that
         | specific version to work exactly as it did whenever it was
         | published.
         | 
         | I can see some instances in which this expectation is
         | important, and others where it is likely not or else certainly
         | less important than the security implications.
         | 
         | For the extremes, I see research using spaCy has a very strong
         | interest in reproducibility and the impact of any security
         | issues would likely be minimal on the whole simply due to the
         | relatively few people likely to run into them.
         | 
         | On the other extreme, say some low-level dependency is somehow
         | so compromised simply running the code will end up with the
         | user ransomware'd after just-long-enough that this whole
         | scenario is marginally plausible. Then say spaCy gets
         | incorporated into some other project that goes up the chain a
         | ways and ultimately ends up in LibreOffice. If all of these
         | projects have pinned dependencies, there is now no way to
         | quickly or reasonably create a safe LibreOffice update. It
         | would require a rather large number of people to sequentially
         | update their dependencies, and publish the new version, so that
         | the next project up the chain can do the same. LibreOffice
         | would remain compromised or at best unavailable until the whole
         | chain finished, or else somebody found a way to remove the
         | offending dependency without breaking LibreOffice.
         | 
         | I'm not sure how to best reconcile these two competing
         | interests. I think it seems clear that both are important. Even
         | more than that, a particular library might sit on both extremes
         | simultaneously depending on how it is used.
         | 
         | The only solution - though a totally unrealistic and terrible
         | one - that comes to mind is to write all code such that all
         | dependencies can be removed without additional work and all
         | dependent features would be automatically disabled. With a
         | standardized listing of these feature-dependency pairs you
         | could even develop more fine-grained workarounds for removal of
         | any feature from any dependency.
         | 
         | The sheer scale of possible configurations this would create is
         | utterly horrifying.
         | 
         | At any rate, your utter rejection of the article's point seems
         | excessively extreme and even ultimately user-hostile. I can
         | understand your point of view, particularly given the library
         | you develop, however I think you should probably give some more
         | thought to indirect users - ie users of programs that (perhaps
         | ultimately) use spaCy. I don't know that it makes sense to
         | practically change how you do anything, but I don't think the
         | other viewpoint is as utterly wrongheaded as you seem to think.
        
         | dbaupp wrote:
         | Libraries pinning dependencies only fixes a narrow portion of
         | the problem and introduces a bunch of others (particularly in
         | ecosystems where only a single version of a package can exist
         | in a dependency tree). In particular, it is great because it
         | makes life slightly easier for the library developers. However,
         | if every library pinned deps, it becomes much harder to use
         | multiple libraries together: suppose an app used libraries A
         | and B, and A depends on X==1.2.3, while B depends on X==1.2.4.
         | It's then pushed on to every downstream developer to work out
         | the right resolution of each conflict, rather than upstream
         | libraries having accurate constraints.
         | 
         | Pinning dependencies in applications/binaries/end-products is
         | clearly the right choice, but it's _much_ fuzzier for
         | libraries.
        
           | john-shaffer wrote:
           | Can you give an example of a real ecosystem that can't handle
           | such a conflict? In my actual experience, the package manager
           | will either automatically use the latest version, or in one
           | case has more complex rules but still picks a version on its
           | own (but I stay away from that one). Your argument has force
           | against either bad package managers or against using very
           | strict dependency requirements, but not against pinning
           | dependencies sensibly in a good ecosystem.
           | 
           | The only conflict I've seen that can't be automatically
           | resolved is when I had some internal dependencies with a
           | common dependency, and one depended on the git repo of the
           | common dep (via sha of a commit), and another depended on a
           | pinned version of the common dep. Obviously there's no good
           | way to auto-resolve that conflict, so you should probably
           | stick with versions for library deps and not git shas.
        
         | Qwertious wrote:
         | >Of course this cannot work. _Think about it_. There are
         | innumerable ways two pieces of code can be incompatible.
         | 
         | There's a very simple solution here: just don't write bugs.
        
         | anderskaseorg wrote:
         | If you publish a Python library with pinned dependencies, your
         | code is broken as soon as someone tries to use it with another
         | Python library with pinned dependencies, unless you happened to
         | pin exactly the same version of the dependencies you have in
         | common.
         | 
         | Python libraries should not pin dependencies. _Applications_
         | can pin dependencies, including all recursive dependencies of
         | their libraries. There are tools like Pipenv and Poetry to make
         | that easy.
         | 
         | This is less of an issue in (say) Node.js, where you can have
         | multiple different versions of a library installed in different
         | branches of the dependency tree. (Though Node.js also has a
         | strong semver culture that almost always works well enough that
         | pinning exact versions isn't necessary.)
        
           | acidbaseextract wrote:
           | The most frustrating thing is that pip doesn't make it easy
           | to use more loose declared dependencies while freezing to
           | actual concrete dependencies for deployment. Everybody rolls
           | their own.
           | 
           | > Python libraries should not pin dependencies.
           | _Applications_ can pin dependencies, including all recursive
           | dependencies of their libraries.
           | 
           | Is the pypi package awscli an application or a library?
           | 
           | poetry is frustrating in that it doesn't allow you to
           | override a library's declared requirements to break
           | conflicts. They refuse to add support [1][2] for the feature
           | too. awscli for example causes huge package conflict issues
           | that make poetry unusable. It's almost impossible not to run
           | into a requirement conflict with awscli if you're using a
           | broad set of packages, even though awscli will operate
           | happily with a more broad set of requirements than it
           | declares.
           | 
           | [1] https://github.com/python-poetry/poetry/issues/697 [2]
           | https://github.com/python-
           | poetry/poetry/issues/697#issuecomm...
        
             | anderskaseorg wrote:
             | For this purpose, I'm defining a "library" as any PyPI
             | package that you expect to be able to install alongside
             | other PyPI packages.
             | 
             | The awscli documentation recommends installing it into its
             | own virtualenv, in which case pinned dependencies may be
             | reasonable. There are tools like pipx to automate that.
             | 
             | Though in practice, there are reasons that installing
             | applications into their own virtualenv might be
             | inconvenient, inefficient, or impossible. And even when
             | it's possible, it still comes with the risk of missing
             | security updates unless upstream is doing a really good job
             | of staying on top of them.
        
         | acidbaseextract wrote:
         | > It's maddening to hear people say things like, "Oh if
         | everyone just used semantic versioning this wouldn't be a
         | problem". Of course this cannot work. _Think about it_. There
         | are innumerable ways two pieces of code can be incompatible.
         | ... If you call these things "breaking" changes, you will
         | constantly be increasing the major version.
         | 
         | One of the things that prompted the OP was this breakage in
         | Python's cryptography package [1] (OP actually opened this
         | issue) due to the introduction of a Rust dependency in a 0.0.x
         | release. The dependency change didn't change the public API at
         | all, but did still cause plenty of issues downstream. It's a
         | great question on the topic of semver to think about how to
         | handle major dependency changes that aren't API changes.
         | Personally, I would have preferred a new major release, but
         | that's exactly your point syllogism -- it's a matter of opnion.
         | 
         | As a sidenote, Alex Gaynor, one of the cryptography package
         | maintainers is on a memory-safe language crusade. Interesting
         | to see how that crusade runs into conflict with the anti-static
         | linking crusade that distro packagers are on. I find both goals
         | admirable from a security perspective. This stuff is hard.
         | 
         | [1] https://github.com/pyca/cryptography/issues/5771
        
       | rosshemsley wrote:
       | Whilst I don't totally disagree with many of the points here, I
       | think there's a wider picture to many of these issues.
       | 
       | The author is concerned with installing packages on user
       | machines: which are typically very long-lived installs - maybe a
       | user has the same machine with the same dependencies for years.
       | 
       | However, for many engineers, (such as myself), a binary may not
       | be used past even a few days from when it was first compiled -
       | e.g. as part of a service in a a quickly continuously integrated
       | system.
       | 
       | I might even argue that _most_ software is used in this way.
       | 
       | When software is built this way, many of the points in this
       | article are very helpful to keep builds stable and to make
       | deployment fast - and in fact for the case of security, we
       | usually _don't_ want dependencies to auto-update, as we do not
       | want to automatically deploy new code if it has not been audited.
       | 
       | Maybe there's a future were OSs become more like this, where
       | binaries are more short lived... maybe not. Although I don't
       | think it's strictly fair to label all of these as "Bad" with a
       | capital B :)
        
         | ampdepolymerase wrote:
         | The way iOS and Fuchsia are dealing with the problem is to
         | completely lockdown the operating system with a tight
         | permissions system. An app can be compromised but the damage is
         | limited. Perhaps it is time for servers to move to a similar
         | model.
        
           | AshamedCaptain wrote:
           | > An app can be compromised but the damage is limited
           | 
           | AKA the "we don't care" security model. What exact use is the
           | fact that the web browser is "contained" if it is
           | compromised? The mail client? Your PIM program? On a server,
           | what use is that the database engine is contained if it is
           | compromised?
           | 
           | I am the first to accept the security benefits of sandboxing,
           | but it is just _one_ thing. It doesn't even help against the
           | majority of issues. Not even on Android/iOS.
        
           | giantrobot wrote:
           | We should call this newly invented and wholly original
           | concept a "container". The software gets "contained". It just
           | might work. /s
        
             | eecc wrote:
             | You mean cgroups, or zones don't you? Docker (was, last
             | time I heard) a security disaster, not generating robust
             | layer hashes, lacking user isolation, and plenty just
             | running as root...
        
               | giantrobot wrote:
               | There's more to containers on Linux than just Docker.
        
           | loloquwowndueo wrote:
           | Just install all your software from snaps on an Ubuntu system
           | ;) (just kidding, snaps have a whole bunch of issues for
           | server software )
        
       | tgbugs wrote:
       | Fantastic article. I now have something to point people to when
       | they ask "what is wrong with pinning?" or "what is wrong with
       | static linking?" or "why can't you just use pip?" Michal has had
       | to deal with some pretty crazy stuff the last couple of months
       | ... scratch that, years.
       | 
       | The recent attempts to empower developers to distribute their own
       | software means that there is now the potential for there to be as
       | many bad security practices as there are pieces of software,
       | because systematic security design that used to be managed by
       | distributions has been pushed down to individual software
       | projects, which have only a tiny view of the issues, and thus
       | repeatedly make locally convenient decisions that are disastrous
       | for everyone else. What do you mean someone is using a project
       | other than ours that shares a dependency? Why would they do that?
       | 
       | One other thing to keep in mind is that from a security
       | standpoint the approach that Nix takes is not good. Nix can deal
       | with pinned dependencies on a per-package basis, but if those
       | pinned versions are insecure and the developers don't have a
       | processes for keeping dependencies up to date, then users are
       | sitting ducks.
       | 
       | Unfortunately pinning is a symptom of at least three underlying
       | issues. The first is that developers do not properly assess the
       | costs of adding a dependency to a project both in terms of
       | complexity, and in terms of maintenance burden. The second is
       | that many of the dependent libraries make breaking changes
       | without providing space for the old api and the new api to
       | coexist at the same time for a certain period so that developers
       | can transition over (I have been bitten by this with werkzeug and
       | pint). Mutual exclusion of versions due to runtime failure is a
       | nasty problem, and pinning hides that issue. Finally it seems
       | that at least some developers are engaged in the Frog and Toad
       | are Cofounders continuous integration story, but with a twist,
       | rather than deleting failing tests, they pin packages so that
       | they don't have to see the accumulating cost of pulling in
       | additional dependencies. Externalities, externalities everywhere.
       | 
       | edit: because it is just too on point https://medium.com/frog-
       | and-toad-are-cofounders/technical-de...
        
         | bscphil wrote:
         | > "why can't you just use pip?"
         | 
         | IMO, pip is _great_ , but it has exactly two use cases where
         | it's warranted. One is where you need to be your own
         | maintainer, e.g. you've developed a private web server to run
         | your website, and you need to manage its dependency versions
         | precisely. The other is for development purposes: it's really
         | great to be able to use different versions of Python or test
         | against different libraries than your distribution ships.
         | 
         | It's _not_ (or shouldn 't be) for shipping software to end
         | users. That's the problem being complained about in the essay.
         | This particular way of handling dependencies (use system
         | libraries by default, but allow the user / developer to create
         | an entire virtualized Python installation if they want one) is
         | actually why I think Python handles this _better_ than most
         | other languages, including older ones.
        
           | tgbugs wrote:
           | I agree. The context was missing from my original, which is,
           | "why can't you just use pip to install dependencies in
           | production," which is effectively the answer that Michal once
           | got from a PyPA maintainer when asking about setup.py
           | install.
        
         | saurik wrote:
         | > I now have something to point people to when they ask "what
         | is wrong with pinning?" or "what is wrong with static linking?"
         | or "why can't you just use pip?" Michal has had to deal with
         | some pretty crazy stuff the last couple of months ... scratch
         | that, years.
         | 
         | Literally the only argument in this article against static
         | linking is "it will take an extra couple hours for the
         | distribution to recompile all affected packages and then
         | require the user to download a larger update, meaning time to
         | fix for a security issue will be _negligibly_ longer "... since
         | you seem to believe this article represents the strongest
         | statement of your argument, this has actually moved me further
         | away from where you want me to be ;P. (FWIW, the best argument
         | I could make for dynamic linking involves memory efficiency for
         | shared pages, the disk cache, and _maybe_ the i-cache, not
         | security.)
        
           | jcranmer wrote:
           | > maybe the i-cache
           | 
           | This I doubt. On a statically-linked application, all calls
           | to functions are going through a regular function call
           | instruction. When you dynamically link, every call to a
           | function that might cross library boundaries (which on ELF
           | systems defaults to every call that's not to a static
           | function) instead calls into the PLT, which will itself
           | dispatch through to the underlying function call.
        
       | 7800k wrote:
       | Gentoo is secure, as far as you can be secure by building a bunch
       | of code few have time to review on hardware few have time to
       | fully understand running in an insecure world.
       | 
       | So, I don't fault it.
       | 
       | However, if you don't include dependencies and you don't manage
       | them, which would be the case in modern environments for the
       | majority of users in the world, how is that safe?
       | 
       | Answer: it's not.
        
       | setheron wrote:
       | Once Nix moves more to fixed output derivations this will be
       | killer!
       | 
       | You can easily upgrade dependencies by having the linker resolve
       | to different libraries without needing to rebuild the world while
       | still retaining a sensible memory model.
        
       | wheybags wrote:
       | Nobody cares how secure my code is if it doesn't work.
        
         | [deleted]
        
       | steveklabnik wrote:
       | There is a lot I could say about this article, but I've kinda
       | been in this whole Texas natural disaster situation. But I do
       | think it's worth pointing out one thing:
       | 
       | > Rust bundles a huge fork of LLVM
       | 
       | It is not super huge, and we try to upstream patches regularly to
       | minimize the fork. We also regularly re-base all current patches
       | on each release of LLVM, so it's less 'fork' and more 'maintain
       | some extra commits to fix bugs.'
       | 
       | > and explicitly refuses to support to distributions using the
       | genuine LLVM libraries.
       | 
       | We always support the latest LLVM release, and try to maintain
       | compatibilities with older ones as long as is reasonable and
       | possible.
       | 
       | IIRC, the last time we raised the base LLVM requirement was in
       | Rust 1.49, at the end of last year. The minimum version it was
       | raised to was LLVM 9, which was released in September of 2019.
       | The current release is 11.
       | 
       | Again, because of the lack of the extra patches, you may see
       | miscompilation bugs if you use the stock LLVM. There's pros and
       | cons to every choice here, of course.
       | 
       | For more on this, please see https://rustc-dev-guide.rust-
       | lang.org/backend/updating-llvm....
       | 
       | The Rust project is and has been interested in collaborating on
       | issues to make things easier for folks when there's demand. We've
       | sought feedback from distros in the past and made changes to make
       | things easier!
       | 
       | (I also posted a slightly modified version of this to their blog
       | comments as well. EDIT: the author has commented that suggesting
       | Rust's fork is large was in error)
        
         | Daho0n wrote:
         | At a minimum a reasonable time for a package for it to be a
         | good citizen in the system is what is in Debian LTS.
        
           | steveklabnik wrote:
           | EDIT: you completely changed your comment. My original
           | response is below the line, but I'll also respond to your new
           | one above it.
           | 
           | I don't actually understand grammatically what you're asking
           | for or saying in this comment. Rust, Cargo, and a bunch of
           | Rust applications are packaged in Debian today, from Buster
           | onwards.
           | 
           | I _think_ what you 're saying is that rustc head should be
           | able to be built with the llvm that lives in Debian stable?
           | That is a choice that could be made, I'm sure, but then
           | you're talking about supporting five API-incompatible
           | releases of LLVM. The folks from Debian aren't even asking
           | for that level of compatibility, and it would be quite a bit
           | of work, for unclear benefit.
           | 
           | --------------------------------------------
           | 
           | There are lots of demands, from lots of people. Managing a
           | project requires prioritizing different demands of all of
           | your different users.
           | 
           | > When will the problem mentioned in the article
           | 
           | The article talks about many more things than one single
           | problem. Some range from easy, like the "we already do let
           | you use stock llvm" I mentioned above, and some are difficult
           | and take a long time, like "define a Rust ABI." The needs of
           | packagers need to be balanced with the needs of other folks
           | as well. "define a Rust ABI," specifically, for example,
           | would help some packagers, but it may harm other users. It
           | also may be something that's good to have, but we aren't
           | quite ready for it yet.
           | 
           | A willingness to _work together_ to solve issues, and to find
           | solutions that help as many different kinds of folks as
           | possible, is what matters here. We have had folks from other
           | distros pitching in to help make things better for their
           | distros, and I can 't imagine we wouldn't accept the help of
           | the OP either.
        
         | archi42 wrote:
         | What's the problem with focusing on upstreaming the patches and
         | getting rid of rust's "staging fork"? Especially if they fix
         | something as serious as miscompilations, wouldn't those patches
         | be VERY much needed upstream? I'm asking out of genuine
         | interest.
        
           | steveklabnik wrote:
           | Quoting from the documentation I linked:
           | 
           | > Strongly prefer to upstream all patches to LLVM before
           | including them in rustc.
           | 
           | That is, this is already the case. We don't _like_
           | maintaining a fork. We try to upstream as much as we can.
           | 
           | But, at the same time, even when you do this, it takes tons
           | of time. A contributor was talking about exactly this on
           | Twitter earlier today, and estimated that, even if the patch
           | was written and accepted as fast as possible, it would still
           | take roughly a year for that patch to make it into the
           | release used by Rust. This is the opposite side of the whole
           | "move slow and only use old versions of things" tradeoff:
           | that would take _even longer_ to get into, say, the LLVM in
           | Debian stable, as suggested in another comment chain.
           | 
           | So our approach is, upstream the patches, keep them in our
           | fork, and then remove them when they inevitably make it back
           | downstream.
           | 
           | (EDIT: Also, what Rusky said: there's basically always going
           | to be more than one patch, in various stages of this
           | process...)
        
             | archi42 wrote:
             | Ok, maybe I'm just to spoiled by my previous experience
             | then. I never had any problems getting patches upstreamed
             | relatively quickly, but then e.g. meson or picolibc are
             | much smaller than LLVM and patches are simpler to reason
             | about (I know some compiler construction).
             | 
             | Thanks for the answer.
        
           | Rusky wrote:
           | That's already done. The point is that it takes time, and
           | there are always more of them, so at any point in time there
           | are likely to be a handful of them still in-process of moving
           | upstream.
        
       | [deleted]
        
       | larusso wrote:
       | It's sounds like pinning dependencies is just done because we
       | developers are a lazy bunch that just want to guard against the
       | rare case of a breaking upstream change. I was burned too often
       | in the past by breaking changes or more often behavior changes in
       | transient dependencies to not put up some defense. But I still
       | don't pin versions in the main dependency files (cargo.toml,
       | gemset or similar). I just have the generated lockfile and put
       | that under version control. so all developers in the team get the
       | same versions of direct and transient dependencies. I define the
       | dependencies with version ranges. Every package manager works
       | differently here. I like cargo the best since a statement like
       | 'version = 1.2.1' means give me a version bigger than '1.2.1' but
       | lower than '2.0.0'. One should never add lockfiles for library
       | projects as they need to be open for future updates. I also add
       | dependabot to all repos and let the bot inform me about updates.
        
       ___________________________________________________________________
       (page generated 2021-02-21 23:00 UTC)