[HN Gopher] The modern packager's security nightmare
___________________________________________________________________
The modern packager's security nightmare
Author : summm
Score : 155 points
Date : 2021-02-20 13:12 UTC (1 days ago)
(HTM) web link (blogs.gentoo.org)
(TXT) w3m dump (blogs.gentoo.org)
| paultopia wrote:
| I feel like this confuses a lot of things by assuming an
| extremely sophisticated end user. Sure, if you're a double-threat
| dev/sysadmin using linux, then when some vulnerability gets
| discovered in some dynamically linked library on your system, you
| have the capacity to (a) receive information about that fact, and
| (b) update it.
|
| But now suppose you're an ordinary person. You use software.
| Maybe you even have a windows machine.[1] Which is more likely to
| actually get a security update to you?
|
| (a) You have to update a single piece of software, which you know
| you've installed, though a recognized distributional channel like
| an app store or something, and all its dependencies come with it.
|
| (b) You have to either learn what a DLL is and learn how to
| update it and then hope that nothing you rely on breaks in some
| mysterious way because of some dependency on a dependency on a
| dependency on a dependency. Or you have to accept a whole
| operating system update, assuming that the operating system comes
| with a DLL update---and hence accepting all of the other crap
| that comes with operating system updates from Microsoft (and
| Apple), such as bugginess from complex updates, incompatibilities
| between new versions of operating systems and software (or
| hardware) that you rely on, new obnoxious security rules that you
| might not agree to (looking at you, Cupertino), and massive
| disruptions to your ability to actually use your computer to do
| your work.
|
| No thanks.
|
| [1] Maybe this article is specifically targeted against the linux
| ecosystem? If so, perhaps this issue is ameliorated somewhat, but
| it still seems to put a fairly substantial burden on end users,
| that seems to be inconsistent with actually letting non-experts
| use linux OSes.
| regularfry wrote:
| Alternatively, realise that the problems in (b) are largely
| solved for reasonable OSes where the vendor takes
| responsibility both for automatically getting you security
| patches and for keeping you working without major disruptions.
|
| I'm not sure where you've got the idea that updates are all-or-
| nothing, but some of us have been living a life you seem to
| think can't exist for _decades_ at this point.
| deadbunny wrote:
| That sounds like a good argument for using package managers
| with automatic security updates. The user doesn't need to be an
| expert, just reboot when the system tells them to.
| mixedCase wrote:
| Yeah well, who's hungry?
|
| No really, neither users nor developers care, nor should they.
| I've been using Linux on the desktop for well over a decade and
| I'm tired of seeing this plea for everything to behave exactly
| like C or scripting languages because every distro wants to be
| its own special snowflake and it would be too hard to adapt.
|
| The world has changed and distros have to stop pretending it
| hasn't. Flatpak, Nix, those are better models that reflect our
| reality where developers don't want to worry about ten thousand
| different distros with wildly different downstream configurations
| producing different kinds of bugs, as well as the desire of users
| of just getting the darned app.
|
| If you're worried about security you must always, first, work
| with upstream. Upstream should be the party responsible for its
| users, not you. Upstream not fast enough and you want to go the
| extra mile? Well then your packaging tool-belt should have
| support for patching a series of libaries in an automated fashion
| and rebuilding the dependency tree; and make sure that you return
| to upstream as soon as possible.
|
| If you want your downstream to diverge from upstream because you
| want to act as a barrier as the old distros do, then you'll have
| to accept the fact that you're _maintaining a fork_ and not
| pretend to upstream that you 're distributing their work or that
| your bug reports are compatible. Otherwise, again, just limit
| yourself on distributing the latest upstream's stable release
| with the bare minimum patching necessary for getting things to
| work with the distro's chosen config.
|
| With some luck, after the Linux community comes to terms with the
| fact that the distro model must shift, we can begin to finally
| share some packaging efforts between all distros and we can leave
| much of the incompatibility bullshit in the past.
|
| Imagine one day everyone just building from shared Nix
| derivations? Very ironically, it would look a lot like Portage
| and Gentoo's USE flags but with everyone building off those
| generic derivations and offering binary caches for them.
| kenniskrag wrote:
| Do I need to recompile the shared library and the clients if one
| of them needs a newer compiler or if the client is provided as
| binary by the vendor?
| ChrisMarshallNY wrote:
| For an OS like Linux, the thing about static linking is 100%
| spot-on.
|
| However, for vetted systems, like app stores, this makes no
| difference, as the only dylibs allowed to be patched at runtime,
| are the OS-supplied (ABI) ones. The whole app needs to be
| rebuilt, re-vetted, and rereleased.
|
| Frankly, I would be worried about dylibs, but for a different
| reason, as more than one hack has been done by hijacking dynamic
| links. I'm trying to remember how it went, but there was once a
| common practice for Mac programs to deliberately take advantage
| of this, in order to add functionality.
| bscphil wrote:
| Over the last several weeks I was working on an essay about this
| exact problem, including the connection between static linking
| and bundling. This one is so well done that I probably won't even
| publish it.
|
| But I'll add this, for people who may not immediately see why
| this is important. I think that the real danger these new
| technologies represent is not inherently bad technology, but the
| possibility of ecosystem damage. The distribution / maintainer /
| package manager approach has proven to be an extremely reliable
| way to get trustworthy software. Many of us love it and want to
| see it stick around. And it's been possible because "upstream"
| developers in the open source ecosystem have been willing (or
| forced) to work with distributions to include their software. But
| this seems to be changing. A highlight from my essay:
|
| 'Many software projects are not good citizens in the open source
| ecosystem. They are working with a model of development and
| distribution that does not mesh well with how open source
| software is predominantly produced. ... These [new] languages are
| now developing _their own_ ecosystems, with their own
| expectations for how software should be created and distributed,
| how dependencies should be handled, and what a "good piece of
| software" looks like. Regardless, an increasing amount of
| software is being built in these ecosystems, _including_ open
| source software.
|
| There might come a day in which open source is fractured. Two
| different communities create two very different kinds of
| software, which run on the same systems, but are created and
| distributed in very different ways. I begin to worry that the
| community I care about might not survive. Even if my ecosystem
| continues on its way, I don't want this split to take place. I
| think being part of the open source ecosystem is good for
| software, and I think having all the software you could want
| available within that ecosystem is good for users. If anything,
| this essay is a call to those who agree to be more careful about
| the software they create. Make sure it's something that Debian
| maintainers could be proud to ship. If you're working on a Rust
| program, for example, ask questions like "how can I make this
| program as easy for maintainers to distribute as possible?" If
| you have the ability to work on projects like Rust or Go, do what
| you can to give their applications the ability to easily split
| dependencies and support system provided libraries. Let's try to
| make sure the software ecosystem we love is around for the next
| generation.'
|
| I'd also strongly recommend the essay "Maintainers Matter" for
| another take on why this is so important.
| http://kmkeen.com/maintainers-matter/ Or read about Debian's
| recent problems with vendoring: https://lwn.net/Articles/842319/
| hctaw wrote:
| > The distribution / maintainer / package manager approach has
| proven to be an extremely reliable way to get trustworthy
| software. Many of us love it and want to see it stick around.
|
| I disagree, it's proven to be inadequate for modern software
| development and that's why these new languages/ecosystems are
| springing up. The least reliable way to package and distribute
| software is by relying on traditional package managers.
|
| > Do what you can to give their applications the ability to
| easily split dependencies and support system provided libraries
|
| This is unrealistic. I do not trust system provided libraries
| to function with my applications because I've been burned so
| many times in the past.
|
| > how can I make this program as easy for maintainers to
| distribute as possible
|
| By statically linking everything as much as possible and
| shipping everything else in a self contained bundle with a
| launcher that overrides any symbols that might inadvertently be
| pulled in from the system.
|
| The universe I'd like to live in is where the only use case for
| dynamic linking are OS vendor APIs and cryptographically secure
| functions like TLS. My dream package manager would whitelist
| those system librarie and forbid distribution for any bundle
| that does contain the shared objects with the symbols it needs.
| bscphil wrote:
| > I disagree, it's proven to be inadequate for modern
| software development
|
| Well, that's exactly why the OP (and my essay) are "anti"
| modern software development in many ways. The view is that
| we're moving away from the traditional open source ecosystem
| and methods of software development with these new
| technologies, which (to be clear) are _good_ technologies,
| but were created mostly to solve problems that some large
| corporations have, not to solve the problems that the open
| source ecosystem has.
|
| > The least reliable way to package and distribute software
| is by relying on traditional package managers.
|
| Not sure what you mean by this, but it's entirely untrue in
| my experience. Anything I install with a package manager just
| works, 100% of the time. Stuff I try to get any other way is
| a shitshow, and the lack of "quality control" provided by
| maintainers speaks for itself. I mean, just look at Android
| apps or the Chrome extension store. Heaven forbid we go back
| to the days of curl | bash off someone's website.
|
| > By statically linking everything as much as possible and
| shipping everything else in a self contained bundle with a
| launcher that overrides any symbols that might inadvertently
| be pulled in from the system.
|
| I know you know this, but just to be clear, that's not a
| solution to the problem of "making things easier for
| maintainers to distribute", that's cutting maintainers out of
| the loop. The whole point of my focus on ecosystems is that
| this is something that I, as a user, don't want to happen.
| hctaw wrote:
| I'd disagree that the problem of large organizations are
| different from the problems of the FOSS ecosystem.
| Organizations just have a financial incentive to fix them,
| the FOSS ecosystem does not. If mutually incompatible
| dependencies and security updates breaking software weren't
| problems for both corporate and FOSS ecoystems, these new
| technologies wouldn't have needed to exist. They'd just use
| the existing platforms.
|
| And mind you, this is not a corporate/open source split.
| The burgeoning ecosystems are also full of FOSS
| technologies doing new and exciting things, they just don't
| break when a dependency updates!
|
| >Anything I install with a package manager just works, 100%
| of the time
|
| I run into issues with packages weekly. So much so I've
| spent engineer days purging references to system packages.
| It's universal too - yum, apt, pacman, brew, macports, I
| have to make sure nothing tries to reference packages
| installed outside a local working directory for an
| application because of mutual incompatibilities. Maybe it's
| because I'm trying to write software that runs on multiple
| targets and not use software where someone else has already
| spent the time and money to resolve these issues.
|
| > I know you know this, but just to be clear, that's not a
| solution to the problem of "making things easier for
| maintainers to distribute", that's cutting maintainers out
| of the loop. The whole point of my focus on ecosystems is
| that this is something that I, as a user, don't want to
| happen.
|
| They should be cut out of the loop. Maintainers don't have
| a right to dictate what design decisions I put into my
| applications because they don't think it adds value (the
| value is, it doesn't just run on their distro!). Another
| comment in this thread put it better, maintainers shouldn't
| place themselves in the development process.
| ticviking wrote:
| > maintainers shouldn't place themselves in the
| development process.
|
| Then perhaps we as developers should spend more time
| thinking about and participating in the maintenance
| process?
| bscphil wrote:
| > They should be cut out of the loop.
|
| I disagree. This is how you get the Google Play store or
| the "freeware" app marketplace. It sucks. As a user, I'm
| quite happy to continue using a traditional distribution
| even if it means I don't get to use a handful of flashy
| programs by developers that disagree with the concept of
| maintainers. So far that choice has been much more
| positive than negative for me, and I'm doing my best (by
| promoting the open source ecosystem) to keep it that way.
|
| > Maintainers don't have a right
|
| If you're creating open source software, they quite
| literally do!
| https://www.gnu.org/licenses/gpl-3.0.en.html
|
| If you put in a bunch of crap that doesn't belong in an
| application (ads, for example), I'm glad I have a
| maintainer that can either strip this out thanks to the
| GPL (or BSD / MIT etc), or else choose not to include
| your app in the distribution at all.
| sanxiyn wrote:
| > a handful of flashy programs
|
| This only works if "core" refuses to use Rust, so it's
| not sustainable. We already can't build Firefox and GNOME
| without Rust. Maybe Apache and curl next. And then?
| alfiedotwtf wrote:
| I've been on Debian since Potato, so I totally see what
| you're saying. But...
|
| > that's cutting maintainers out of the loop
|
| Is this necessarily a bad thing? The market has seen the
| need to fill a hole, and it seems to be working.
|
| I first started with Slackware, and dependency nightmares
| is what got me into Debian in the first place. Although
| Debian is nice _because_ of its slow and stable base (which
| makes me happy for production), I've recently moved to Arch
| and have been so happy as it's brought back Slackware's
| idea of getting as close to upstream as possible _and_ it
| handles dependencies! And to be honest, I'm loving it. And
| as an even added bonus, I'm getting more and more surprised
| how a tonne of packages that I've installed _are_ Rust
| apps.
|
| So, coming back to your comment:
|
| > that's cutting maintainers out of the loop
|
| With systems like Arch that get us closer and closer to
| upstream, are maintainers the unnecessary middlemen? Of
| course they're not entirely redundant, but maybe a new
| model of distros like Arch will be more commonplace in the
| future
| bscphil wrote:
| Thoughtful comment, thanks. I'm an Arch user as well and
| agree broadly with its approach to a desktop operating
| system. (I.e. stick as closely to upstream as possible.)
|
| That said, I disagree that this means maintainers are
| unnecessary middlemen, even though their role on a
| distribution like Debian is obviously more prominent. The
| essay I linked to in my top level comment is actually by
| an Arch maintainer, explaining why they still see
| maintainers as playing an important role.
| http://kmkeen.com/maintainers-matter/
| rsj_hn wrote:
| The problem is that the modern practices are being adopted
| to meet the changing needs of this code, which weren't an
| issue in the enthusiast linux and BSD communities, but are
| an issue when open source code is being put in mission
| critical professionally managed production environments on
| AWS, or into cars and appliances.
|
| The userland is much more complex now, so freezing
| dependencies and bundling them in order to have a smaller
| number of test cases may not have been necessary when the
| environment was smaller and simpler. The reliability
| expectations were lower, and there weren't big money
| professional support contracts which required you to
| validate your application against a set of well defined
| environments. All that has changed.
| oblio wrote:
| The open source solutions are primarily for C/C++. A bit
| for Perl, a bit for Python. But they haven't really moved
| on. Java has been tacked on since forever. Same for .NET,
| JavaScript, whatever.
|
| And if you wanted your program propagated to all major
| distros you'd have to wait a decade.
|
| Nobody has time for that. Not corporations, not mom and pop
| stores, and I doubt many hobbyists.
| ticviking wrote:
| Then you always have the option to package your software
| for the ones you use and not worry about the rest.
|
| If you want to make your program popular then packaging
| is part of the process needed.
| lucideer wrote:
| > _The universe I 'd like to live in is where the only use
| case for dynamic linking are OS vendor APIs and
| cryptographically secure functions like TLS. My dream package
| manager would whitelist those system librarie and forbid
| distribution for any bundle that does contain the shared
| objects with the symbols it needs._
|
| This idea seems to be predicated on a belief that OS vendor
| APIs and cryptographic libraries are the only attack surface
| for serious user-affecting software exploits.
| hctaw wrote:
| They're obviously not, but they're the ones that package
| managers can help mitigate automatically. The rest is going
| to be up to the developers to patch.
| nextos wrote:
| Some of the things you mention are not incompatible with
| package managers. Have you considered Nix?
|
| I actually agree with the parent post in terms of strongly
| preferring package managers to other means of distributing
| software. I've always found Linux much easier to admin than
| other OSes simply because of package managers.
| kuratkull wrote:
| Was really pumped after playing with Nix for a bit. But got
| bitten twice in the first day. I tried to run my work
| project using Nix. It needs Pythons python-prctl library -
| that doesn't work in Nix. So that's a dud. Next I try to
| use it as a Nim environment - Nim is unable to compile
| anything in Nix due to some missing (bundled) Glibc symbol.
| (Nim works nicely in my standard installation). The cool
| "export full container" thing mentioned in the tutorial
| failed for all, even the simplest cases. So I am kinda
| disillusioned by Nix.
| nextos wrote:
| You can always emulate FHS environments for things that
| are hard to turn into a Nix package without significant
| effort.
|
| Aside from that, Guix is an alternative implementation
| that might have what you need.
| [deleted]
| edoceo wrote:
| Based on this writing, I'm keen to read your full essay
| jauer wrote:
| > Two different communities create two very different kinds of
| software, which run on the same systems, but are created and
| distributed in very different ways.
|
| This is where the disconnect is coming from. The distro
| maintainers are coming from a world of multi-user systems where
| backwards compatibility and updating deps without disturbing a
| user's workload / forcing them to recompile is paramount.
|
| Go (and a fair amount of rust/python work) come from the land
| of CI/CD and, to a lesser extent, monorepos. When you are
| rebuilding the world above a bare minimum of the OS literally
| on every commit (or at least several times per day), it's
| easier to reason about code that is running if you can look at
| the commit that a bin was built from and know exactly what's
| inside (including all deps).
| bscphil wrote:
| I agree. I think the difference has been that until recently
| "the land of CI/CD" and so on has been certain segments of
| the corporate world, and not how typical open source
| developers did things*. So when the former developed new
| technologies and new languages, they created build tools for
| them that anticipated being used in the ways that _they_
| usually produce software.
|
| The "problem", in the sense that it's a problem, is that
| these languages and related technologies are all pretty good!
| And so it's understandable that many developers who would
| traditionally be in the open source ecosystem want to use
| them. As a result they end up creating software that can't
| easily be shipped in traditional distributions. Ecosystem
| fragmentation is the unavoidable result.
|
| * By typical open source developers, I mean the sort of
| developers (and their development practices) that produced
| most of the software on my computer. I don't mean Firefox:
| Mozilla and Google have much more standard corporate
| development practices despite both producing quite a bit of
| open source software.
| choeger wrote:
| Would it be feasible for rust to ship "re-link" scripts? That is,
| when a cargo dependency gets updated, the distribution can check
| whether packages code needs to be re-linked? Similarly, when
| distributed code (say libssl) changes, the rust packages can be
| re-linked by the distribution on-site?
| charliesome wrote:
| You can achieve this by recompiling the software. Cargo will
| not recompile dependencies that have not changed.
| Ericson2314 wrote:
| This is a moronic by saying everything is equally bad.
|
| 1. The linking strategy doesn't matter. Just rebuild everything.
| The reason Rust and Haskell don't do dynamic linking is pervasive
| inlining. This is required for many abstractions to work well,
| and asking optimization boundaries to coincide with software
| origin boundaries is stupid anyways.
|
| The ABI arguement is stupid because replacing all the dynamic
| libraries with the ABI changes is no worse than rebuilding
| everything cause static linking. Therefor ignore that part and
| just think about inlining.
|
| 2. Lock files are fine as long as one can in fact bump the minor
| versions. Yes, we should make tools to enforce minor versions are
| non-breaking. And it's crazy we didn't have those from day one.
| But until we do lock files are fine.
|
| 3. Vendoring is pure evil.
| Ericson2314 wrote:
| The tl;dr is we always need reproducible and composible builds.
| You can derive basically everything else from that.
| nomercy400 wrote:
| It's a risk calculation: do I want to risk being vulnerable to
| 0days, or do I want to risk my application not running for any
| user because a dependency changed its headers/api?
|
| As software engineers we want to be in control of as much as
| possible when running our application, to make it as
| deterministic as possible. For that we select versions of our
| dependencies, build it, and then test it, and if our tests pass,
| we release it.
|
| If we would let the OS determine the dependency versions, without
| testing, what guarantees can we give that our code will work? Do
| you write tests for future, unknown changes in libraries? Do you
| write tests with Random, because that's the kind of
| unpredictability you're introducing?
| regularfry wrote:
| Keeping the application working in the face of dependency
| updates is the distro's job, if they're supported in doing it.
| They won't be pushing dependency updates at random except to
| channels explicitly marked as unstable (at least, assuming a
| minimally sane distro).
|
| I'm less familiar with Gentoo, but Debian-based distros ought
| to be safe from that threat (and if anything, you ought to be
| worries about the reverse problem). So your question becomes:
| do I want to risk being vulnerable to zero-days?
| hctaw wrote:
| The day I stop static linking is the day I can compile on one
| distro and ship to many without worrying about users reporting
| loader errors. That day is not today.
|
| Until then, I'll keep doing it, because it saves me time and
| money.
|
| I don't really buy that dynamic linking all the things is such a
| boon to security. But I'll link to this:
| https://drewdevault.com/dynlib
|
| > Not including libc, the only libraries which had "critical" or
| "high" severity vulnerabilities in 2019 which affected over 100
| binaries on my system were dbus, gnutls, cairo, libssh2, and
| curl. 265 binaries were affected by the rest.
|
| > The total download cost to upgrade all binaries on my system
| which were affected by CVEs in 2019 is 3.8 GiB. This is reduced
| to 1.0 GiB if you eliminate glibc.
|
| > It is also unknown if any of these vulnerabilities would have
| been introduced after the last build date for a given statically
| linked binary; if so that binary would not need to be updated.
| Many vulnerabilities are also limited to a specific code path or
| use-case, and binaries which do not invoke that code path in
| their dependencies will not be affected. A process to ascertain
| this information in the wake of a vulnerability could be
| automated.
| sigjuice wrote:
| _The day I stop static linking is the day I can compile on one
| distro and ship to many without worrying about users reporting
| loader errors. That day is not today._
|
| Maybe do the trendy thing and link your whole distro together
| with your app (a.k.a containers)?
| forrestthewoods wrote:
| The fact that containers exist and are prevalent is a damning
| indictment on the Linux model imho.
|
| Producing software that can simply launch is so wildly
| complex that the solution is... to snapshot an entire
| operating system install to run the application.
|
| That's INSANE. But it's unfortunately required.
| https://xkcd.com/1987/
|
| > Now, for the worst of all -- one that combines all the
| aforementioned issues, and adds even more. Bundling (often
| called vendoring in newspeak) means including the
| dependencies of your program along with it. The exact
| consequences of bundling vary depending on the method used.
|
| Let's consider this statement at face value. The WORST thing
| a program can do is... include the things the program needs
| to run? So programs should... NOT include the things they
| REQUIRE to run?
|
| I get it. I understand the argument. But if you take just a
| half step back it should be clear how utterly broken the
| whole situation is.
| orra wrote:
| > I don't really buy that dynamic linking all the things is
| such a boon to security.
|
| Agreed. It's really not a panacea. When you upgrade a library,
| you probably want to restart the running applications that
| depend upon this. Dynamic linking won't save you.
|
| The solution is having a graph of your dependencies! IIRC,
| NixOS gets this right. I don't think Debian's apt did?
| AshamedCaptain wrote:
| > But I'll link to this: https://drewdevault.com/dynlib
|
| We already discussed this in HN
| https://news.ycombinator.com/item?id=23654353 . Both data and
| conclusions are utterly suspect.
| [deleted]
| anotherhue wrote:
| Correct me if I'm wrong (really) but isn't static linking mostly
| a problem for packagers? The less a package maintainer modifies
| the application the better IMO.
|
| If application A and B rely on dependency D, which turns out to
| have a vulnerability, fixed in D', Then _why_ do we think it is
| anyone but A & B's developers responsibility to update to D' and
| distribute patched versions? If the packager tries to do it, the
| chances of diverging A and B due to some other incompatibility
| are too high.
|
| Either, you're running well maintained software (OSS or
| commercial), and get the update in a timely manner, or you're
| running unmaintained software and don't get the update.
|
| Only in the second case does it make sense to patch in a
| dynamically linked dependency. And I'm sure there's plenty of
| examples of this, but the real issue is running un-maintained
| software!
|
| I am much happier keeping in close sync with timely releases from
| Go/Rust projects than I am with Debian et. al's style of freezing
| the world.
| bombcar wrote:
| The sad truth is without the package maintaining too many
| people would be running horribly out of date and insecure
| software - as there is zero incentive to upgrade a working
| system.
| jay_kyburz wrote:
| I think if the software was labeled "insecure - requires
| update" people would update. If you don't know that something
| is insecure then there is zero incentive.
| evmar wrote:
| The OP is talking about technical details like static linking
| and language ecosystems, but if you zoom one level out, this
| comment correctly pins where the underlying problem actually
| lies: that the distribution maintainers inject themselves into
| the development process of all the software they ship.
|
| When A/B are not well-maintained upstream, distributions can
| assist by updating its dependency D, but even when A/B are
| well-maintained upstream, distributions still might monkey with
| them when D changes, even when that change to D actually breaks
| the software. To me, the root cause is that the distributions
| are effectively attempting to participate the software they
| ship, but do so by getting in the middle, rather than
| participating upstream.
|
| As a former author of some well-maintained upstream software I
| found their involvement made the software overall worse, but as
| a user of a distribution I find their maintenance of otherwise
| unmantained software sometimes helpful. In other words, I think
| the goal is admirable but the mechanism is wrong, and doing
| things like adding more dynamic linking only further enable the
| bad behavior.
| choeger wrote:
| I think it would help if distributions could snatch a piece
| of real estate in upstream software. Something like, "in
| every project, the debian/ root folder belongs to the debian
| project and follows their rules". The packaging could then
| verify this folder and put their patches, build scripts, etc.
| there. This would help upstream communication a lot, I guess.
| CJefferson wrote:
| The problem is there isn't just Debian, there is Debian and
| arch and Gentoo and nix and guix and freebsd and cygwin (at
| least).
|
| Then, how do I check who should have merge access to all
| these directories?
| choeger wrote:
| The distribution provides support for much longer than upstream
| does. Also, packaging is actual work. Just because some
| hipsters in some company consider it cool to release new
| features every 4 weeks, it does not mean that the volunteers of
| some linux distribution can keep up with thag. So if you want
| your distribution to work well, you should focus on workable
| and transparent interfaces.
|
| If you are happy with the support cycles of your upstream and
| can live with a black box, on the other hand you don't need a
| distribution in the first place.
|
| Besides, the whole "vendor everything, link statically" idea
| works well only for the leafs of the tree. Guess what Rust
| would do if llvm was _not_ a readily usable library but a bunch
| of C++ files directly used inside clang? Thanks to the Rust
| mode of operation, it is impossible to do with the rust
| compiler what they did with llvm.
| anotherhue wrote:
| > on the other hand you don't need a distribution in the
| first place.
|
| I think you're right about that, I personally want as little
| distribution as possible. FreeBSD ports or Arch AUR work well
| for me.
|
| The idea of solidifying other people's software into a bundle
| and then maintaining it (with necessarily limited expertise)
| seems like a loosing battle.
| beojan wrote:
| As someone who uses the AUR a lot, it's really the perfect
| example of how you just won't update software if it isn't
| done automatically by the distro package manager,
| particularly when you use -git packages.
| zxcvbn4038 wrote:
| If your package management is worth it's salt then you don't care
| about static linking, you just update every package that links to
| that package. IBM's AIX had the best package management ever IMO,
| you could roll forward or back and things would just work.
| Completely deterministic. On the backend whenever a package was
| updated they would regression test it and everything that
| depended on it to ensure the update didn't break anything - when
| a bug did get through they would fix it and add another
| regression test so it didn't happen again. All of that kinda
| broke when they adopted RPM as a secondary format because the
| RPMs didn't make the same guarantees.
|
| One of the best features of Golang is being able to cut a single
| static linked Linux binary that works on Alpine, Red Hat, CentOS,
| etc. With Go your also not linking to OpenSSL or Libxml, and
| those two packages are responsible for every emergency roll out
| I've ever had to do.
|
| Time and again end result of version pinning is that when forced
| to update developers are required to advance through multiple
| major version updates and what should be a patch becomes a
| rewrite. I have to constantly deal with a bunch of web sites on
| jquery 2 and the developers boo-hooing every day because jquery
| 3.5 is completely different and all I can tell them is that
| jquery 2 is no longer maintained so they need to stop using it
| for new projects and they need to update or retire their existing
| projects using it.
|
| One of the things I liked about Golang was that it didn't have
| any versions to pin so it avoided all of that nonsense
| altogether. However then they added "modules" and it's getting
| harder to not use those, so that killed that magic. Though I did
| make sure the CI/CD pipelines I use do a go get -u so hopefully
| that will force the local Go devs to keep their code fresh.
| dataflow wrote:
| Why do only *nix folks pull their hair out about this? On Windows
| programs bundle their dependencies all the time (sometimes even
| as shared libraries! but without the independent update benefits)
| and hardly anybody loses sleep over it. Heck, users actually like
| the fact that it minimizes friction. Nobody claims it's rock-
| solid security, but does it need to be?
|
| Actually, now that I wrote it above, I think I might have found
| an answer to my own question: while of course infosec experts
| will freak out about any vulnerability existing anywhere for even
| a shred of a nanosecond, in the real world this is really only a
| big deal for servers, not clients. And I'm guessing this issue
| affects Linux folks more because servers tend to run on Linux,
| with security-sensitive software exposed to the open internet all
| the time. Maybe we need to realize & embrace this trade-off
| instead of fighting it till the end of time?
|
| (I suppose one could even argue servers don't need this either if
| all the packages are kept up-to-date anyway; I guess that might
| also be debatable, but it's beside my point.)
| staticassertion wrote:
| > while of course infosec experts will freak out about any
| vulnerability existing anywhere for even a shred of a
| nanosecond, in the real world this is really only a big deal
| for servers, not clients
|
| I don't think that the linked article really reflects a
| consensus among the security community at all. I don't really
| find the "dynamic libraries are more secure" argument overly
| compelling, certainly not stated so broadly.
| GartzenDeHaes wrote:
| The unix culture comes from a shared multi-user perspective,
| whereas the windows point of view tends to be towards a single
| user desktop. In a shared environment, it's not acceptable for
| applications to change shared system components. Although this
| is _maybe_ less important today, the culture still persists.
| dataflow wrote:
| > In a shared environment, it's not acceptable for
| applications to change shared system components.
|
| That era is _long_ gone on Windows (literally since XP I
| think). In fact I 've had to do this far more on Linux than
| on Windows in recent memory... the latest instance ironically
| being the glibc CLOCK_REALTIME mess on WSL. Right now
| programs bundle whatever they need, and these go into the app
| directory (like \Program Files). e.g., I have maybe ~20
| sqlite3 DLLs on my system besides the system ones.
|
| Your larger point about the culture might still be correct
| though, I don't know.
| oblio wrote:
| You're misreading his comment. He's saying Windows is share
| nothing.
| IshKebab wrote:
| This article doesn't make a good case against static linking, and
| the author doesn't seem to understand what vendoring is either:
|
| > Bundling (often called vendoring in newspeak) means including
| the dependencies of your program along with it.
|
| No, vendoring means including a copy of the _source code_ of
| dependencies in your repo. You can bundle dependencies without
| vendoring them.
|
| The only argument presented against static linking is that when a
| library is updated rebuilding dependants takes longer (and people
| will have to download bigger updates but I doubt many people care
| about that).
|
| That may be true, but is it really that big of an issue? I
| somewhat doubt it (for sane distros that don't make users build
| everything from source themselves anyway).
|
| The author clearly doesn't like how people do development these
| days but hasn't stopped to think _why_ they do it like that.
| UncleEntity wrote:
| > The only argument presented against static linking is that
| when a library is updated rebuilding dependants takes longer
| (and people will have to download bigger updates but I doubt
| many people care about that).
|
| Not really, the argument is that instead of rebuilding _one_
| library you now have to rebuild _hundreds_ of applications to
| see the benefits for whatever the library update was
| for...assuming upstream even cares enough to bump the version
| number for the _who knows how many libs they are vendoring_
| because they have other stuff to do which is way more
| important.
| young_unixer wrote:
| Centralized repositories are not a good, general solution to the
| software distribution problem.
|
| First: They don't have all the packages and versions the user
| could ever need, which means that you'll always have a mixture of
| software installed through the packagr manager and software
| installed through tarballs or curl | bash
|
| Second: They distort the relationship between application
| developer and application user.
|
| Third: they neglect offline use cases.
|
| The only sane, generalizable solution is for the OS to be a
| platform upon which bundles can be installed. The OS should just
| define a sane format for said bundles and the user gets their
| applications directly from the application developer.
| posix_me_less wrote:
| Here is the solution to this Debian/Gentoo/<other true free
| software distribution> packager's dilemma. Packagers realize that
| their numbers are small and they can't keep up fixing all the
| modern big software (BS) projects that go against their
| philosophy.
|
| They define the core of the OS that they can keep managing in
| this traditional way (kernel, basic userland, basic desktop,
| basic server services, basic libraries, security support for all
| that).
|
| "But people want Firefox, Rust, Kubernetes, modern OS has to
| provide them".
|
| No it doesn't. Let the new complicated juggernaut software be
| deployed and configured by the users, according to developers'
| instructions, using whatever modern mechanism they prefer,
| containers/chroots/light-weight VMs, whatever. Packagers can then
| forget about the nasty big software and focus on quality of the
| core OS. Users will have all the newest versions of BS they need
| from the developers directly.
|
| There is no reason to keep fast-pace, ever-changing and very
| complicated programs/projects in the main OS distribution. It
| only leads to lots of OS-specific packaging work with mediocre
| and obsolete results, and that leads to discontent users,
| developers and packagers.
| sanxiyn wrote:
| > basic desktop
|
| I actually wish this is feasible, but it isn't. GNOME is basic
| desktop, GNOME includes librsvg, librsvg build-depends on Rust.
| So you need to package Rust (hence LLVM) to have GNOME. Your
| proposal only works if "core OS" refuses to use Rust.
| wheybags wrote:
| This is why I think the snap store is going the right
| direction. My web browser and my kernel don't need the same
| level of attention.
| tgbugs wrote:
| I think that there is a good reason to keep those projects in
| the main distribution: it makes them more robust and prevents
| the build systems and bootstrapping process from becoming a
| massive pile of technical debt.
|
| For example, in order to get clojure 10 running on Gentoo I had
| to go dig through the git history to figure out where spec-
| alpha and core-specs-alpha and clojure itself did not form a
| circular dependency in maven. Because clojure was not being
| packaged in a variety of ways, they now are at risk of making
| maven central a hard dependency, which makes the whole project
| less robust.
| posix_me_less wrote:
| Do you mean "robust for people using Gentoo" or "robust for
| everybody"? Isn't the latter responsibility of the Clojure
| developers rather than packagers?
| tgbugs wrote:
| Robust for everyone. In theory it is the responsibility of
| the Clojure devs, but if there isn't a packaging workflow
| that they are aware of, how can they be expected to
| integrate with it and make design and development decisions
| that it would reveal to them? The point is more that, the
| greater variety of different environments a piece of
| software runs in the more likely it is that systematic
| weaknesses will be revealed, sort of like chaos
| engineering, except for whole environments instead of
| sporadic failures.
| pvorb wrote:
| Exactly. When you want to install one of those fast-paced
| pieces of software on your system, you're often dissatisfied
| with the old version coming with your OS anyway.
| delroth wrote:
| You're making a distinction between "packagers" and "users"
| which does not exist. Packagers are advanced users that take
| the initiative to improve their distro when they find software
| they want to use and that isn't integrated in their distro.
|
| > Let the new complicated juggernaut software be deployed and
| configured by the users, according to developers' instructions,
| using whatever modern mechanism they prefer
|
| I'm not sure what "they" refers to here (developers or users).
| The existence of packagers is proof that some subset of users
| "prefer" that software "be deployed and configured" via distro
| package managers.
| phkahler wrote:
| >> Why do people pin dependencies? The primary reason is that
| they don't want dependency updates to suddenly break their
| packages for end users, or to have their CI results suddenly
| broken by third-party changes.
|
| Or because we dont want accidental or malicious security
| vulnerabilities to get automatically incorporated into the
| software.
|
| This stuff works both ways. You dont automatically incorporate
| fixes, nor new problems.
| lucb1e wrote:
| The vast, vast majority of updates fix security issues. It's
| like not vaccinating in case you're one of the million people
| that has an allergic reaction. Supply chain attacks are rare,
| not the norm. We hear about such things (and only rarely at
| that) because it's exceptional enough to make the news.
| eecc wrote:
| Nah, I don't buy it. If it's "just" bug fixes (for which I
| might have implemented a hack that now depends on the bug) I
| prefer Nightly builds with the latest (and re-pinned)
| dependencies available. Releases are just a re-tag after
| extra QA
| 7sidedmarble wrote:
| Yeah, but 'pinning' dependencies is useful so that you can
| choose when you get those changes.
| lucb1e wrote:
| Which means extra maintenance work to check for every piece
| of software that anyone uses whether it uses another
| library that it needs to be recompiled against and, if it
| fails, how to use the new version.
|
| If there are automatic updates, at least it either works
| and is more secure, or it breaks automatically and unsafe
| software stops working.
|
| Whether you prefer people to use MSIE6 because "it just
| works" or whether you prefer old sites that only worked
| with MSIE6 to break because it's no longer maintained,
| that's the trade-off you have to choose between.
|
| As a security person, I'm obviously biased, I can only
| advise what I see from a professional perspective. All I
| was saying above is that automatic updates being considered
| a security risk is on the same scale of odds as considering
| vaccines dangerous -- in regular cases, that is: of course
| the advice is different if you're a special (sensitive)
| organisation or a special (immunocompromised) person.
| syllogism wrote:
| This will be a somewhat intemperate response, because as a
| developer of a significant library I found this quite irritating.
|
| If you publish a Python library without pinned dependencies, your
| code is broken. It happens to work today, but there will come a
| day when the artifact you have published no longer works. It's
| only a matter of time. The command the user had run before, like
| "pip install spacy==2.3.5" will no longer work. The user will
| have to then go to significant trouble to find the set of
| versions that worked at the time.
|
| In short unpinned dependencies mean hopeless bit-rot. It
| guarantees that your system is a fleeting thing; that you will be
| unable to today publish an end-to-end set of commands that will
| work in 2025. This is completely intolerable for practical
| engineering. In order to fix bugs you may need to go back to
| prior states of a system and check behaviours. If you can't ever
| go back and load up a previous version, you'll get into some
| extremely difficult problems.
|
| Of course the people who are doing the work to actually develop
| these programs refuse to agree to this. No we will not fucking
| unpin our dependencies. Yes we will tell you to get lost if you
| ask us to. If you try to do it yourself, I guess we can't stop
| you, but no we won't volunteer our help.
|
| It's maddening to hear people say things like, "Oh if everyone
| just used semantic versioning this wouldn't be a problem". Of
| course this cannot work. _Think about it_. There are innumerable
| ways two pieces of code can be incompatible. You might have a
| change that alters the time-complexity for niche inputs, making
| some call time-out that used to succeed. You might introduce a
| new default keyword argument that throws off a *kwargs. If you
| call these things "breaking" changes, you will constantly be
| increasing the major version. But if you increase the major
| version every release, what's the point of semver! You're not
| actually conveying any information about whether the changes are
| "breaking".
| AshamedCaptain wrote:
| > In short unpinned dependencies mean hopeless bit-rot.
|
| No, this is not true, for the simple reason that there will
| _always_ be unpinned dependencies (e.g. your compiler. your
| hardware. your processor) and thus _those_ are the ones that
| will guarantee bitrot.
|
| Pinning a dependency only _guarantees you rot the same or even
| faster_ because now it's less likely that you can use an
| updated version of the dependency that supports more recent
| hardware.
| kuschku wrote:
| > No, this is not true, for the simple reason that there will
| _always_ be unpinned dependencies (e.g. your compiler. your
| hardware. your processor) and thus _those_ are the ones that
| will guarantee bitrot.
|
| Docker with sha256 tags fixes that issue (and docker
| container even specify a processor architecture).
| gpm wrote:
| > your compiler
|
| Compilers of languages like C, C++, Rust, Go etc go above and
| beyond to maintain backwards compatibility. It is extremely
| likely that you will still be able to compile old code with a
| modern compiler.
|
| > your processor
|
| Hardware is common enough that people go out of their way to
| make backwards compatibility shims. Things like rosetta,
| qemu, all the various emulators for various old gaming
| systems, etc.
|
| > your hardware
|
| Apart from your CPU (see above) your hardware goes through
| abstraction layers designed to maintain long term backwards
| compatibility. Things like opengl, vulkan, metal, etc. The
| abstraction layers are in widespread enough use that as older
| ones stop being outdated people start implementing them on
| top of the newer layers. E.g. here is OpenGL on top of
| Vulkan: https://www.collabora.com/news-and-
| blog/blog/2018/10/31/intr...
|
| > [Your kernel]
|
| Ok, you didn't say this part, but it's the other big unpinned
| dependency. And it too goes above and beyond to maintain
| backwards compatibility. In fact Linus has a good rant on
| nearly this exact topic that I'd recommend watching:
| https://www.youtube.com/watch?v=5PmHRSeA2c8&t=298s
|
| > Pinning a dependency only _guarantees you rot the same or
| even faster_ because now it's less likely that you can use an
| updated version of the dependency that supports more recent
| hardware.
|
| Dependencies are _far_ more likely to rot because they change
| in incompatible ways than the underlying hardware does, even
| before considering emulators. It 's hard to take this
| suggestion seriously at all.
| AshamedCaptain wrote:
| > Dependencies are far more likely to rot because they
| change in incompatible ways than the underlying hardware
| does
|
| Yes, that is true. It is also very likely that you can more
| easily go back to a previous version of a dependency than
| you can go back to a previous hardware. The argument is
| that, therefore, pinning can only speed up your rotting.
|
| If you don't statically link your dependencies, and due to
| an upgrade something breaks, you can always go back to the
| previous version. If you statically link, and the hardware,
| compiler, processor, operating system, whatever causes your
| software to break, now you can't update the dependency that
| is causing the breakage. And it is likely that your issue
| is within that dependency.
|
| Pinning can only make you rot faster.
| foerbert wrote:
| I don't have a particularly strong viewpoint on this, but I
| find it noteworthy that in your example the user themselves is
| asking for a specific version of the software. You don't seem
| to be intending for users to ask for simply the latest version
| and have that work, but a specific one, and you want that
| specific version to work exactly as it did whenever it was
| published.
|
| I can see some instances in which this expectation is
| important, and others where it is likely not or else certainly
| less important than the security implications.
|
| For the extremes, I see research using spaCy has a very strong
| interest in reproducibility and the impact of any security
| issues would likely be minimal on the whole simply due to the
| relatively few people likely to run into them.
|
| On the other extreme, say some low-level dependency is somehow
| so compromised simply running the code will end up with the
| user ransomware'd after just-long-enough that this whole
| scenario is marginally plausible. Then say spaCy gets
| incorporated into some other project that goes up the chain a
| ways and ultimately ends up in LibreOffice. If all of these
| projects have pinned dependencies, there is now no way to
| quickly or reasonably create a safe LibreOffice update. It
| would require a rather large number of people to sequentially
| update their dependencies, and publish the new version, so that
| the next project up the chain can do the same. LibreOffice
| would remain compromised or at best unavailable until the whole
| chain finished, or else somebody found a way to remove the
| offending dependency without breaking LibreOffice.
|
| I'm not sure how to best reconcile these two competing
| interests. I think it seems clear that both are important. Even
| more than that, a particular library might sit on both extremes
| simultaneously depending on how it is used.
|
| The only solution - though a totally unrealistic and terrible
| one - that comes to mind is to write all code such that all
| dependencies can be removed without additional work and all
| dependent features would be automatically disabled. With a
| standardized listing of these feature-dependency pairs you
| could even develop more fine-grained workarounds for removal of
| any feature from any dependency.
|
| The sheer scale of possible configurations this would create is
| utterly horrifying.
|
| At any rate, your utter rejection of the article's point seems
| excessively extreme and even ultimately user-hostile. I can
| understand your point of view, particularly given the library
| you develop, however I think you should probably give some more
| thought to indirect users - ie users of programs that (perhaps
| ultimately) use spaCy. I don't know that it makes sense to
| practically change how you do anything, but I don't think the
| other viewpoint is as utterly wrongheaded as you seem to think.
| dbaupp wrote:
| Libraries pinning dependencies only fixes a narrow portion of
| the problem and introduces a bunch of others (particularly in
| ecosystems where only a single version of a package can exist
| in a dependency tree). In particular, it is great because it
| makes life slightly easier for the library developers. However,
| if every library pinned deps, it becomes much harder to use
| multiple libraries together: suppose an app used libraries A
| and B, and A depends on X==1.2.3, while B depends on X==1.2.4.
| It's then pushed on to every downstream developer to work out
| the right resolution of each conflict, rather than upstream
| libraries having accurate constraints.
|
| Pinning dependencies in applications/binaries/end-products is
| clearly the right choice, but it's _much_ fuzzier for
| libraries.
| john-shaffer wrote:
| Can you give an example of a real ecosystem that can't handle
| such a conflict? In my actual experience, the package manager
| will either automatically use the latest version, or in one
| case has more complex rules but still picks a version on its
| own (but I stay away from that one). Your argument has force
| against either bad package managers or against using very
| strict dependency requirements, but not against pinning
| dependencies sensibly in a good ecosystem.
|
| The only conflict I've seen that can't be automatically
| resolved is when I had some internal dependencies with a
| common dependency, and one depended on the git repo of the
| common dep (via sha of a commit), and another depended on a
| pinned version of the common dep. Obviously there's no good
| way to auto-resolve that conflict, so you should probably
| stick with versions for library deps and not git shas.
| Qwertious wrote:
| >Of course this cannot work. _Think about it_. There are
| innumerable ways two pieces of code can be incompatible.
|
| There's a very simple solution here: just don't write bugs.
| anderskaseorg wrote:
| If you publish a Python library with pinned dependencies, your
| code is broken as soon as someone tries to use it with another
| Python library with pinned dependencies, unless you happened to
| pin exactly the same version of the dependencies you have in
| common.
|
| Python libraries should not pin dependencies. _Applications_
| can pin dependencies, including all recursive dependencies of
| their libraries. There are tools like Pipenv and Poetry to make
| that easy.
|
| This is less of an issue in (say) Node.js, where you can have
| multiple different versions of a library installed in different
| branches of the dependency tree. (Though Node.js also has a
| strong semver culture that almost always works well enough that
| pinning exact versions isn't necessary.)
| acidbaseextract wrote:
| The most frustrating thing is that pip doesn't make it easy
| to use more loose declared dependencies while freezing to
| actual concrete dependencies for deployment. Everybody rolls
| their own.
|
| > Python libraries should not pin dependencies.
| _Applications_ can pin dependencies, including all recursive
| dependencies of their libraries.
|
| Is the pypi package awscli an application or a library?
|
| poetry is frustrating in that it doesn't allow you to
| override a library's declared requirements to break
| conflicts. They refuse to add support [1][2] for the feature
| too. awscli for example causes huge package conflict issues
| that make poetry unusable. It's almost impossible not to run
| into a requirement conflict with awscli if you're using a
| broad set of packages, even though awscli will operate
| happily with a more broad set of requirements than it
| declares.
|
| [1] https://github.com/python-poetry/poetry/issues/697 [2]
| https://github.com/python-
| poetry/poetry/issues/697#issuecomm...
| anderskaseorg wrote:
| For this purpose, I'm defining a "library" as any PyPI
| package that you expect to be able to install alongside
| other PyPI packages.
|
| The awscli documentation recommends installing it into its
| own virtualenv, in which case pinned dependencies may be
| reasonable. There are tools like pipx to automate that.
|
| Though in practice, there are reasons that installing
| applications into their own virtualenv might be
| inconvenient, inefficient, or impossible. And even when
| it's possible, it still comes with the risk of missing
| security updates unless upstream is doing a really good job
| of staying on top of them.
| acidbaseextract wrote:
| > It's maddening to hear people say things like, "Oh if
| everyone just used semantic versioning this wouldn't be a
| problem". Of course this cannot work. _Think about it_. There
| are innumerable ways two pieces of code can be incompatible.
| ... If you call these things "breaking" changes, you will
| constantly be increasing the major version.
|
| One of the things that prompted the OP was this breakage in
| Python's cryptography package [1] (OP actually opened this
| issue) due to the introduction of a Rust dependency in a 0.0.x
| release. The dependency change didn't change the public API at
| all, but did still cause plenty of issues downstream. It's a
| great question on the topic of semver to think about how to
| handle major dependency changes that aren't API changes.
| Personally, I would have preferred a new major release, but
| that's exactly your point syllogism -- it's a matter of opnion.
|
| As a sidenote, Alex Gaynor, one of the cryptography package
| maintainers is on a memory-safe language crusade. Interesting
| to see how that crusade runs into conflict with the anti-static
| linking crusade that distro packagers are on. I find both goals
| admirable from a security perspective. This stuff is hard.
|
| [1] https://github.com/pyca/cryptography/issues/5771
| rosshemsley wrote:
| Whilst I don't totally disagree with many of the points here, I
| think there's a wider picture to many of these issues.
|
| The author is concerned with installing packages on user
| machines: which are typically very long-lived installs - maybe a
| user has the same machine with the same dependencies for years.
|
| However, for many engineers, (such as myself), a binary may not
| be used past even a few days from when it was first compiled -
| e.g. as part of a service in a a quickly continuously integrated
| system.
|
| I might even argue that _most_ software is used in this way.
|
| When software is built this way, many of the points in this
| article are very helpful to keep builds stable and to make
| deployment fast - and in fact for the case of security, we
| usually _don't_ want dependencies to auto-update, as we do not
| want to automatically deploy new code if it has not been audited.
|
| Maybe there's a future were OSs become more like this, where
| binaries are more short lived... maybe not. Although I don't
| think it's strictly fair to label all of these as "Bad" with a
| capital B :)
| ampdepolymerase wrote:
| The way iOS and Fuchsia are dealing with the problem is to
| completely lockdown the operating system with a tight
| permissions system. An app can be compromised but the damage is
| limited. Perhaps it is time for servers to move to a similar
| model.
| AshamedCaptain wrote:
| > An app can be compromised but the damage is limited
|
| AKA the "we don't care" security model. What exact use is the
| fact that the web browser is "contained" if it is
| compromised? The mail client? Your PIM program? On a server,
| what use is that the database engine is contained if it is
| compromised?
|
| I am the first to accept the security benefits of sandboxing,
| but it is just _one_ thing. It doesn't even help against the
| majority of issues. Not even on Android/iOS.
| giantrobot wrote:
| We should call this newly invented and wholly original
| concept a "container". The software gets "contained". It just
| might work. /s
| eecc wrote:
| You mean cgroups, or zones don't you? Docker (was, last
| time I heard) a security disaster, not generating robust
| layer hashes, lacking user isolation, and plenty just
| running as root...
| giantrobot wrote:
| There's more to containers on Linux than just Docker.
| loloquwowndueo wrote:
| Just install all your software from snaps on an Ubuntu system
| ;) (just kidding, snaps have a whole bunch of issues for
| server software )
| tgbugs wrote:
| Fantastic article. I now have something to point people to when
| they ask "what is wrong with pinning?" or "what is wrong with
| static linking?" or "why can't you just use pip?" Michal has had
| to deal with some pretty crazy stuff the last couple of months
| ... scratch that, years.
|
| The recent attempts to empower developers to distribute their own
| software means that there is now the potential for there to be as
| many bad security practices as there are pieces of software,
| because systematic security design that used to be managed by
| distributions has been pushed down to individual software
| projects, which have only a tiny view of the issues, and thus
| repeatedly make locally convenient decisions that are disastrous
| for everyone else. What do you mean someone is using a project
| other than ours that shares a dependency? Why would they do that?
|
| One other thing to keep in mind is that from a security
| standpoint the approach that Nix takes is not good. Nix can deal
| with pinned dependencies on a per-package basis, but if those
| pinned versions are insecure and the developers don't have a
| processes for keeping dependencies up to date, then users are
| sitting ducks.
|
| Unfortunately pinning is a symptom of at least three underlying
| issues. The first is that developers do not properly assess the
| costs of adding a dependency to a project both in terms of
| complexity, and in terms of maintenance burden. The second is
| that many of the dependent libraries make breaking changes
| without providing space for the old api and the new api to
| coexist at the same time for a certain period so that developers
| can transition over (I have been bitten by this with werkzeug and
| pint). Mutual exclusion of versions due to runtime failure is a
| nasty problem, and pinning hides that issue. Finally it seems
| that at least some developers are engaged in the Frog and Toad
| are Cofounders continuous integration story, but with a twist,
| rather than deleting failing tests, they pin packages so that
| they don't have to see the accumulating cost of pulling in
| additional dependencies. Externalities, externalities everywhere.
|
| edit: because it is just too on point https://medium.com/frog-
| and-toad-are-cofounders/technical-de...
| bscphil wrote:
| > "why can't you just use pip?"
|
| IMO, pip is _great_ , but it has exactly two use cases where
| it's warranted. One is where you need to be your own
| maintainer, e.g. you've developed a private web server to run
| your website, and you need to manage its dependency versions
| precisely. The other is for development purposes: it's really
| great to be able to use different versions of Python or test
| against different libraries than your distribution ships.
|
| It's _not_ (or shouldn 't be) for shipping software to end
| users. That's the problem being complained about in the essay.
| This particular way of handling dependencies (use system
| libraries by default, but allow the user / developer to create
| an entire virtualized Python installation if they want one) is
| actually why I think Python handles this _better_ than most
| other languages, including older ones.
| tgbugs wrote:
| I agree. The context was missing from my original, which is,
| "why can't you just use pip to install dependencies in
| production," which is effectively the answer that Michal once
| got from a PyPA maintainer when asking about setup.py
| install.
| saurik wrote:
| > I now have something to point people to when they ask "what
| is wrong with pinning?" or "what is wrong with static linking?"
| or "why can't you just use pip?" Michal has had to deal with
| some pretty crazy stuff the last couple of months ... scratch
| that, years.
|
| Literally the only argument in this article against static
| linking is "it will take an extra couple hours for the
| distribution to recompile all affected packages and then
| require the user to download a larger update, meaning time to
| fix for a security issue will be _negligibly_ longer "... since
| you seem to believe this article represents the strongest
| statement of your argument, this has actually moved me further
| away from where you want me to be ;P. (FWIW, the best argument
| I could make for dynamic linking involves memory efficiency for
| shared pages, the disk cache, and _maybe_ the i-cache, not
| security.)
| jcranmer wrote:
| > maybe the i-cache
|
| This I doubt. On a statically-linked application, all calls
| to functions are going through a regular function call
| instruction. When you dynamically link, every call to a
| function that might cross library boundaries (which on ELF
| systems defaults to every call that's not to a static
| function) instead calls into the PLT, which will itself
| dispatch through to the underlying function call.
| 7800k wrote:
| Gentoo is secure, as far as you can be secure by building a bunch
| of code few have time to review on hardware few have time to
| fully understand running in an insecure world.
|
| So, I don't fault it.
|
| However, if you don't include dependencies and you don't manage
| them, which would be the case in modern environments for the
| majority of users in the world, how is that safe?
|
| Answer: it's not.
| setheron wrote:
| Once Nix moves more to fixed output derivations this will be
| killer!
|
| You can easily upgrade dependencies by having the linker resolve
| to different libraries without needing to rebuild the world while
| still retaining a sensible memory model.
| wheybags wrote:
| Nobody cares how secure my code is if it doesn't work.
| [deleted]
| steveklabnik wrote:
| There is a lot I could say about this article, but I've kinda
| been in this whole Texas natural disaster situation. But I do
| think it's worth pointing out one thing:
|
| > Rust bundles a huge fork of LLVM
|
| It is not super huge, and we try to upstream patches regularly to
| minimize the fork. We also regularly re-base all current patches
| on each release of LLVM, so it's less 'fork' and more 'maintain
| some extra commits to fix bugs.'
|
| > and explicitly refuses to support to distributions using the
| genuine LLVM libraries.
|
| We always support the latest LLVM release, and try to maintain
| compatibilities with older ones as long as is reasonable and
| possible.
|
| IIRC, the last time we raised the base LLVM requirement was in
| Rust 1.49, at the end of last year. The minimum version it was
| raised to was LLVM 9, which was released in September of 2019.
| The current release is 11.
|
| Again, because of the lack of the extra patches, you may see
| miscompilation bugs if you use the stock LLVM. There's pros and
| cons to every choice here, of course.
|
| For more on this, please see https://rustc-dev-guide.rust-
| lang.org/backend/updating-llvm....
|
| The Rust project is and has been interested in collaborating on
| issues to make things easier for folks when there's demand. We've
| sought feedback from distros in the past and made changes to make
| things easier!
|
| (I also posted a slightly modified version of this to their blog
| comments as well. EDIT: the author has commented that suggesting
| Rust's fork is large was in error)
| Daho0n wrote:
| At a minimum a reasonable time for a package for it to be a
| good citizen in the system is what is in Debian LTS.
| steveklabnik wrote:
| EDIT: you completely changed your comment. My original
| response is below the line, but I'll also respond to your new
| one above it.
|
| I don't actually understand grammatically what you're asking
| for or saying in this comment. Rust, Cargo, and a bunch of
| Rust applications are packaged in Debian today, from Buster
| onwards.
|
| I _think_ what you 're saying is that rustc head should be
| able to be built with the llvm that lives in Debian stable?
| That is a choice that could be made, I'm sure, but then
| you're talking about supporting five API-incompatible
| releases of LLVM. The folks from Debian aren't even asking
| for that level of compatibility, and it would be quite a bit
| of work, for unclear benefit.
|
| --------------------------------------------
|
| There are lots of demands, from lots of people. Managing a
| project requires prioritizing different demands of all of
| your different users.
|
| > When will the problem mentioned in the article
|
| The article talks about many more things than one single
| problem. Some range from easy, like the "we already do let
| you use stock llvm" I mentioned above, and some are difficult
| and take a long time, like "define a Rust ABI." The needs of
| packagers need to be balanced with the needs of other folks
| as well. "define a Rust ABI," specifically, for example,
| would help some packagers, but it may harm other users. It
| also may be something that's good to have, but we aren't
| quite ready for it yet.
|
| A willingness to _work together_ to solve issues, and to find
| solutions that help as many different kinds of folks as
| possible, is what matters here. We have had folks from other
| distros pitching in to help make things better for their
| distros, and I can 't imagine we wouldn't accept the help of
| the OP either.
| archi42 wrote:
| What's the problem with focusing on upstreaming the patches and
| getting rid of rust's "staging fork"? Especially if they fix
| something as serious as miscompilations, wouldn't those patches
| be VERY much needed upstream? I'm asking out of genuine
| interest.
| steveklabnik wrote:
| Quoting from the documentation I linked:
|
| > Strongly prefer to upstream all patches to LLVM before
| including them in rustc.
|
| That is, this is already the case. We don't _like_
| maintaining a fork. We try to upstream as much as we can.
|
| But, at the same time, even when you do this, it takes tons
| of time. A contributor was talking about exactly this on
| Twitter earlier today, and estimated that, even if the patch
| was written and accepted as fast as possible, it would still
| take roughly a year for that patch to make it into the
| release used by Rust. This is the opposite side of the whole
| "move slow and only use old versions of things" tradeoff:
| that would take _even longer_ to get into, say, the LLVM in
| Debian stable, as suggested in another comment chain.
|
| So our approach is, upstream the patches, keep them in our
| fork, and then remove them when they inevitably make it back
| downstream.
|
| (EDIT: Also, what Rusky said: there's basically always going
| to be more than one patch, in various stages of this
| process...)
| archi42 wrote:
| Ok, maybe I'm just to spoiled by my previous experience
| then. I never had any problems getting patches upstreamed
| relatively quickly, but then e.g. meson or picolibc are
| much smaller than LLVM and patches are simpler to reason
| about (I know some compiler construction).
|
| Thanks for the answer.
| Rusky wrote:
| That's already done. The point is that it takes time, and
| there are always more of them, so at any point in time there
| are likely to be a handful of them still in-process of moving
| upstream.
| [deleted]
| larusso wrote:
| It's sounds like pinning dependencies is just done because we
| developers are a lazy bunch that just want to guard against the
| rare case of a breaking upstream change. I was burned too often
| in the past by breaking changes or more often behavior changes in
| transient dependencies to not put up some defense. But I still
| don't pin versions in the main dependency files (cargo.toml,
| gemset or similar). I just have the generated lockfile and put
| that under version control. so all developers in the team get the
| same versions of direct and transient dependencies. I define the
| dependencies with version ranges. Every package manager works
| differently here. I like cargo the best since a statement like
| 'version = 1.2.1' means give me a version bigger than '1.2.1' but
| lower than '2.0.0'. One should never add lockfiles for library
| projects as they need to be open for future updates. I also add
| dependabot to all repos and let the bot inform me about updates.
___________________________________________________________________
(page generated 2021-02-21 23:00 UTC)