[HN Gopher] Torvalds: Shared libraries are not a good thing in g...
___________________________________________________________________
Torvalds: Shared libraries are not a good thing in general
Author : pengaru
Score : 199 points
Date : 2021-05-01 19:22 UTC (3 hours ago)
(HTM) web link (lore.kernel.org)
(TXT) w3m dump (lore.kernel.org)
| smitty1e wrote:
| It strikes me that packaging applications & dependencies as
| Docker containers is a huge empirical example of Torvalds' point.
| userbinator wrote:
| IMHO it's really the way Linux and other Unix-likes do dynamic
| linking that's "not a good thing", as well as the
| "hypermodularisation" that Linus alludes to (the left-pad fiasco
| would be an extreme example of that.)
| marcus_holmes wrote:
| Windows does DLL's too
| gsich wrote:
| True, but I have seen software where the DLL files are in the
| directory where the EXE lies. So copy+paste from another
| computer works just fine (in most cases).
|
| This does not really work in Linux.
| incrudible wrote:
| Left-pad wasn't really a fiasco. It briefly broke a couple of
| people's build. If anything, it showed how resilient the system
| is. People are actually starting to use NPM as a _system_
| package manager.
|
| Meanwhile, dependency hell on Linux distributions keeps people
| stuck on outdated versions of software, because some piece
| somewhere is either too old or too new.
| VWWHFSfQ wrote:
| > It briefly broke a couple of people's build.
|
| do you think this is an accurate characterization of what
| happened
| incrudible wrote:
| It's an accurate representation of the actual technical
| fallout.
|
| As for the "this is a sign of the end times" narrative that
| has been spun around it, it's mostly just for the
| headlines.
|
| I wouldn't even disagree that it _is_ a sign of the end
| times, but practically speaking, it just wasn 't a big
| deal.
| fxtentacle wrote:
| From what I observe, most people use dynamic libraries simply
| because they cannot get stuff setup correctly for linking
| statically.
|
| So they're usually not shared, but merely use the shared library
| file format. For a practical example, most software on Windows
| will bring its own folder with DLLs.
|
| One step further, this is probably why we're now seeing docker
| everywhere: it's a Linux equivalent to Windows and Mac app store
| apps.
| thraway123412 wrote:
| I agree that shared libraries are mostly unneeded complexity, as
| far as free & open source programs are concerned.
|
| I kinda like how they open up opportunities for fixing or
| tweaking the behavior of broken proprietary binary-only garbage
| without requiring you to RE and patch a massive binary blob. Of
| course, in a better world, that garbage wouldn't exist and we'd
| fix the source instead.
| ExcavateGrandMa wrote:
| Everything is not good thing in general...
| dTal wrote:
| "Don't abstract your code into functions! Just copy and paste it
| all over the place. That way, when you need to change the
| behavior of one of them, the rest don't break!"
|
| /s
|
| Abstraction is _good_. Surely anyone can see that having _one_
| copy of code is superior - technically, and from a standpoint of
| elegance - than duplicating it once per program. A huge amount of
| effort has gone into making this infrastructure work. This isn 't
| just about disk space, or memory, or the ease of patching - it's
| about composability. I _like_ that I can override behavior with
| LD_PRELOAD. If we think of our systems as a collection of
| unrelated, opaque blobs, then static linking looks seductively
| simple; but if we think of them as a network of interacting code
| that transcends application boundaries - vivisected and
| accessible, like a Lisp machine - then static linking is a
| repulsive act of violence against deeply held aesthetic
| principles.
|
| If dynamic linking doesn't work well - if we've messed it up -
| that's a sign we should try harder, not give up.
| eddieh wrote:
| On desktops this is 100% correct. On mobile it sucks to have to
| ship stuff statically linked (or even dynamically linked) but
| included in the .apk or .ipa. But mobile is a different world, it
| is more like the past where every resource is constrained. You
| have limited storage (and storage is relatively expensive),
| limited memory, limited bandwidth (despite what the carriers want
| the consumers to believe), and the app stores impose limits too.
| tediousdemise wrote:
| I just made a generic version of the title since I see it every
| so often:
|
| _Torvalds: [hot take that almost no one else agrees with]_
| pengaru wrote:
| > Torvalds: [hot take that almost no one else agrees with]
|
| Considering the popularity of vendoring and/or static linking
| dependencies, I'm not sure where you're getting the "almost no
| one else agrees with" from.
|
| glibc's hostility towards static linking is arguably a
| substantial part of why musl has attracted so much interest.
| Static linking support is explicitly stated as one of musl's
| core tenets [0].
|
| One can also argue the popularity of containers has in part
| been due to glibc's dynamic linking requirements, and
| containers have become the bloated "modern" version of static
| linked binaries, bundling all the dependencies without eliding
| all the dynamic linking overhead.
|
| Torvalds is definitely not alone in this particular opinion.
|
| [0] https://www.musl-libc.org/intro.html
| attractivechaos wrote:
| > _glibc 's hostility towards static linking is arguably a
| substantial part of why musl has attracted so much interest._
|
| Anothre reason is that glibc is equally bad for dynamic
| linking. Often a binary compiled on recent glibc doesn't work
| with older glibc. It is certainly possible but quite
| challenging to compile portable executables on Linux.
| AnthonyMouse wrote:
| > Considering the popularity of vendoring and/or static
| linking dependencies, I'm not sure where you're getting the
| "almost no one else agrees with" from.
|
| Vendoring is hugely popular for projects in year zero,
| because it makes it easier to ship. Until they learn that
| packaging specific versions of all your dependencies with
| your project implies taking on responsibility for backporting
| all security updates to that version of every dependency when
| upstream is only actively maintaining the latest version. Or
| you have a project riddled with known published security
| vulnerabilities, which is what most of those turn into.
| eikenberry wrote:
| > implies taking on responsibility
|
| You use a library, you've already taken on responsibility
| for it. You're application has bugs due to bugs in a
| library, you're users aren't going to accept "we reported
| it to the library and they said it wasn't important" as the
| resolution.
| AnthonyMouse wrote:
| At which point you either submit a patch to the library
| or switch to a different library. That much of the work
| isn't avoidable, except maybe by choosing a better
| library at the outset.
|
| But having to fix it yourself when the library
| maintainers fail once is a far cry from having to
| backport every security update in every library yourself
| even if the library maintainers are properly doing it for
| a newer version than you're using.
|
| The problem is that doing the backporting is typically
| more work than updating your software to use the newer
| version of the library, but not maintaining the library
| version you use at all is invisible in the way that
| security vulnerabilities are until somebody exploits
| them.
| lanstin wrote:
| You have two choices here. Keep updating as you go along
| and amortize the price for so much open source deps
| inability to avoid breaking change; or don't do that and
| then eight years later lead a heroic update the deps
| project that takes ten people and millions of dollars and
| some availability hits. You keep the principal of change
| only one thing at a time the first way and get promotions
| the second way.
|
| Static vs dynamic is just do you want the code owners to be
| responsible or some centralized ops group to be
| responsible. Usually ops doesn't have the insight to do a
| good job with deps, so it's better to have dev do it, e.g.
| statically linked go binaries, statically linked C/c++,
| frozen package versions in Python, I think mvn has a
| similar capability, also version pinned base container
| images etc etc. hence the fact all the packaging systems
| add new versions and only rarely remove things as that
| breaks it all worse than a security hole.
|
| As a dev person with a lot of ops experience I like static
| because I don't want some janky dependency change to come
| into the execution path when I am not aware of making a
| change. On the other hand, people usually just freeze
| everything and then don't worry about it, which is wrong.
| But your security team needs to review source and source
| deps not just deployed versions. So if code is still owned,
| the bad deps can be found and fixed either way.
| lanstin wrote:
| I think a case for dynamic loading of your SSL library
| can be made.
| hibbelig wrote:
| If you are using upstream libxyz 1.3 but they don't provide
| security updates for it then you should do something even
| if you are using dynamic linking. Imagine that redhat
| provides security updates but Debian uses 1.4, are you
| going to tell your users that they must use redhat?
|
| I don't see that dynamic linking makes a lot of difference
| here.
| AnthonyMouse wrote:
| Typically with dynamic linking, the library is maintained
| as a separate package by a separate party who takes
| distribution-wide responsibility for backporting
| anything.
|
| So you have two alternatives. One is that you make your
| software compatible with both version 1.3 and version
| 1.4, so it can be used on both Redhat and Debian with
| dynamic linking. This is a bit of work up front which is
| why everybody grumbles and looks for an alternative.
|
| The other is that you vendor in version 1.3 so that
| you're using version 1.3 even on Debian. But now that
| library is your responsibility to patch instead of the
| distribution's. So instead of doing that work up front,
| you either maintain the library yourself on an ongoing
| basis (which may be more work than supporting two only
| slightly different library versions up front), or you
| neglect to do it and have a security vulnerability.
| Orphis wrote:
| You can also automate the upgrade process with good
| tooling.
|
| Detect a new version? Update your local copy, run your CI,
| if it passes, you can automatically merge. Or you could ask
| for someone of your team to review the changes, knowing
| your tests passed. And if it doesn't, you can warn your
| team, who can then either fix the build manually (hopefully
| a small incremental change) or open a bug report to the
| library.
|
| The point is to do that continuously, so that the deltas
| are smaller and easier to handle. I do that with some
| pretty critical libraries at work and it's mostly fine,
| takes a few minutes usually to approve changes every week.
| Doing an upgrade after 6 months+? That'd require a long
| rollout.
| tediousdemise wrote:
| Maybe I hyperbolized that a bit. I think there are pros and
| cons to each approach. The uses cases for static vs dynamic
| are also different. There are compile time disadvantages to
| static libraries, and there are security disadvantages to
| shared libraries. But calling one method better than the
| other is inane.
| Arnt wrote:
| Are you aware of the runtime differences? Both time and
| space.
|
| Shared libraries (at least with ELF) requires fixups, which
| involve COW and a considerable number of per-process pages.
| Compiling with -fPIC reserves one register, which isn't a
| problem unless you're on a register-starved CPU, which is
| all of them.
| floatboth wrote:
| > register-starved CPU, which is all of them
|
| Really? None of the 64 bit ISAs seem particularly
| starved, especially not aarch64.
| kelnos wrote:
| I think the parent's point is that all registers are
| precious, and freeing up even one by avoiding dynamic
| might be worth doing.
| zozbot234 wrote:
| > Shared libraries (at least with ELF) requires fixups,
| which involve COW and a considerable number of per-
| process pages.
|
| You can avoid load-time fixups via prelinking.
| incrudible wrote:
| Imagine a world where Linux binaries "just worked", every
| distribution's major selling point, the package manager, would
| be obsolete.
|
| These people are propagandizing the bogus security argument
| that static linking is bad practice.
|
| Take the red pill: statically linked executables are actually
| _good practice_.
| CrLf wrote:
| I don't think this is the case here. It's not hard to agree
| that shared libraries are meant to be _shared_ among different
| applications and when they're not (or not shared enough, which
| is a subjective measure) it's best to statically link.
| DoomHotel wrote:
| From some of the replies further down that thread, it appears
| this is not as big a problem as it first seems.
|
| First of all, the dynamic linker is slow and not optimized for
| program startup times. [
| https://lore.kernel.org/lkml/4b77489c05f6459ead7233c3fb69f61... ]
|
| And second, compiling clang with Profile Guided Optimization
| makes a big difference. [
| https://lore.kernel.org/lkml/YItU3YrFi8REwkRA@archlinux-ax16... ]
| stephc_int13 wrote:
| In my opinion, this idea of shared libs should have been
| abandoned at least two decades ago.
|
| I see it as a huge weakness on the GNU/Linux world because it's
| so widespread.
| ratww wrote:
| It's also a weakness in other systems IMO.
|
| In Windows, "DLL hell" used to be a thing. It's mostly solved
| by (among other things) Microsoft forbidding installers from
| overwriting system DLLs. Also today most apps bundle the DLLs
| along with the apps and don't really share them.
|
| In MacOS, brew the popular package manager recently took upon
| using shared libs for everything. While it mostly works it can
| cause some version incompatibilities, you tend to accumulate
| garbage in your disk after uninstalling stuff and now you have
| to routinely update 3x more packages...
| forrestthewoods wrote:
| I quite like shared libraries. What I don't like is actually
| sharing them.
|
| I firmly believe that all applications should ship their
| dependencies. Launching an application should never be more
| complication than "unzip and double-click executable".
|
| Docker only exists because run-time environments are too
| complicated, convoluted, and mutually incompatible. It's insane.
|
| > Docker containers are kind of neat. They are also kind of a
| craven surrender to the rotting mess of excessive software
| complexity. - John Carmack
| (https://mobile.twitter.com/id_aa_carmack/status/138510311097...)
| marcus_holmes wrote:
| this really feels like one of those roundabouts that CS/IT does.
|
| "every program the system runs depends on library foo, so why is
| this loaded into memory x times? Let's load it once and share it"
|
| "there are 10 versions of library foo, none of them are
| backwardly compatible. Every application loads its own version of
| foo. There are now at least 5 versions of foo in memory at once.
| Why is the OS trying to manage this? We should statically link
| every application and stop worrying about it"
|
| This has already gone back and forth at least once since I
| started working in the industry.
| herf wrote:
| When you look at the differences in speed between distros, often
| the main thing is which glibc is being used!
|
| Regular string ops can vary in speed by 2-5x based on which old
| glibc you have.
| nix23 wrote:
| Time that Linux suckless right?
| ttt0 wrote:
| To be fair, LLVM is not just clang. It's also clang++, lld, lldb,
| clang-format, clang-tidy, clangd and dozens of other utilities
| that rely on the LLVM library.
| phendrenad2 wrote:
| Ah he gets it. Shared libraries are a good idea if you only
| consider performance and idealized scenarios. In the real world,
| where users have to use programs that they want to not be broken,
| vendoring absolutely everything (the way Windows apps do it) is
| essential.
| macksd wrote:
| Huh - are DLL's not a thing anymore?
|
| IMO the best reason to use shared libraries has always been
| cases where updates are likely critical and there's a well-
| defined interface that is unlikely to have breaking changes.
| Like libssl. If there's a security fix, I'd love to have `apt
| update` apply that fix for everything, and not worry about
| verifying that everything that might have statically linked it
| gets updated too. Although ideally the latter would be easy
| anyway.
|
| It's also nice for licensing - being able to have dependencies
| that you don't have to redistribute yourself.
| rodgerd wrote:
| > If there's a security fix, I'd love to have `apt update`
| apply that fix for everything, and not worry about verifying
| that everything that might have statically linked it gets
| updated too. Although ideally the latter would be easy
| anyway.
|
| While it is one failure mode, and a well-understood one, to
| end up finding that you've been owned by an application with
| a hidden dependency that had a security hole that you didn't
| know about, the "apt update" means that your ability to fix a
| security hole is now waiting on the time it takes you to
| update thousands or tends of thousands of programs, 99% of
| which you don't and never will use, against the One True Copy
| of the library.
| city41 wrote:
| Many apps on Windows bring their own local dlls that no other
| apps on the system use.
| R0b0t1 wrote:
| To expand on the other answer you got, there's nowhere to put
| DLLs by default. You install what you need in your directory
| in Program Files and at that point you may as well statically
| link them.
| alkonaut wrote:
| > Huh - are DLL's not a thing anymore?
|
| There are _some_ shared dll 's too, perhaps most notably the
| OS APIs. But 99% of the dll's on my system sit in
| Programs\specificprogram\x.dll
|
| "Dynamically linked", doesn't imply "Shared between multiple
| executables". The trend on windows just like on Linux and Mac
| is for bundled-everything even for the thing you'd think was
| last to get there on windows: things like C++ runtimes and
| .NET frameworks are now bundled with applications.
|
| The tradeoff between patching in one place, and being forced
| to maintain backwards compatibility and effectively make the
| runtime an OS component was won hands down by the bundle
| everything strategy.
| jollybean wrote:
| 'Dynamic linking' and 'shared' are different things though.
|
| You can have DLL's bundled with your app, so that the app
| links at runtime, but really they're not for 'sharing' at the
| OS level so to speak.
|
| I think the number of DLL's shared should be very small, and
| as you say 'very stable ABI' like for OS level things.
|
| Everything else - in the bundle.
| ratww wrote:
| As others mentioned, most now sit alongside the executables.
|
| To give some historical perspective of why, check out this
| Wikipedia article on "DLL hell", more specifically the "DLL
| stomping" section:
|
| https://en.wikipedia.org/wiki/DLL_Hell
| CalChris wrote:
| I like this (shared libraries _are_ mindlessly overused) but
| there were two points I didn 't understand. And
| the memory savings are often actually negative
|
| Not sure but I think this is for a library with only a single
| user (ever). The GOT ... and the larger call/ret code cost
| memory. And the dynamic linking will actually
| cause a lot _less_ sharing because of all the fixups.
|
| I have no idea why fixups cause less sharing.
| atq2119 wrote:
| Fixups are done by usermode code, which means the pages
| containing them can't be shared between different processes.
| The kernel has no visibility into _what_ is being fixed up.
|
| (Also, the fixups themselves could actually be different
| between different processes due to address space layout
| randomization.)
| CalChris wrote:
| Thanks, I've always thought of fixups as being a linker
| thing. But I guess it's more general than that.
|
| https://llvm.org/doxygen/classllvm_1_1MCFixup.html
| summerlight wrote:
| > And the memory savings are often actually negative
|
| If you consider inlining and dead code elimination from inter-
| procedural optimizations, this could sometime offset the
| savings from a library shared across 2~5 binaries.
| marcus_holmes wrote:
| Your program uses one function from a library.
|
| If that library is statically linked, that one function is
| compiled into your program. The rest of the library is ignored.
|
| If that library is dynamically linked, the entire library is
| loaded into memory, and the function address is shared with
| your program. More memory used.
|
| The advantage of dynamic linking is if that specific version of
| the library is already in memory (because another application
| is using it) then it takes up no additional memory. Less memory
| used.
| quelsolaar wrote:
| A good reason to use dynamic libraries is for plug-in systems.
| The application can then dynamically find out what plugins are on
| the machine and run them. If a plugin isn't installed nothing
| breaks. It makes it possible to have a open API to a closed
| application. Say if you have a paint application and you have
| plugins for file loading, then users can write/install additional
| file formats as plugins, without the app changing. It can also be
| helpful in development since you can move dependencies to
| separate modules. If a file loader requires libpng, then you need
| that dependency to rebuild the plugin but not the core
| application.
|
| In general I agree with Linus, an application should not be
| separated in to multiple parts unless it serves the user.
| kragen wrote:
| That's what he said in his last paragraph, quelsolaar.
| quelsolaar wrote:
| I read it as, modules that are loaded and unloaded
| dynamically, facilitating things like the ability to update a
| running application. On second reading you are probably right
| that plug-in interfaces even if plugins are only loaded at
| startup, would fall in to the category he describes.
| pdkl95 wrote:
| As Linus points out, the utility of shared libraries depends on
| the situation. Widely used libraries like libxcb, libGL, and zlib
| that tend to have reasonably stable ABIs are good candidates for
| sharing, while the "couple of libraries that _nobody_ else used.
| Literally _nobody_ " should obviously be statically linked.
| Software engineering is mostly about learning how to choose which
| method is appropriate for the current project.
|
| However, there important advantage of using .so files instead of
| building into a big statically linked binary is _NOT_ that a
| library can be shared between different programs! The important
| benefit is _dynamic linking_. The ability to change which library
| is being used _without_ needing to rebuild the main program is
| really important. Maybe rebuilding the program isn 't practical.
| Maybe rebuilding the program _isn 't possible_ because it's
| closed/proprietary or the source no longer exists. If the program
| is statically linked, then nothing can happen - either it works
| or it doesn't. With dynamic linking, changes are possible by
| changing which library is loaded. I needed to use this ability
| last week to work around a problem in a closed, proprietary
| program that is no longer supported. While the workaround was an
| ugly LD_LIBRARY_PATH hack, at least it got the program to run.
| herodoturtle wrote:
| > that tend to have reasonably stable ABIs
|
| Today I learned about Application Binary Interfaces.
|
| (and thanks for the generally insightful comment)
| innagadadavida wrote:
| Another important consideration is security. If there is an
| exploit in the library code and the original program authors do
| not release updates regularly, then it is really important to
| be able to patch the security issue.
| amelius wrote:
| Perhaps then the OS should figure out which parts of an
| executable are common, so we can still have shared libraries
| except they are smarter and not visible to the user.
| choeger wrote:
| You are right on track here. The complexity is real, but so are
| the benefits of a modular executable vs. a huge monolith.
|
| I fully understand why languages like rust demand static
| linking (their compilation model doesn't allow dynamic
| linking). But once you encounter a C-API boundary, you can as
| well make that a dynamic C-ABI boundary.
| brundolf wrote:
| It seems to me that several circumstances have changed since
| the idea of dynamic linking first came about:
|
| - A whole lot of software is now open and freely available
|
| - The code-changes-to-binary-updates pipeline has been greased
| by a couple orders of magnitude, from internet availability to
| CI to user expectations around automatic updates (for both
| commercial and non-commercial software)
|
| - Disk space, and arguably even RAM, have become very cheap by
| comparison
|
| Given these, it seems worthwhile to re-evaluate the tradeoffs
| that come with dynamic linking
| drewcoo wrote:
| To underscore the gp's point, I have had to work with a 3rd
| party-provided .so before. No sources. A binary. And strict
| (if unenforceable) rules about what we could do with the
| binary.
| ndesaulniers wrote:
| Oh look, our conversation's made the news! _goes back to writing
| a cpufreq driver for my Nintendo 3DS_
| cjohansson wrote:
| This sounds like an opinion that welcomes Rust and Go since they
| use static linking or am I wrong?
| loeg wrote:
| I think you can dynamic link Rust, but the ABI is unstable.
|
| https://doc.rust-lang.org/reference/linkage.html
| [deleted]
| kmeisthax wrote:
| _Kind of_.
|
| Go already has several other deal-breakers for systems
| programmers (garbage collected everything, runtime-provided
| greenthreading, etc). Rust is a lot closer to what Linus would
| approve for in-kernel development, but it still has a few
| problems, such as failed allocations panicking the system
| instead of returning an error that can be handled normally.
| LukaD wrote:
| Torvalds and the OP are talking about userland software.
| _ph_ wrote:
| Go is by default statically linked. You can however create
| shared libraries from Go which can be called like any C library
| and call any shared library.
| encryptluks2 wrote:
| I think he has a good point overall, but at the same time I think
| there can be an equal benefit of shared libraries when it is
| likely that the library will be shared among multiple
| applications. It doesn't have to be a tremendous amount.
|
| On Arch, users pick their display manager and windows system, and
| many Qt applications or GTK applications may share multiple
| libraries and I can see the benefit of smaller packages and being
| able to update those libraries separately.
| kelnos wrote:
| Torvalds specifically calls out those kinds of shared libraries
| and considers it reasonable to leave them shared.
| [deleted]
| Jach wrote:
| Note to 1-bit people: saying that "X is not a good thing in
| general", is not the same as saying that "X is a bad thing in
| general". All that's being said here is "tradeoffs exist" and
| highlighting some issues some people apparently like to ignore.
|
| One of my favorite libraries is SDL. Of course, when SDL2 came
| out, they moved away from the LGPL license to a more liberal zlib
| license, which allows static linking without infecting the
| license of your project. Great, except now as years go by and
| games are abandoned, they stop working in new environments, or at
| least don't work as well, and you can't just force in a new
| dynamic version of SDL to save things like you used to be able
| to.
|
| The solution they came up with: even if the program is static
| linked, the user can define an environmental variable to point to
| a dynamic lib and that will be used instead. Best of both worlds,
| at least going forward for programs that use a version of SDL2
| that incorporates the feature. (Some more details:
| https://www.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a...)
| RcouF1uZ4gsC wrote:
| I think one area where shared libraries is are a huge toll is in
| C++. ABI concerns have crippled a lot of improvements to the
| library. In addition, because C++ development by default uses the
| system C++ standard library, many times the latest C++ languages
| and libraries can't be used.
|
| In addition, with regards to saved disk space and central
| patching, Docker negates a lot of those benefits. Basically
| instead of static linking, you end up shipping a whole container
| with a bunch of shared libraries
| Yaa101 wrote:
| Shared libraries are a tradeoff from a time (80's and 90's) where
| big projects like GUI systems needed to be run on small
| computers.
|
| Today with the cheap memory resources and fast interfacing you
| are often better off with programs that are statically linked as
| they have no run dependencies other than the OS.
| zabzonk wrote:
| > big projects like GUI systems needed to be run on small
| computers
|
| I'm not sure that I would want all of Windows libraries to be
| statically linked to all my applications.
|
| Sometimes you want static linking - sometimes you don't.
| oscargrouch wrote:
| Its not all the windows libraries, but often the OS creates
| just one interface library to wrap the syscalls. Those do not
| tend to be big libraries where you should worry about being
| statically linked to your program.
|
| The problem with Linux also have to do with license, giving
| GPL can get some program into trouble by statically linking
| GPL code.
|
| There are 3 cases for me that can force one to use shared
| libraries: Licensing, Dynamic loading a DSO at runtime (like
| in Apache modules) and when you are developing a big project,
| but once you need to ship the production code, get it back to
| statically linking everything.
|
| Other from that i don't see why you would want this.
| ex3ndr wrote:
| GUI is something like an API, let's take React Native - you
| just have an api to rendering (via bridge), same for X
| server, same for almost any GUI framework.
|
| This is a very very tiny part of an app. UI libraries with
| prebuilt controls are already almost always bundled with
| apps.
| ratww wrote:
| Well, in Windows and MacOS you _need_ libraries like
| kernel32.dll etc to sit in-between userland and kernel
| because the kernel ABI is not stable, like Linux.
|
| Also, Linus talks about this sort of exception in his post:
| _" unless it's some very core library used by a lot of things
| (ie particularly things like GUI libraries like gnome or Qt
| or similar)"_.
| rvz wrote:
| Exactly. I don't see the benefits of shared libraries in terms of
| software distribution. It makes sense for them to be used as
| standard system libraries, like in macOS with System.dylib or
| libc.so in many distros.
|
| For general apps using third-party libraries, nowadays static
| libraries is the way to go, as I have said months ago [0].
|
| [0] https://news.ycombinator.com/item?id=24606580
| lamontcg wrote:
| Dynamic linkers aren't really dynamic enough.
|
| If I don't use any features that require libbz2 or whatever a
| piece of software shouldn't complain if libbz2 isn't there on the
| system. If I add it to the system I should be able to use those
| features without recompiling and relinking everything.
|
| Half of the features of autoconf such as "is this library with
| this method signature available?" should be moved into the
| dynamic linker.
|
| Software would need to be rewritten to be more lazy and modular,
| but it'd make dynamic shared libraries actually dynamic. Having
| to make all those decisions at compile time instead of run time
| isn't very dynamic.
| tyingq wrote:
| Dynamic loading does exist. I guess you're just saying it's not
| used often enough? Many interpreted languages use it
| extensively.
| ChiefOBrien wrote:
| Torvalds:
|
| > Disk is cheap
|
| I certainly don't want each installed desktop app to have a copy
| of base gnome/kde runtime and everything down to libc. And the
| implication is even the graphics would be duplicated, for example
| the Adwaita icon pack is huge. So if I have a full set of gnome
| applications (say 50) would I have 50 copies of Adwaita icon set?
| Suddenly disk space isn't cheap. Shared libs are good and we
| could do better than flatpaks and containers and static linking.
|
| And just because shared libs are PITA it's not just because of
| their nature, it's the lack of tooling from supposedly modern
| languages, lack of guidance, lack of care for versioning and API
| stability, and distro agnostic convention. Each of these problems
| can be solved by not sweeping them under the rug.
| somebodythere wrote:
| I think you can deduplicate at the OS level
| nix23 wrote:
| Great idea and hold the hashtable of all files in the
| ram...what a great idea.
| Traster wrote:
| I don't know whatyour point is. He literally says:
|
| >Yes, it can save on disk use, but unless it's some very core
| library used by a lot of things (ie particularly things like
| GUI libraries like gnome or Qt or similar), the disk savings
| are often not all that big - and disk is cheap
|
| He's literally making the point you're arguing. He says, core
| libraries _should_ be shared.
| ChiefOBrien wrote:
| Tell me how much of the system libraries are written in C or
| C++, and how much of them are being written in newer
| languages.
|
| C is the default choice because of its ABI, and the tooling
| around it is made for using shared libraries.
|
| What can you say about modern languages? Each of the
| languages are designed to work in a silo and not much
| cooperation for let's say, system plumbing. Their own package
| manager makes it easy to pull in code, and only make code for
| that language only. You can't just create a package that
| works as a shared library for other programs without mushing
| around C FFI. They make it hard by default, which creates
| resistance in developers to make a piece of code usable by
| others other than their own language. This trend is pretty
| alarming, especially when hidden and manipulative language
| fanboyism is showing its ugly head everywhere.
| jameshart wrote:
| We'll surely individual applications only use a few icons, not
| the entire set, so they can statically link in only the
| resources they actually depend on, right?
| superkuh wrote:
| Shared libraries libraries amplify future shock. Combined with
| the modern rapid pace of new features being added to libraries
| and languages themselves even distros that aren't even released
| yet are already out of date and unable to compile and run $latest
| application. The symptom of this amplified future shock is the
| rise of containers on the desktop.
|
| The problems of updating applications individually because
| they're statically compiled are insignificant compared to the
| problems that come from having to do everything within
| containers.
| ufo wrote:
| As Serge Guelton said in a followup email, one of the reasons
| that Linux distros use dynamic linking is to reduce the disk
| usage and bandwidth requirement for updates.
|
| Technically speaking, would it be possible to design a Linux
| distro so that the installed executables are statically linked
| but an update for a library doesn't require redownloading large
| executables for every package that depends on the library?
| wizzwizz4 wrote:
| Yup: distribute source code. (Step 0 is making compilation not
| require the world's most powerful gaming rig.)
| mjbrusso wrote:
| What about distribute LLVM bitcode?
| drdec wrote:
| I think a distribution similar to Gentoo, where the software is
| compiled at install time, could do something like this.
|
| (I don't know enough about Gentoo to know if it already does.)
| brandmeyer wrote:
| The single number one gripe about Gentoo was how long it took
| to build packages, closely coupled to the difficulty in
| keeping a system up-to-date.
|
| In response, Gentoo developed methods to ship binaries for
| the common cases.
| klyrs wrote:
| Absolutely possible, with source distribution. It keeps
| downloads small, and upgrades (notionally) transparent. The
| downsides are that it's computationally expensive on the user
| end, in addition to storage bloat. Does the latest firefox
| depend on the latest llvm? Depending on how awful your hardware
| is, you might wait a day for those two packages to build.
| rodgerd wrote:
| > As Serge Guelton said in a followup email, one of the reasons
| that Linux distros use dynamic linking is to reduce the disk
| usage and bandwidth requirement for updates.
|
| There are a number of ways to solve the latter problem, though:
| dnf, for example, includes the ability to download only the
| diff between your installed RPM and the version you're
| upgrading to.
| est31 wrote:
| I think it would be possible, yes. Google has even developed a
| binary diffing system.
|
| https://www.chromium.org/developers/design-documents/softwar...
|
| https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=538339
|
| It would still not be pleasant for disk space. Ideally I want
| my OS as well as the VMs to be small and require as little ram
| as possible. I don't want to update my entire OS just because
| glibc had a critical bug.
| [deleted]
| flowerlad wrote:
| The main supposed benefit is not disk or memory economy, but
| central patching.
|
| The theory is as follows: If a security flaw (or other serious
| bug) is found, then you can fix it centrally if you use shared
| libraries, as opposed to finding every application that uses the
| library and updating each separately.
|
| In practice this doesn't work because each application has to be
| tested against the new version of the library.
| ufo wrote:
| But is this still as important if the executable in question is
| part of the Linux distribution? In theory, Fedora knows know
| that Clang depends on LLVM and could automatically rebuild it
| if there was a change in LLVM.
| aidenn0 wrote:
| You also get bugs introduced into a program by changes to
| shared libraries. I've even seen a vulnerability introduced
| when glibc was upgraded (glibc changed the direction of memory,
| and the program was using memcpy on overlapping memory).
| ttt0 wrote:
| Can't we just link statically but still maintain the list of
| dependencies as if everything was linked dynamically?
| R0b0t1 wrote:
| Problem if you're not set up to build everything yourself.
| ttt0 wrote:
| Is it? Just update everything regularly.
|
| Actually now that I think about it, building by yourself
| might put you at a disadvantage here, as you'd have to
| remember to rebuild everything. I'm kinda lazy when it
| comes to updates so not sure if I like the idea anymore
| with having to rebuild all the software I built myself lol,
| but it probably could be solved by also shipping .so
| versions of libraries.
| carlhjerpe wrote:
| Nix will solve this for you in a breeze.
| johnny22 wrote:
| sure, some packages in some of the popular distros are indeed
| like that. If the package is important enough (like firefox)
| and the required dependencies are a bit out of step with
| what's currently used by the rest of the distribution you
| will sometimes see that for at least some of the
| dependencies.
|
| Most distros dislike doing so, and scale it back as soon as
| it becomes technically possible.
| ttt0 wrote:
| But they dislike just packages requiring different versions
| of libraries, right? My point is to do literally everything
| as it is right now, but simply link it statically. You can
| still have the same version of a library across all
| packages, you just build a static library instead of a
| dynamic one.
| danShumway wrote:
| This is odd to me, because surely they have to maintain
| that list anyway so they know which dependent packages need
| to be tested before releasing bugfixes?
|
| Or is that step just not happening?
|
| I just feel like, one of the big points of a package
| manager is that I can look up "what is program X's
| dependencies, and what packages rely on it?"
| agumonkey wrote:
| I don't know the term for this pattern.
|
| - shared libs
|
| - name indirection (dns domains, css classes)
| tannhaeuser wrote:
| Late binding?
| calvinmorrison wrote:
| overloading?
| Lammy wrote:
| Loose coupling? https://en.wikipedia.org/wiki/Loose_coupling
| RcouF1uZ4gsC wrote:
| > as opposed to finding every application that uses the library
| and updating each separately.
|
| Now with Docker, it becomes, find all the Docker containers
| that have that shared library, which may even be a bigger pain.
|
| Not all uses, but it seems as if a significant use of Docker is
| just a a very complicated replacement for a single statically
| linked binary.
| fxtentacle wrote:
| Fully agree. I have a collection of tensorflow docker
| containers because installing multiple versions in parallel
| won't work.
| danShumway wrote:
| > then you can fix it centrally if you use shared libraries, as
| opposed to finding every application that uses the library and
| updating each separately.
|
| This comes up a lot, but how often do you end up in a scenario
| where there's a critical security hole and you _can't_ patch it
| because one program somewhere is incompatible with the new
| version? Maybe even a program that isn't security critical.
|
| Plus what you mentioned about testing. If you update each
| application individually, you can do it piecemeal. You can test
| your critical security software and roll out a new version
| immediately, and then you can test the less critical software
| second. In some systems you can also do partial testing, and
| then do full testing later because you're confident that you
| have the ability to roll back just one or two pieces of
| software if they break.
|
| It's the same amount of work, but you don't have to wait for
| literally the entire system to be stable before you start
| rolling out patches.
|
| I don't think it's 100% a bad idea to use shared libraries, I
| do think there are some instances where it does make sense, and
| centralized interfaces/dependencies have _some_ advantages,
| particularly for very core, obviously shared, extremely stable
| interfaces like Linus is talking about here. But it 's not like
| these are strict advantages and in many instances I suspect
| that central fixes can actually be downsides. You don't want to
| be waiting on your entire system to get tested before you roll
| out a security fix for a dependency in your web server.
| fao_ wrote:
| > This comes up a lot, but how often do you end up in a
| scenario where there's a critical security hole and you
| _can't_ patch it because one program somewhere is
| incompatible with the new version? Maybe even a program that
| isn't security critical?
|
| Talking about user usecases, every time I play a game on
| Steam. At the very least there are GNU TLS versioning
| problems. That's why steam packages it's own library system
| containing multiple versions of the same library -- thus
| rendering all of the benefits of shared libraries completely
| null.
|
| One day, game developers will package static binaries, and
| the compiler will be able to rip out all of the functions
| that they don't use, and I won't have to have 5 - 20 copies
| of the same library on my system just sitting around -- worse
| if you have multiple steam installs in the same /home
| partition because you're a distro hopper.
|
| One day...
| ronsor wrote:
| > the compiler will be able to rip out all of the functions
| that they don't use
|
| Link-time optimization is already a thing. It's sad it
| doesn't seem to be used more often.
| dsr_ wrote:
| In practice this works really well as long as the change is to
| eliminate an unwanted side effect rather than to change a
| previously documented behavior.
|
| But it doesn't really matter. What matters is that whatever
| system is in use needs to have a control system that can
| quickly and reliably tell you everything that uses a vulnerable
| library version, and can then apply the fixes where available
| and remind you of the deficiencies.
|
| That metadata and dependency checking can be handled in any
| number of ways, but if it's not done, you are guaranteed not to
| know what's going on.
|
| If a library is used inside a unikernel, inside a container,
| inside a virtual machine, inside venvs or stows or pex's or
| bundles, the person responsible for runtime operations needs to
| be able to ask what is using this library, and what version,
| and how can it be patched. Getting an incomplete answer is bad.
| Quekid5 wrote:
| Absolutely correct.
|
| The whole "shared libraries are better for security" idea is
| basically stuck in a world where we don't really have proper
| dependency management and everyone just depends on "version
| whatever".
|
| This is interestingly also holding back a lot of ABI breaking
| changes in C++ because people are afraid it'll break the
| world... which, tbf, it will in a shared library world. If
| dependencies are managed properly... not so much.
|
| I wish distros could just go back to managing _applications_
| rather than worrying so much about libraries.
|
| EDIT: There _are_ advantages with deduplicating the memory
| for identical libraries, etc., but I don 't see that as a
| major concern in today's world unless you're working with
| seriously memory-constrained systems. Even just checksumming
| memory pages and deduplicating that way might be good enough.
| adrianmonk wrote:
| I strongly agree that the reporting and visibility you're
| talking about are important.
|
| But there's one other advantage of the shared library thing,
| which is that when you need to react fast (to a critical
| security vulnerability, for example), it is possible to do it
| without coordinating with N number of project/package
| maintainers and getting them all to rebuild.
|
| You still do want to coordinate (at least for testing
| purposes), but maybe in emergencies it's more important to
| get a fix in place ASAP.
| variaga wrote:
| >In practice this works really well as long as the change is
| to eliminate an unwanted side effect rather than to change a
| previously documented behavior
|
| ...and then you deploy and discover that _somebody_ was
| depending on that "unwanted" side effect.
| hodgesrm wrote:
| Another issue with security patches is that specific CVEs are
| not of equal severity for all applications. This likewise
| changes the cost/benefit ratio.
| eikenberry wrote:
| Works the exact same way without shared libraries, just at the
| source level instead of the binary level. "Finding every
| application" is simple. The problem is the compile time and
| added network load. Both are solvable issues, via fast
| compiling and binary patching, but the motivation isn't there
| as shared libraries do an OK job of mitigating them.
| jakear wrote:
| For a rough measure of how many packages will break if you
| perform a minor update to a dependency, try searching
| "Dependabot compatibility score" on GitHub (or just
| author:dependabot I suppose), each PR has a badge with how many
| public repo's CI flows were successful after attempting to
| apply the patch. By my eye it seems to be generally anywhere
| from 90-100%. So the question is would you be happy if every
| time a security vulnerability came out approximately 1 in 20 of
| your packages broke but you were "instantly" protected, or
| would you rather wait for the maintainers to get around to
| testing out and shipping the patch themselves. Given the vast
| majority of these vulnerabilities only apply in specific
| circumstances and the vast majority of my packages are only
| used in narrow circumstances with no external input, I'd take
| stability over security.
| viccuad wrote:
| > In practice this doesn't work because each application has to
| be tested against the new version of the library.
|
| In practice it works: see autopkgtest and
| https://ci.debian.net, which reruns the tests for each reverse
| dependency of a package, every time a library gets updated.
|
| I know for a fact that other commercial, corporate-backed
| distributions are far away from that sophistication, though.
| Aissen wrote:
| If you can retest everything, you can rebuild everything. But
| that doesn't help the network bandwidth issue. Debian doesn't
| have incremental binary diff packages anyway (delta debs) in
| the default repos anyway, so there's room for improvement
| there.
| toast0 wrote:
| So if the CVE says that you need to update library L, and
| program A uses the thing that's broken and it's ok to update,
| but program B doesn't use the broken thing, but the L upgrade
| breaks it, CI would let you know that you're screwed before
| you upgrade... but you're still screwed.
|
| It's worse if the program that needs the update also is
| broken by the update, of course.
| Conan_Kudo wrote:
| > _I know for a fact that other commercial, corporate-backed
| distributions are far away from that sophistication, though._
|
| No, they're not. Both Fedora/RHEL and openSUSE/SLE do similar
| checks. Every update submission goes through these kinds of
| checks.
|
| Heck, the guy who _created_ autopkgtest works at Red Hat and
| helped design the testing system used in Fedora and RHEL now.
| openSUSE has been doing dependency checks and tests for
| almost a decade now, with the combination of the openSUSE
| Build Service and OpenQA.
| ltbarcly3 wrote:
| Central patching had greater utility before operating systems
| had centralized build/packaging systems connected via the
| internet. When you had to manually get a patch and install it
| via a tape archive or floppy drive, or manually copy files on a
| lan, only having to update one package to patch a libc issue
| was a big help. Today it would require a single command either
| way, and so doesn't really have any significant benefit.
___________________________________________________________________
(page generated 2021-05-01 23:00 UTC)