[HN Gopher] "Static Linking Considered Harmful" Considered Harmful
___________________________________________________________________
"Static Linking Considered Harmful" Considered Harmful
Author : yagizdegirmenci
Score : 134 points
Date : 2021-10-03 11:15 UTC (11 hours ago)
(HTM) web link (gavinhoward.com)
(TXT) w3m dump (gavinhoward.com)
| davidwf wrote:
| This is very situational, but I have recently been part of a
| project that does a lot of C server-side development and we have
| found that static linking our non-glibc dependencies has really
| improved our developer experience. Using ceedling's dependency
| plugin[1] and producing a single "statically" linked library has
| made our C development much closer to using a language with a
| more modern package manager. Don't get me wrong, if I was trying
| to distribute binaries to machines I didn't control I'd
| definitely be willing to invest in the Linux packaging "fun", but
| for a server-side application it's been a good choice for our
| team overall.
|
| [1]
| https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...
| PaulDavisThe1st wrote:
| Yes and no. In Ardour, which has a substantial dependency tree
| (80+ libraries), if we statically link, the edit/compile/debug
| cycle becomes incredibly bogged down by static linkage. It
| takes a long time (even with lld) to complete the link step. If
| we use shared libraries/dynamic linkage, the edit/compile/debug
| cycle is nice and fast, but application startup is a bit slower
| (not by as much as static linkage is slower, however).
|
| For users, it would be better if the application was statically
| linked, at least in terms of startup cost. But because
| developers do the edit/compile/debug cycle much, much more
| often than users start the application, we opt for dynamic
| linkage.
| xyzzyz wrote:
| You can dynamically link during development, and then ship
| statically linked binary (with LTO etc). That's what Chrome
| does, for example.
| PaulDavisThe1st wrote:
| It's not that simple for us (we have tried this).
| sharikous wrote:
| Correct me if I am wrong but OpenBSD introduced the constraint
| that syscalls must come from a specific section of memory, that
| assigned to their libc.
|
| In a case like that static linking is not only unstable, as in
| Windows or macOS, but also impossible.
|
| And since that has security benefits we will probably see it
| being migrated to other platforms in the near future.
| ris wrote:
| I eagerly await some concrete implementations of these new ideas.
|
| In the meantime I'm waiting for the first _big_ vulnerability in
| a widely used golang or rust library to see if downstream
| projects pinning it _also_ release CVEs as they should (which
| would probably DDoS the CVE system from the resulting avalanche)
| _or_ they quietly bump it and move on (in which case their users
| won 't get the nudge to upgrade). This is where someone says
| "well you should just always be running the latest version of
| everything", which is of course infeasible on a real life system
| with thousands of installed packages.
|
| And it's not that I don't completely sympathize or understand the
| advantages of static linking from a developer's point of view -
| you don't have to sell it to me there.
| ghoward wrote:
| Author here.
|
| > I eagerly await some concrete implementations of these new
| ideas.
|
| I'm working on them now! They are not public because of a
| licensing issue where I need to consult a lawyer first (thanks,
| GitHub Copilot), but they will be public as soon as that's
| sorted out.
| Zababa wrote:
| > They are not public because of a licensing issue where I
| need to consult a lawyer first (thanks, GitHub Copilot), but
| they will be public as soon as that's sorted out.
|
| Do you mean that you built them using Copilot, or that you
| don't want Copilot to use them? Or something else entirely?
| ghoward wrote:
| I don't want Copilot to use them. Sorry; should have said
| that.
| Zababa wrote:
| No worries, it's nice to know that someone is working on
| that with the help of actual lawyers!
| DaiPlusPlus wrote:
| The design of the Go build and packages system kinda renders
| that point moot because the expectation is that provided you
| follow Go's guidelines then a Go application should _always_ be
| easily recompilable, and all external dependencies should
| always be distributed in source-form, not as binaries (e.g.
| https://github.com/golang/go/issues/2775 ).
| Macha wrote:
| Right, but if every application did distribute as source and
| could just be recompiled, there would be fewer complaints
| about the difficulties distributing Linux binaries
| Zababa wrote:
| > provided you follow Go's guidelines then a Go application
| should always be easily recompilable, and all external
| dependencies should always be distributed in source-form, not
| as binaries
|
| So it's a human solution, and not a technical one, which
| means that it can and will fail.
| DaiPlusPlus wrote:
| That's like saying condoms are ineffective because you
| included people too lazy to put them on in the first place
| in your calculations.
|
| Provided you (and your org) use Go as Google intends then
| that isn't an issue.
| Zababa wrote:
| > That's like saying condoms are ineffective because you
| included people too lazy to put them on in the first
| place in your calculations.
|
| I think it's closer to say that the existance of condoms
| won't make AIDS and unwanted pregnancies disappear, even
| if they were 100% effective. Which, if you're trying to
| eliminate completly AIDS, is a fair claim to make.
|
| My point is that you shouldn't assume that everyone is
| going to follow the Go guidelines, because some people
| won't, and you don't want your security to rely on that.
| Wowfunhappy wrote:
| > This is where someone says "well you should just always be
| running the latest version of everything", which is of course
| infeasible on a real life system with thousands of installed
| packages.
|
| Is updating a library with thousands of dependents, without
| individually testing those dependents, any more feasible?
| CyberShadow wrote:
| You would have to do that anyway. With static linking, you
| also have to rebuild those dependents and make sure their
| users update them as well.
| Wowfunhappy wrote:
| I would rather the developers rebuild the dependents, test
| them, make any necessary fixes, and send me the updated
| (presumably static) binaries. As opposed to me, as an
| individual user with no familiarity with the code base,
| just switching out the pieces and preying nothing goes
| wrong.
| ris wrote:
| And if the upstream developers are unresponsive, or only
| provide the bump with the latest bleeding-edge version
| which you can't upgrade to yet?
| Wowfunhappy wrote:
| On a system where security is critical, I would probably
| need to re-evaluate my use of that software.
|
| It doesn't necessarily need to be the upstream developer,
| it could be some other organization analogous to a distro
| maintainer. What matters is that _someone_ who is
| familiar with the software and code base has actually
| tested the update in a purposeful way.
| AussieWog93 wrote:
| As a small developer who handles my own support, I agree
| with this sentiment completely!!
| xg15 wrote:
| You'll have the opposite problem though, that you have to
| trust every single developer to properly manage their
| dependencies and publish timely patches if there is a
| vulnerability.
|
| > _As opposed to me, as an individual user with no
| familiarity with the code base, just switching out the
| pieces and preying nothing goes wrong._
|
| Wouldn't this be the task of your distro's mainteners,
| not you? (Who _will_ have familiarity with all codebases
| as well as options to contact the original developers if
| there is an unsolvable issue)
| ris wrote:
| Well, it's what traditional distributions do every day.
|
| In NixOS, we try to work the test suites into the build
| process and rebuilds happen when any dependencies change, so
| I'd like to think we get (almost) the best of both worlds.
| Wowfunhappy wrote:
| > Well, it's what traditional distributions do every day.
|
| Which coincidentally is why I'm not using Linux right now.
| :) A move towards static linking in the Linux world would
| go a long way towards changing that.
|
| (Everything I've read about NixOS seems great, thanks for
| helping with that--it's just that it also has a large
| barrier to entry.)
| ris wrote:
| (I'm merely a lowly maintainer in NixOS, shoulders of
| giants etc.)
| candiddevmike wrote:
| That's the choice you're making everytime you add another
| dependency--you are now bound to that repo and all of it's
| upstreams. You need to follow along send stay on top of
| Caves, API deprecations, etc. I don't think people weigh this
| cost properly when doing a build vs integrate decision. Turns
| out dependency management is a lot like the "when you sleep
| with someone you're sleeping with everyone they've ever slept
| with" ideology.
| dcsommer wrote:
| Are you sure about the "CVE explosion"? From the CNA counting
| rules
| https://cve.mitre.org/cve/cna/rules.html#section_7_assignmen...
| : 7.2.4 If multiple products are affected by
| the same independently fixable vulnerability, then the CNA:
| a. MUST NOT assign more than one CVE ID if the products are
| affected, because they share the vulnerable code. The assigned
| CVE ID will be shared by the affected products.
| ris wrote:
| I'm not sure what would actually happen in reality, whether a
| single CVE would get endless addenda listing the packages
| affected by an upstream vulnerability. I certainly see plenty
| of new CVEs go past which are of the form "xyz had a vendored
| version of abc, which was vulnerable to ..."
| Zababa wrote:
| > This is where someone says "well you should just always be
| running the latest version of everything", which is of course
| infeasible on a real life system with thousands of installed
| packages.
|
| Maybe the problem is have a system with thousands of installed
| packages? If every application is in its own "space", then a
| security failure won't affect the rest. If you want the best
| security, you're going to have to do pretty big tradeoffs. If
| you're not ready to do them, you're going to have to live with
| an insecure system. I don't think there is a way around this.
| kbenson wrote:
| Part of the problem is not just getting all the different
| applications and libraries up to date, but knowing which of
| them need to be updated.
|
| For a system that widely uses dynamic libs, that's fairly
| easy to do. Check the version of the installed library, along
| with any patches it might have, and you have a good idea of
| whether it needs to be updated.
|
| Now, imagine there's a zlib exploit. That's a _lot_ of things
| in the system that need to be updated. It 's so ubiquitous
| that even in a system that's almost entirely built with
| dynamic libs, there are a few that use it as a static lib,
| and you'll have to make sure those are updated too (go look
| at the last zlib CVE and the major distro updates to it and
| you'll see all those packages, sometimes a couple days later
| as they discover them too).
|
| A world where these are all statically compiled (or bundled
| separately in containers) is much more complex. It's not
| impossible, but you don't naturally get some of the work done
| for you. To some degree we're already going this way with
| containers though.
| Zababa wrote:
| > Part of the problem is not just getting all the different
| applications and libraries up to date, but knowing which of
| them need to be updated.
|
| In which was? Is it because it's hard to know which version
| of zlib was used in X application, or is it because there
| is no centralized information about this? Maybe that's an
| opportunity right there: build a graph of dependencies of a
| system, and alert when something has been compromized and
| which are the consequences.
| kbenson wrote:
| To some degree most distro's have this graph and it's
| somewhat queryable (rpm has to agilities for this, if not
| easily consumed). The difference is in how accurate and
| queryable it is, and how easy it is to be fairly sure
| everything is up to date.
|
| It's not a show stopping problem, but it is something
| that should be considered and thought closely about. This
| is a security adjacent problem, and changes in it can
| have consequences that are larger than "I'm running a
| library that doesn't have the feature I want".
| xg15 wrote:
| > _build a graph of dependencies of a system_
|
| You mean in the sense that every application that comes
| with statically linked libraries should also come with
| some metadata indicating which libraries/versions are in
| there?
|
| This would work, but then you have the problem metadata
| always has: It can get out of sync with the actual thing,
| developers can forget to properly maintain it (or even to
| provide it at all) etc etc.
| Zababa wrote:
| You're right, I didn't answer the question of "who should
| maintain that stuff?". I was imagining something like
| libraries in JS or Rust, where people pin the
| dependencies of their libraries, so with this, you can
| build a graph of dependencies. I guess I have a very
| "github" centric vision of things and that most software
| isn't built this way, which would complicate things.
| bregma wrote:
| It would certainly make the embedded industry more secure if it
| was always running the latest version of everything. Or at
| least the ISPs, because imagine the bandwidth they're be able
| to charge for as everything from your washing machine to your
| watch to your lightbulbs to your car downloads new images
| hourly.
| ris wrote:
| Just imagining the joy of broken firmware updates being
| pushed out to all my lightbulbs...
| mst wrote:
| I believe when heartbleed happened quite a bunch of go progams
| were yoinking in openssl because the go ssl tooling was still
| relatively young.
|
| I have no idea what actually happened there, I just recall some
| moderately annoyed sysadmins figuring out how to rebuild go
| applications by developers who'd since left the company.
|
| (this is not a shot at go, this is merely a "there might
| already be -some- data out there" wrt your question)
| alkonaut wrote:
| The difference between dynamically and statically linking
| doesn't change the patching story so long as applications ship
| with all their libs and share nothing.
|
| The only difference is that a statically linked app is a large
| binary blob in a directory while a dynamically linked app is
| multiple smaller blobs in a directory.
|
| Whether or not statically or dynamically linked apps are the
| future, the idea of system wide shared libraries seems like
| it's going away.
| xg15 wrote:
| > _The difference between dynamically and statically linking
| doesn't change the patching story_
|
| I think there are some edge cases where the mainteners will
| always be caught between a rock and a hard place and the only
| way to move forward is to involve the application developers
| - at least as long as library compatibility can't be defined
| more rigidly.
|
| As a dev, you'd like to pin the versions of your dependencies
| as tightly as possible, so you know the library will behave
| in production exactly as it did in your tests.
|
| As a maintener, you'd like applications to specify their
| dependency versions as loosely as possible, so you can
| transparently update a dependency in case of critical
| patches.
|
| This means, there can always occur a case where version x of
| a library has a critical vulnerability but version x+1 breaks
| the assumptions that the software is making.
|
| No matter if the library is linked statically or dynamically,
| this situation can't be solved by the maintener alone.
|
| Of course the GP is also correct in a sense: If applications
| bundle libraries or set tight version restrictions, it's now
| the responsibility of the _developers_ to track critical
| vulnerabilities in all of their dependencies and integrate
| the patches as quickly as possible.
|
| The problem with bundled libraries is that now we have to
| trust every single developer to get this right.
| IshKebab wrote:
| Exactly. The issue is not static vs dynamic; it's bundled vs
| unbundled.
|
| You could probably even do an "unbundled" statically linked
| system, e.g. imagine Debian but everything is statically
| linked. Doesn't matter that everything is statically linked -
| if there's a vulnerability in libpng you can still easily
| update dependencies. Just uses more network / disk space.
| dralley wrote:
| > Doesn't matter that everything is statically linked - if
| there's a vulnerability in libpng you can still easily
| update dependencies. Just uses more network / disk space.
|
| It's not that simple
|
| * Someone, somewhere - probably distro volunteers - have to
| manage this process of rebuilding dozens, hundreds, or
| thousands of packages, and knowing which packages need to
| be updated to begin with. The amount of work required from
| distros (again, mostly volunteers) would be an order of
| magnitude greater even if the process was automated.
|
| * Likewise, the amount of resources needed to rebuild all
| of those packages, and distribute them, would be orders of
| magnitude greater.
|
| * The amount of extra work (both for humans and computers)
| required would delay the rollout of patches.
|
| * And in the real world, a lot of users don't have
| limitless network bandwidth and disk space.
|
| * And auditing systems for security vulnerabilities would
| be geometrically more difficult. Now you need to track a
| recursive bill of materials for every package on the
| system.
|
| If there's a security bug in libc or something along those
| lines - which are fairly common- distros would need to
| rebuild practically every package in the OS. It's all well
| and good to talk about how it would be no big deal but I
| think if it was ever actually implemented no user would
| accept that trade-off.
| Wowfunhappy wrote:
| I agree, but if you're bundling the dependencies why not
| just statically link them too? I've never understood the
| way macOS does it (dynamic linking, but app bundles ship
| with the libraries they need included).
| alkonaut wrote:
| This is also how windows apps always worked. It's good
| for deployment because an update might not touch more
| than a few files.
|
| Also not all language ecosystems support static linking
| at all, e.g .NET
| dragonwriter wrote:
| > Also not all language ecosystems support static linking
| at all, e.g .NET
|
| The "language ecosystem" of .NET supports static linking
| via .NET Native, doesn't it?
| mastax wrote:
| .NET native is a dead end technology that only supported
| old UWP apps.
|
| There's the new single file builds but those just
| concatenate the dynamic libs together into one file.
|
| There's also the new .NET linker for trimmed builds, but
| that requires special annotations from many libs to work.
| xg15 wrote:
| Not a mac user, but one upside I can see is that this
| forces apps to at least declare which libraries they are
| using. If everything is statically linked, it might be
| hard to figure out where on the system a vulnerable
| library is being used at all.
| ziml77 wrote:
| In some cases it's required such as if you're using a
| closed-source library that doesn't provide a static
| version or if you're calling LGPL code from non-GPL-
| compatible code.
| bbarnett wrote:
| shared ram too. it adds up.
| mpyne wrote:
| It does, but we're already moving to a world where deployed
| apps are one of the very few things running in a container,
| which is itself running somewhere in a VM, so there's less
| sharing to achieve.
| ithkuil wrote:
| Yeah. In that scenario the distinction between dynamic
| and static linking is moot. In both cases you need to
| update the container image and when you do you only fixed
| that container image and you still need to update all the
| other images
| mercurialfck wrote:
| There's absolutely no need for just one-ring-to-rule-them-all
| copy of libz.so. Shared libraries can be versioned and
| garbage collected like how habitat (hab) does it: in separate
| directories.
| forrestthewoods wrote:
| I care far more about about bundled vs unbundled than dynamic vs
| static.
|
| I am passionately in-favor of bundled. Unbundled distribution is
| what has led to software being so outrageously impossible to
| execute that the only sane path to distribution is to create a
| bundled docker image. https://xkcd.com/1987/
| flohofwoe wrote:
| Unfortunately not even Linux allows full static linking (as far
| as I'm aware at least) as soon as libraries like OpenGL are
| involved.
|
| Dynamic linking with core system libraries which are guaranteed
| to exist(!) is fine though, for everything else dynamic linking
| mostly has downsides.
|
| Also: it's _only_ Linux where this is an issue, because it 's
| nearly impossible to build small self-contained executables which
| run on another Linux machine, thanks to glibc's versioning mess
| (at least there's MUSL as an alternative to create portable
| command line tools).
| Macha wrote:
| Didn't Go have to throw in the towel on not dynamically linking
| libc on macOS as there's no guarantee of syscall stability at
| all?
| patrakov wrote:
| I don't know about macOS, but they also did this on OpenBSD.
| The reason is a recent security addition in OpenBSD, where
| they made it illegal (as in: violate it, and the process will
| be killed) to issue syscalls to the kernel from anything
| other than libc.
| Someone wrote:
| https://lwn.net/Articles/806776/ discusses that feature and
| says "Switching Go to use the libc wrappers (as is already
| done on Solaris and macOS)", so yes, they did that on
| macOS, too.
|
| Reading https://golang.org/doc/go1.11#runtime, that's
| fairly recent (since go 1.11)
| flohofwoe wrote:
| Don't know, I only know that I can build a macOS command line
| tool on one machine, and run it on a different machine
| without problems, while on Linux it's very easy to run into
| the glibc version compatibility problem.
| onei wrote:
| I always understood the glibc versioning problem to be
| solved by just building on the oldest common ancestor. If
| you have rhel5 through 8, you build on rhel5 and move on.
| If you depend on more than glibc... Well rhel5 is less fun.
| ghoward wrote:
| Author here.
|
| Unfortunately, building on the oldest common ancestor
| will still not save you from ABI breaks. An example is
| [1]. If the size of `intmax_t` changes between that
| oldest ancestor and the machines you are running on, get
| ready for pain.
|
| [1]: https://thephd.dev/intmax_t-hell-c++-c
| ptsneves wrote:
| Indeed, the only difference is that you might be using
| less optimal system calls, but there is no reason to be
| blocked and not target ancient kernels. I linked to 2.6
| this year and everything ran fine.
| p_l wrote:
| At least one can statically link the application except for
| forcibly-dynamic libs like libGL (also, apparently using
| dlopen() is often _faster_ than naive dynamic linking...)
| tyingq wrote:
| It's also tricky if you need to call gethostbyname() and expect
| system local behavior.
| edflsafoiewq wrote:
| Well yeah, the GL calls are _actually different_ on different
| machines right? Isn 't that the absolute first consideration
| when deciding if you should dynamically link?
| flohofwoe wrote:
| I'd rather expect that the 'user-facing' GL implementation
| (GL2.x, GL3.x, ...) is identical and can be statically
| linked, and that only the graphics driver layer differs
| between machines (which would be similar to an executable
| which links the C library statically, which in turn talks to
| the operating system through syscalls).
|
| But to be fair, dynamically linked GL wouldn't be a problem,
| if this wouldn't also pull in a dynamically linked glibc (it
| always comes back to glibc unfortunately).
| cesarb wrote:
| > I'd rather expect that the 'user-facing' GL
| implementation (GL2.x, GL3.x, ...) is identical and can be
| statically linked, and that only the graphics driver layer
| differs between machines (which would be similar to an
| executable which links the C library statically, which in
| turn talks to the operating system through syscalls).
|
| The problem is that the "graphics driver layer" (except a
| small amount of security-sensitive pieces) runs in user
| space, in the same process which is doing the
| OpenGL/Vulkan/Metal/etc calls. The way modern graphics
| hardware works is that the application writes a very
| complex and hardware-dependent set of command buffers,
| which are then submitted to the hardware, and only this
| final submission goes through the kernel (for security
| reasons). If the application had to call the kernel for
| each and every step of the creation of the command buffers,
| it would kill the performance.
| creshal wrote:
| Isn't this the point of libglvnd?
| simias wrote:
| That's normally what kernel abstractions are for, but
| unfortunately for various historical reasons graphics API
| often bypass the kernel in part or in full. There's also the
| problem that graphics are often very performance-sensitive so
| costly abstractions may not be desirable.
| PaulDavisThe1st wrote:
| > I don't think Ulrich Drepper could have foreseen all of the
| problems with DLL Hell (though there were signs on Windows),
|
| Drepper's article is from 2006. "DLL Hell" was beyond fully
| understood at that point, certainly by Drepper and large numbers
| of other well informed programmers.
| ghoward wrote:
| Author here.
|
| In everything in the post, I attempted to give Drepper the
| benefit of the doubt, though there are cases where I don't want
| to since he sounds (to me) paternalistic in the way Apple and
| Microsoft do.
|
| So perhaps he understood, but I didn't want to assume that,
| especially since it might inflame the discussion, which is
| exactly what I was trying to avoid doing ( _again_ ).
| PaulDavisThe1st wrote:
| DLL Hell was well understood in the mid-90s. I think you need
| to give him more benefit of the doubt on this point.
| torstenvl wrote:
| If someone glosses over an issue, it's either innocent
| (didn't know about it, didn't understand it) or deceptive
| (intentionally ignored it). Giving Drepper the benefit of
| the doubt here means assuming he didn't know or understand
| the problem he failed to adequately address.
| PaulDavisThe1st wrote:
| Here is most of the relevant text from TFA:
|
| > Ulrich Drepper claims, rightly, that you only need to
| update a shared library to apply security fixes to all
| executables that rely on that library.
|
| > This is a good thing! It is also a good vision.
|
| > Unfortunately, it is only a vision because it's not the
| whole story.
|
| > As of yet, there is no way to automatically determine
| if the ABI or API of a shared library changed between
| updates. This is called DLL Hell.
|
| The final two words are a link to the wikipedia page on
| "DLL Hell". The final sentence of the intro section there
| states:
|
| > DLL Hell is the Windows ecosystem-specific form of the
| general concept dependency hell.
|
| That is, TFA defines "DLL Hell" using a wikipedia page
| that explicitly states that it is a Windows-specific
| version of a more general problem.
|
| Drepper's article from 2006 fully tackles the general
| problem, albeit without (as the TFA puts it): "a way to
| automatically determine if the ABI or API of a shared
| library changed between updates." Drepper's primary
| suggestion there is a naming/versioning scheme which
| describes specifically whether the ABI/API has changed in
| ways that matter for shared linkage. It's not automatic -
| that part is true. But is is a solution, widely used in
| Linux and *nix more broadly.
|
| Did Drepper address the Windows specific parts of "DLL
| Hell". He did not, but that's because "DLL Hell" is
| Windows specific (as other comments here fully clarify),
| and he was not writing about how to fix the full scope of
| that particular nightmare. He likely understood "DLL
| Hell" as well as anyone, nevertheless.
| torstenvl wrote:
| I think you're missing the point. When the only
| alternatives are intellectual dishonesty or ignorance,
| assuming ignorance is the more generous option.
| Therefore, assuming he _didn 't_ know _is_ giving him the
| benefit of the doubt.
|
| The reasoning behind your differing assumption is beside
| the point, _even though your reasoning is more likely to
| be true_.
| PaulDavisThe1st wrote:
| I think you're missing my point.
|
| The TFA's claim is that Drepper didn't get all the issues
| with DLL Hell. My claim is that Drepper did get all of
| the issues, and addressed those that were not Windows-
| specific (e.g. per-process COM servers etc.)
| torstenvl wrote:
| No. I am not addressing your point because your point is
| not logically related to the topic under discussion.
|
| Specifically, your post said:
|
| > _DLL Hell was well understood in the mid-90s. I think
| you need to give him more benefit of the doubt on this
| point._
|
| The substantive question about whether he adequately
| addressed the issues with dynamic linking is beside the
| point. Feel free to agree or disagree with the above
| poster. Not relevant.
|
| The post I am responding to is about, _given ghoward 's
| position that X did not adequately address topic Y_,
| whether it is "giving X the benefit of the doubt" to
| further assume (a) X _did_ know about topic Y; or (b) X
| _did not_ know about topic Y.
|
| Because (a) implies dishonesty but (b) merely implies
| ignorance, "giving X the benefit of the doubt" means you
| should assume (b) until proven otherwise.
|
| If you want to respond to other topics, feel free to hit
| the reply button below someone else's post.
| PaulDavisThe1st wrote:
| The point under discussion is this line (and the
| surrounding text) from TFA:
|
| > I don't think Ulrich Drepper could have foreseen all of
| the problems with DLL Hell (though there were signs on
| Windows),
| cesarb wrote:
| > DLL Hell was well understood in the mid-90s.
|
| Also, there are some additional facets of DLL Hell which
| happened in the Windows of the mid-90s which are not as
| relevant to Linux. What made DLL Hell so bad there was that
| installing any random _program_ could _replace_ globally
| shared libraries, sometimes even with an _older_ version.
| That is, installing a game could make an unrelated
| productivity app stop working, because the game helpfully
| installed newer (or older!) versions of the shared
| libraries it needed, into the same globally shared
| directory in which nearly all DLLs lived. It got so bad
| (even DLLs which came with the operating system were being
| overwritten) that Microsoft IIRC initially introduced a
| system which detected when this happened, and _replaced the
| DLLs again_ with a clean copy it had stashed somewhere else
| (and later, made these files more directly protected).
|
| That's before considering the disaster that is in-process
| COM servers; presenting a standard "open file" dialog, or
| doing some printing, is enough to make Windows load
| arbitrary DLLs into your process (shell extensions and/or
| printer drivers), and these often don't have the highest
| code quality. And then there are some things which inject
| arbitrary DLLs into _every_ process...
|
| Compared to that, the dynamic linking issues in the Linux
| world are much more bearable. You don't see arbitrary
| programs overwriting the global copy of something like
| libgtk or openssl or zlib or libc, the only arbitrary
| dynamic libraries being loaded into a process are things
| like the NSS ones (usually from a small well behaved set)
| or plugins from the graphics libraries (also usually from a
| small well behaved set), and the only dynamic library being
| injected into every process is the vDSO from the kernel.
| ghoward wrote:
| I'm having a hard time understanding because if DLL Hell
| was well understood by 2006, it seems I should have given
| him _less_ benefit of the doubt, but you say I should give
| him _more_.
|
| Apologies.
| spacechild1 wrote:
| I would even claim DLL didn't really exist anymore in 2006. I
| remember it from the late 90s, but haven't encountered it in
| the last 15 years. Fortunately, people have learned their
| lessons.
| themulticaster wrote:
| > Second, ldd, the dynamic linker, can be manipulated into
| executing arbitrary code.
|
| ldd is _not_ the dynamic linker, it 's only a tool to debug the
| process of dynamic linking. The dynamic linker is ld-linux.so
| (exact name depends on glibc version, architecture, etc.)
|
| Also, I think the linked article [1] about the security of ldd is
| somewhat useless. The ldd(1) manpage [2] is very explicit about
| the security of ldd and tells you not to run ldd on untrusted
| executables:
|
| > [...] Thus, you should _never_ employ ldd on an untrusted
| executable, since this may result in the execution of arbitrary
| code.
|
| It's a little amusing how the linked blog post explains how to
| create a malicious executable that runs arbitrary code when
| inspected with ldd, noting that "I researched this subject
| thoroughly and found that it's almost completely undocumented. I
| have no idea how this could have gone unnoticed for such a long
| time." and concluding with "Never run ldd on unknown
| executables!" - all while the manpage literally mentions that
| precise known limitation in the third paragraph!
|
| To be fair, you could argue that this limitation is not widely
| known and people should be made aware of the risks of ldd, but on
| the other hand you can hardly criticize ldd when its manpage is
| that explicit about the issue.
|
| [1] https://catonmat.net/ldd-arbitrary-code-execution
|
| [2] https://www.man7.org/linux/man-pages/man1/ldd.1.html
| ratmice wrote:
| While you are technically correct, ldd is literally a bash
| script which runs ld-linux.so...
| throwaway09223 wrote:
| ld-linux.so's job is to link and run an executable. It's not
| a vulnerability that it runs executable it's been handed. If
| you run "ld-linux.so /bin/ls" you're just running "ls"
| There's no security issue with this behavior. It's an
| interpreter like any other - same as /usr/bin/python.
|
| The argument around ldd having a vulnerability is that it
| appears to be an introspective tool. It is not immediately
| obvious that it just executes "LD_TRACE_LOADED_OBJECTS=1 ld-
| linux.so /bin/ls" to print what the linker does during an
| actual, live linking operation.
|
| ldd documenting this fact to remind people seems reasonable
| to me. There are other tools like readelf which can inspect
| objects without executing their contents. Dynamic linking is,
| well, dynamic and it can depend on code executed at runtime
| -- so it is necessary to do so to get an accurate report of
| what will happen.
| ghoward wrote:
| Author here.
|
| Thank you for pointing out my mistake. I'll fix it.
| Hello71 wrote:
| ld isn't the dynamic linker either, it's the object linker.
| the name of the dynamic linker is ld.so or ld-linux.so. this
| is also documented in the ldd man page.
| eqvinox wrote:
| Fun fact: despite the library path and name, ld-linux.so is
| executable. Try running /lib64/ld-
| linux-x86-64.so.2
|
| On your favorite Linux box. (Adjust path if needed for
| different architecture / distro.)
| cataphract wrote:
| In fact, /lib64/ld-linux-x86-64.so.2 is set as the
| interpreter of the dynamic executables, so executing
| ./dyn_exec ends up being equivalent to /lib64/ld-
| linux-x86-64.so.2 ./dyn_exec. The explicit form is
| sometimes useful, for instace if you want to execute the
| file with an interpreter other than the one specified in
| the executable, or if you want to pass one of the
| parameters that ld.so accepts.
| rkeene2 wrote:
| It's also useful if you accidentally chmod -x chmod...
| eqvinox wrote:
| Secure - and more useful - replacement: lddtree.
|
| Part of "pax-utils" or similar package on some distros. It's
| actually a shell script wrapper around scanelf (also pax
| utils), which reads the appropriate bytes from ELF files
| without just executing the ELF file.
| rkeene2 wrote:
| I usually just do: objdump -x <binary> | grep NEEDED but this
| is a great tip !
| colonwqbang wrote:
| Often you need to find not only the soname but the actual
| .so file that is found and linked. This can be affected by
| environment variables like LD_PRELOAD and LD_LIBRARY_PATH.
| Unfortunately I don't know of a way to check this without
| ldd. If somebody knows of a tool that can do this without
| executing the binary, please share.
| [deleted]
| rwmj wrote:
| I read this interesting article recently about all the security
| problems that static linking, bundling and pinning cause for
| Linux distributions:
|
| https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...
| ghoward wrote:
| Author here, and I am also a Gentoo user on my daily driver.
|
| I read that article when it came out, and it felt disingenuous.
|
| mgorny helps maintain Gentoo, which of all distributions, does
| rebuild software as necessary.
|
| It would be a simple change to portage to rebuild all
| dependencies of a library when that library is updated,
| regardless of if the user uses static linking or not. In fact,
| there is a project to have a statically-linked musl-based
| Gentoo.
|
| So it's possible and (I would argue) would be an easy change.
| mgorny would not have to worry about much because he could, in
| essence, treat all libraries as dynamic libraries. The only
| pain to be felt would be by users who would see their build
| times increase (to rebuild dependents).
|
| As a user, I would _happily_ take that tradeoff. I already
| build my browser, which takes 5-8 hours; I 'll happily rebuild
| dependents of updated libraries.
| AshamedCaptain wrote:
| > I already build my browser, which takes 5-8 hours; I'll
| happily rebuild dependents of updated libraries
|
| This is ridiculous. I most definitely do NOT want to rebuild
| literally EVERY SINGLE PACKAGE on my system the moment they
| fix a typo in libz.
|
| With static linking literally every other Gentoo upgrade
| becomes a world rebuild. See Nix. It is just insane and the
| main reason I don't use NixOS (without a 3rd party binary
| cache/host) for anything other than very small systems.
| ghoward wrote:
| > This is ridiculous. I most definitely do NOT want to
| rebuild literally EVERY SINGLE PACKAGE on my system the
| moment they fix a typo in libz.
|
| I understand. You don't. I do.
|
| What my post is arguing for is not one or the other, but
| giving the user the choice.
|
| I think you would be okay with making the choice of _not_
| rebuilding everything, and I would make the opposite
| choice.
|
| I want to live in a world where we can both choose.
| AshamedCaptain wrote:
| But you can already rebuild the entire world on every
| package change if you want (for sadomasochistic reasons?
| it basically makes gentoo with say KDE _unusable_); and I
| fail to see if there would be any practical difference on
| whether you statically link, you dynamically link to a
| hash-based soname a la nix, or you dynamically link --
| you are just rebuilding everything on every library
| change, and you are already assuming ASLR is useless.
| stefantalpalaru wrote:
| > It would be a simple change to portage to rebuild all
| dependencies of a library when that library is updated
|
| No change needed. It can already be done with the special
| ":=" dependency slot: https://devmanual.gentoo.org/general-
| concepts/dependencies/#...
| mst wrote:
| > It would be a simple change to portage to rebuild all
| dependencies of a library when that library is updated
|
| I believe e.g. freebsd's portmaster takes a -r switch to mean
| "rebuild this and all its dependents" (I presume you meant
| revdeps here), which gets you quite a long way.
| Aqueous wrote:
| But should distributing and ensuring the security of software
| that users install on their own systems even be the
| responsibility of operating system maintainers?
| rwmj wrote:
| Isn't it obvious? Or do you suggest that end users should be
| responsible for checking and patching every package on their
| system? Given that Red Hat has a large department just to do
| that for RHEL, I can't imagine where end users would begin.
| Aqueous wrote:
| No, I'm suggesting that the authors of the applications are
| responsible for patching and distributing their own
| application like on every other major OS.
| eqvinox wrote:
| Yes.
|
| If I installed software through my OS package manager, where
| else would I get security updates from?!
| Aqueous wrote:
| Like on every other major OS the publisher of those
| applications. And I mean they should be distributing the
| applications as well. Not the OS publishers.
| api wrote:
| My personal view is that only standard core system libraries
| should be dynamically linked. Anything that is not totally
| ubiquitous should be static.
|
| That seems to be the Go and Rust default approach, or close to
| it. Link libc and such, but build in more obscure things.
|
| The idea of dynamically linking everything doesn't scale. It asks
| too much of distribution maintainers and makes them the choke
| point for every little upgrade.
| PaulDavisThe1st wrote:
| So, you link against a GUI toolkit that can be used to display
| images. Images in many different formats, including some that
| haven't been used in more than a decade and are unlikely to
| ever be encountered.
|
| Do you want the GUI toolkit linkage to automatically include
| every object module for every possible image file format in
| your static image, or do you want the GUI toolkit to use
| runtime dynamic linkage (explicit or implicit) to import only
| the modules required for the image formats actually
| encountered?
| TillE wrote:
| If I'm distributing binaries as a commercial product, I
| definitely want the former.
| PaulDavisThe1st wrote:
| There are many "commercial products" that run on Linux
| whose installation includes various *.so files that will be
| dynamically linked at run time. Are you saying this is a
| bad idea?
| TeMPOraL wrote:
| What about formats that don't exist yet? What about giving
| you the ability to supply your own encoder/decoder for some
| niche format you're using?
| josefx wrote:
| > That seems to be the Go and Rust default approach
|
| I think Go tried to go fully static but ran into problems as
| most mainstream OSes except for Linux do not provide a stable
| system call interface, so applications have to link at least
| one dynamic library. The Linux system call interface is stable,
| but expects a C style stack layout and that also bit the Go
| devs. multiple times. So you can probably go fully static if
| you are only targeting Linux.
| cesarb wrote:
| > I think Go tried to go fully static but ran into problems
| [...] So you can probably go fully static if you are only
| targeting Linux.
|
| There's yet another problem Go ran into on Linux: resolving
| hostnames. The official way to resolve hostnames in a glibc-
| based system is to read /etc/nsswitch.conf, dynamically load
| the libraries referenced there, and call each of them in the
| correct order. Go tried to emulate that mechanism, going
| through glibc only when it saw a NSS module it didn't know,
| but that caused issues on musl-based systems which used a
| different mechanism
| (https://github.com/golang/go/issues/35305). And it cannot
| ever be fully static on glibc-based systems, since there's
| always the chance of someone adding a new module to
| /etc/nsswitch.conf (for instance, some Linux distributions
| have now added the newer "resolve" module to that line, so Go
| would have to implement yet another one to keep being fully
| static).
|
| The Rust approach of dynamically linking to the shared C
| libraries (statically linking only the Rust libraries)
| avoided all these issues.
| mst wrote:
| > Operating systems can statically map libraries.
|
| I have "fond" memories of running perebaseall on cygwin that this
| sentence reminded me of.
| cesarb wrote:
| The first section, on security fixes ("[...] you only need to
| update a shared library to apply security fixes to all
| executables that rely on that library"), starts talking about
| security fixes but then suddenly switches to talking about ABI
| and API breaks. But that sleight-of-hand hides the fact that many
| (perhaps even most) security fixes do _not_ break the ABI or API;
| they are completely contained to the implementation (one obvious
| exception would be if the security issue was caused by bad API
| design, but even then often there are ways to fix it without
| breaking the ABI).
| rewma wrote:
| > But that sleight-of-hand hides the fact that many (perhaps
| even most) security fixes do not break the ABI or API; they are
| completely contained to the implementation (one obvious
| exception would be if the security issue was caused by bad API
| design, but even then often there are ways to fix it without
| breaking the ABI).
|
| Right you are. I was also perplexed when I read that non
| sequitur. The author's reference to DLL Hell also suggests
| there's some confusion in his analysis of the underlying
| problem, given that DLL Hell is very specific to windows and at
| best is only orthogonally related to ABI. The author's
| references to API changes make even less sense, and definitely
| cast doubt over his insight into the issue.
| salawat wrote:
| Differentiating DLL and SO hell is getting a bit beyond
| pedantic as they are implementations of the same fundamental
| abstraction. Any substantial difference in merely one of
| implementation details.
| rewma wrote:
| > Differentiating DLL and SO hell is getting a bit beyond
| pedantic (...)
|
| It really isn't. Unlike linking problems, where the problem
| is focused on how you need to fight your dependencies to be
| able to resolve symbols, DLL Hell has been for over a
| decade a dependency resolution problem that is solved at
| the packaging level.
|
| More importantly, on Windows, where sharing DLLs is not a
| thing, you can simply drop a DLL in the app dir and be done
| with it. In fact, it's customary for windows apps to just
| bundle all their dependencies.
| pjmlp wrote:
| And was kind of fixed with application manifests, on XP.
| ghoward wrote:
| Author here.
|
| Sure, most security fixes do not break the ABI or API. But it
| only takes _one_ , and then you have a security issue.
|
| So I can understand your sentiment that it was sleight-of-hand,
| but I argue that it's not.
|
| Also, we only _think_ that most security updates don 't break
| API or ABI because we have no automatic way of _checking_. It
| 's all thoughts and prayers. I'm a religious person, but
| thoughts and prayers are _not_ a good way to do computer
| security.
| gnufx wrote:
| > we have no automatic way of checking
|
| I do for Fedora packages I maintain. There can be false
| positives, like on symbols that shouldn't be exported, but
| you can look at the API to start with. (The library author
| should do that, of course.)
| zozbot234 wrote:
| If a security fix _must_ break the API, you can bump up the
| SONAME which forces all packages depending on your shared
| library to recompile against the new version. This is
| comparable to the requirements under static linking, but you
| only have to do it when absolutely needed.
| throw0101a wrote:
| > _Sure, most security fixes do not break the ABI or API. But
| it only takes one, and then you have a security issue._
|
| As someone who has been a Unix sysadmin for almost twenty
| years now, I don't recall every installing a patch or package
| update that broke the ABI/API.
|
| If one should happen to occur next week, I'll still go with
| leaning towards linked libraries, as their convenience out-
| weights (has out-weighted) the once-in-twenty-years
| occurrence of breakage.
|
| > _It 's all thoughts and prayers. I'm a religious person,
| but thoughts and prayers are_ not _a good way to do computer
| security._
|
| Ditto. But "faith" and "trust" are (almost) synonyms. And I
| trust the updaters of libraries to be able to get out a
| security update in short order more than I trust the
| possibility of all the package maintainers that link to it to
| be able to coordinate a quick update.
|
| When OpenSSL's Heartbleed occurred it was one package update
| and restarting of a bunch of processes (helped by the
| _checkrestart_ utility in the _debian-goodies_ package). That
| 's a lot quicker than updating a bazillion dependents.
| ghoward wrote:
| I understand where you are coming from. That's actually why
| I gave my ideas to get the best from both types of linking.
| xgbi wrote:
| Not related to the OP but I've had many issues with non
| standard vendors placing breaking changes in their
| patch/minor releases. You think you're updating to a
| relatively normal version and it actually breaks your
| entire production.
|
| I'm looking at you, MySQL, docker, Mongodb, or harbor.
|
| I'm not complaining, those are provided free of charge and
| we are happy when it works, but semver is really all that
| matters in this: either you adhere and it's smooth sailing,
| or you don't and then please don't use semver-looking
| numbering.
| dralley wrote:
| There's a difference between vendor provided packages and
| distro provided packages. Distros pay a lot more
| attention to breaking changes.
| [deleted]
| taviso wrote:
| I've (probably?) found more linker vulnerabilities than
| anyone else, but don't really understand your argument. I
| definitely understand library search path vulnerabilities
| (I've literally found bugs in $ORIGIN parsing, e.g.
| CVE-2010-3847 was one of mine!).
|
| We think security updates don't break API/ABI because they're
| backported to all previous major versions that are still
| actively supported. This isn't "thoughts and prayers", it's
| work that the packagers and maintainers do. I can't tell you
| how useful being able to fix a vulnerability in every program
| is, just think of updating every application that uses libpng
| on your desktop, or every server application that uses
| OpenSSL.
|
| I just can't imagine wanting to give that up, because of name
| mangling ABI issues!
| gnufx wrote:
| I hope that puts it to rest; I can hope. Unfortunately, my
| experience is that people who haven't fixed multiple
| security problems, and probably have never heard of taviso,
| know a lot more about security than people who have...
| ghoward wrote:
| I _have_ heard of taviso. That doesn 't mean I can't
| disagree with him.
| gnufx wrote:
| Sorry, I was talking generally. I try to avoid being
| personal in posts.
| [deleted]
| froh wrote:
| That's why you run a trusted Linux distro in production, like
| sles or rhel or ubuntu, their core value proposition is
| _exactly_ this: Provide binary compatible drop in replacement
| security updates for shared objects over the lifetime of the
| platform. They even maintain kernel internal abi
| compatibility for the released and maintained product, so
| your exotic-hardware.ko module doesn't need to be updated.
|
| Also there is mature tooling around ABI compliance checking:
| https://lvc.github.io/abi-compliance-checker/
|
| This is used to document the ABI stability of mature projects
| like so: https://abi-laboratory.pro/index.php?view=tracker
|
| E: re thoughts and prayers, the mature Linux distro all run
| extensive revalidation for their security updates. Breaking
| customer binary applications with a security update is the
| absolute worst case scenario. For yocto or similar cost free
| projects, or Linux distros that are source, not binary, that
| may be a different story. But for sles, rhel, ubuntu it's not
| thoughts and prayers but a maintenance verification and
| release process on their side and a monthly bill on yours.
| tinus_hn wrote:
| Considered harmful essays considered harmful:
|
| https://meyerweb.com/eric/comment/chech.html
| colejohnson66 wrote:
| I was wondering when something like this would be written.
| We've come full circle.
| ghoward wrote:
| Author here.
|
| I read that essay a while ago and reread it before publishing
| my post.
|
| I don't think the author makes a good case. They make bare
| assertions without examples and without evidence, so it's
| unconvincing.
|
| For example, they say that a "Considered Harmful" essay:
|
| > often serves to inflame whatever debate is in progress, and
| thus makes it that much harder for a solution to be found
| through any means.
|
| But this could be true of any essay.
|
| More to the point, I think that the _tone_ of the essay or post
| matters more. Case in point, my "Dynamic Linking Needs to Die"
| post probably inflames the debate more than this new
| "Considered Harmful" post.
|
| Maybe "Considered Harmful" essays have a _reputation_ for
| inflaming the debate, and there 's an argument to be made for
| that. But I personally don't think my post is one that follows
| that reputation.
|
| Also, because I was writing a post about a pre-existing
| "Considered Harmful" essay, it was natural to use it,
| especially since I needed to make it clear that my previous
| post is harmful too.
| jasode wrote:
| _> Maybe "Considered Harmful" essays have a reputation for
| inflaming the debate, and there's an argument to be made for
| that. But I personally don't think my post is one that
| follows that reputation._
|
| Neither your post title nor your content bothered me but just
| as fyi... some people are very irritated by _" snowclones"_.
| My previous comment with various examples:
| https://news.ycombinator.com/item?id=19071532
| ghoward wrote:
| Yeah, I suspected people might be. But since I was
| answering a "Considered Harmful" essay, it was too easy.
| Eh, oh well.
| IshKebab wrote:
| Either way, "considered harmful" is a tired and unoriginal
| cliche. Using it twice in one title is just stonkingly
| uncreative.
|
| I agree with most of the article but the title sucks!
|
| Also I'm not sure how you wrote so much about using LLVM IR
| as a portable executable format without mentioning
| WebAssembly or PNaCl once!
| ghoward wrote:
| > Either way, "considered harmful" is a tired and
| unoriginal cliche. Using it twice in one title is just
| stonkingly uncreative.
|
| > I agree with most of the article but the title sucks!
|
| You are correct. I don't claim to be creative, especially
| with post titles. To be honest, I'm surprised this post got
| as much traction as it did.
|
| > Also I'm not sure how you wrote so much about using LLVM
| IR as a portable executable format without mentioning
| WebAssembly or PNaCl once!
|
| Because, in my opinion, they will not solve the problem.
|
| WebAssembly is too constrained.
|
| PNaCl uses LLVM, which would need to be redesigned to do
| all that I want.
| lamontcg wrote:
| I sorta think that static linking needs to die, but dynamic
| linking also needs to be a lot better.
|
| More of the choices that get made during autoconf needs to be
| made lazily at link time. That would make executables and dynamic
| libraries more like containers or lego blocks that could be
| rearranged without compilation. Which would fix a lot of issues
| that people hate about package managers and bloated distros along
| the way.
| gnufx wrote:
| There are examples of the sort of reason I want dynamic linking
| under https://news.ycombinator.com/item?id=28734875 and you also
| really want it for profiling/tracing, and possibly debugging,
| e.g. in HPC with the MPI PMPI mechanism.
|
| libabigail is one tool for checking that a library's ELF
| versioning hasn't lied about ABI compatibility when you're
| building packages, for instance.
|
| Unfortunately people typically don't follow Drepper's advice (as
| far I remember the article) on building shared libraries,
| particularly using symbol versioning a la glibc.
| ghoward wrote:
| Author here.
|
| Yeah, there are reasons to use dynamic linking, but I'm still
| not sure why dynamic linking gives you better profiling and
| tracing.
|
| > Unfortunately people typically don't follow Drepper's advice
| (as far I remember the article) on building shared libraries,
| particularly using symbol versioning a la glibc.
|
| I mention this in the post, but I quoted another person, Ian
| Lance Taylor, who wrote the gold linker, as saying that symbol
| combining, a portion of symbol versioning, is an _unsolved
| problem_.
|
| Even if developers don't follow Drepper's advice, I wouldn't
| put the blame on them because of that fact; I would argue that
| it's more accurate to say they _can 't_ really follow his
| advice.
| dchapp wrote:
| > Yeah, there are reasons to use dynamic linking, but I'm
| still not sure why dynamic linking gives you better profiling
| and tracing.
|
| It's not so much that the tracing becomes _better_ , but that
| it becomes feasible at all. Two specific situations come to
| mind, both MPI-adjacent. (1) Running a PMPI-based tool on
| code you can't recompile yourself (e.g., you need a Q
| clearance to see the source, but not to actually execute it--
| weird I know, but not that uncommon in the DOE labs.); and
| (2) running multiple PMPI-based tools simultaneously which
| are composed at runtime via PnMPI.
| gnufx wrote:
| Exactly, but even if you can rebuild it, you don't want to
| under most circumstances. In principle with static binaries
| you can use dyninst, for instance, but in practice you may
| not be able to for various reasons. Then as a system
| manager you'd like to have global profiling of what runs --
| for various reasons, including putting the result for a
| program in front of a user. If it's all dynamically linked,
| you can hook in and do that by default, and with various
| levels of insistence, depending on the hooks.
| ghoward wrote:
| I addressed these arguments. I think that if you don't have
| access to the source, it's a security problem.
|
| That includes not having access to the source because of
| security clearances. As far as I am concerned, your
| superiors in the DOE are adversaries to you and your
| machine.
|
| Other people will have different viewpoints on that, and
| that's fine.
| gnufx wrote:
| I want free software, and I'm aghast at what some people
| run, but this is not the real world. How many examples do
| you want? If you talk security, what's the threat model?
| What I run on a decently managed (with the aid of
| dynamically linked libraries) compute cluster should only
| put my data at risk.
|
| As I understood it, there actually has been a push for
| free software solutions on CORAL systems, but I don't
| remember where that came from.
| gnufx wrote:
| Drepper's scheme obviously works tolerably well with glibc,
| in particular, even if he didn't know what he was talking
| about.
|
| If you don't have symbol versioning, and you want to
| statically link something with parts that depend on
| incompatible versions of a library, you're obviously stuffed.
| [deleted]
| arduinomancer wrote:
| It seems a pain point of static linking is that if a library
| changes you need to rebuild everyone
|
| Maybe a stupid idea but has there ever been thought of
| distributing applications with source code included?
|
| That way you could have a system that automatically rebuilds the
| application if a static library changes.
|
| Meaning when a user sees "updating" it's actually rebuilding the
| app underneath.
| ChrisSD wrote:
| > However, there is one big elephant in the room: right now,
| there are a lot of platforms that do not fully support static
| linking because you have to dynamically link libc or other system
| interfaces. The only one I know of that supports static linking
| is Linux.
|
| > The ironic thing is that operating systems only supported a
| stable syscall ABI, rather than requiring programs to dynamically
| link system libraries, they would have less problems with ABI
| breaks.
|
| > With a statically-linked binary, you can copy it to another
| machine with a Linux that supports all of the needed syscalls,
| and it will just run.
|
| I think this is a misunderstanding. With Linux there is a stable
| interface at the kernel boundary. Other OSes have their stable
| interfaces in user space. The difference is mostly irrelevant in
| this context. In one case you dynamically call into the kernel,
| in another case you dynamically link a user space library. Either
| way the code you're calling isn't static but the interface is.
| AshamedCaptain wrote:
| And to assume that only the syscall ABI layer is important is
| quite the understatement. I have software from the 90s that
| would work perfectly fine if it weren't for the fact that
| /etc/mtab is now a symlink. And that is just an example of the
| most smallish change -- not going to enter into the bazillions
| of binaries that are broken due to changes in audio APIs (even
| though the syscall numbers are still the same).
|
| Static linking for future-proofing/compatibility has never been
| a valid argument in my experience. Not only it is much easier
| to fix (and debug) a broken dynamically linked binary than a
| statically linked one, I would even dare to say that statically
| linked binaries break much more frequently than dynamically
| linked ones when you change the rest of the system. Even when
| you include the breakage from libraries changing under the feet
| of the dynamic binaries.
| CyberShadow wrote:
| There is a distinction between a fully-static binary and a non-
| fully-static one, which is what I think the article refers to.
| A fully static binary will not need to call the dynamic linker
| at all, and tools such as "file" or "ldd" will identify such
| binaries differently. A fully static binary will also run even
| in the complete absence of any userspace support for its
| architecture (assuming the hardware and kernel do support it) -
| e.g. a static AArch64 binary will run on a 32-bit ARM
| distribution, if the CPU supports AArch64.
| [deleted]
| ghoward wrote:
| Author here.
|
| This is exactly what I was getting at. Thank you.
| ChrisSD wrote:
| True, however...
|
| > assuming the hardware and kernel do support it
|
| Assuming kernel support is doing a lot of heavy lifting
| there. To put it another way, why is a syscall better than
| any other stable ABI? If both kernel and stable library are
| distributed together then why _should_ it be considered
| different?
| garethrowlands wrote:
| Indeed, Linux is the outlier here. Maybe it's better. But
| the designers of, say, Solaris, Mac and Windows didn't
| think so.
| SAI_Peregrinus wrote:
| Even Linux isn't really the outlier. The syscall
| convention is stable, but you're not statically linking
| the kernel into your binary.
|
| A "library OS" like FreeRTOS does exactly that: the
| kernel is just another library, with functions that you
| call like any other static dependency. You can only run
| one process (the OS & your userspace code are that
| process).
|
| Really the "fully static" side is only seen in practice
| in RTOSes and similar embedded systems work. Being able
| to dynamically load more than a single process is just
| too handy to give up entirely. I don't think the opposite
| extreme has even been tried (every single symbol in its
| own .so, even within the kernel), the overhead would be
| ridiculous.
| ghoward wrote:
| Author here.
|
| Kernel support is actually easy: since Linux hardly ever
| removes syscalls, just build your fully-static executable
| on the oldest Linux you have, and then deploy it on all of
| your relevant machines.
|
| The syscalls used by all of your libraries, if they work on
| that oldest Linux, would work on the newer ones.
|
| In fact, this is why AppImage "Best Practices" includes
| building on the oldest system. [1]
|
| [1]: https://docs.appimage.org/reference/best-
| practices.html?high...
| gnufx wrote:
| If you build against, say, RHEL5, you presumably acquire
| vulnerabilities in relevant libraries, give up hardware
| support, and still can't guarantee it will run correctly.
| That's at least because Linux interfaces aren't stable in
| general, specifically the pseudo-filesystem ones,
| thinking of real examples.
| Brian_K_White wrote:
| If a kernel and library really are always distributed
| together, so religiously iron clan that you can and should
| treat them as a single object, then why are they not in
| fact a single object?
|
| A syscall may not be inherently too different from any
| other interface, but a single interface is certainly
| different from two interfaces.
| PaulDavisThe1st wrote:
| The word "static" is being conflated with "stable".
|
| A "static interface" isn't a thing. There's stable and unstable
| interfaces, and static and dynamic linkage. Not strongly
| related.
| amelius wrote:
| I'm distributing my code as a bootable USB drive.
| 0xbadcafebee wrote:
| I wish I had three hours to waste on why almost all of these
| points are either innacurate or willfully misleading. But I
| don't, so I'll just point out the single biggest reason why
| static linking is harmful.
|
| Encryption libraries.
|
| If OpenSSL, or LibreSSL, or Go's encryption modules, or _any
| gigantic encryption library_ has a vulnerability, you basically
| have to recompile, distribute, and then have your users download,
| every single god damn networking program.
|
| But it's worse than that. Because everyone wants to ship a
| statically compiled Go app, nobody packages for the distros. So
| now every user needs to go track down where they downloaded each
| of their static Go apps, _safely_ download the update, verify its
| checksum or signature via a 3rd party, and upgrade the app.
|
| That is a logistical nightmare. Not only are apps going to remain
| vulnerable for way, way longer, more users will be compromised by
| phishing and other attacks because there's no package manager to
| safely automate all these updates (distros sign and verify
| packages for you).
|
| I've actually written a Shell quasi-package manager/installer
| just for statically compiled apps (cliv) so in theory that
| problem could be somewhat solved... But nobody even knows about
| that program, so the manual update problem still stands.
| AshamedCaptain wrote:
| If you ever statically link OpenSSL, your binary now has a
| builtin expiration date. You are doing your users a disservice.
| marcosdumay wrote:
| To be fair, if you dynamically link to OpenSSL, your binary
| has a builtin expiration date too, and it's very likely the
| same date.
| AshamedCaptain wrote:
| Not really. I have made updated versions of libSSL that
| allow use of new ciphers with the old ABI, for example.
| (Retro purposes). But for example this has been done to
| extend the life of other binary-mostly platforms (e.g.
| webOS). As long as I can change libSSL separately, this is
| trivial.
| nly wrote:
| OpenSSL ABI has been fairly stable within a given series.
|
| Something compiled against 1.1.0a (Sep 2016) has a fair
| chance of running on a system that has 1.1.0l (Aug 2021). 5
| years of binary compatibility is nothing to be sniffed at
| for a C library.
|
| https://abi-
| laboratory.pro/index.php?view=timeline&l=openssl
| KronisLV wrote:
| > If OpenSSL, or LibreSSL, or Go's encryption modules, or any
| gigantic encryption library has a vulnerability, you basically
| have to recompile, distribute, and then have your users
| download, every single god damn networking program.
|
| Good, then you'd know that these 50 of your networking apps
| have each been tested by their developers' test suites and have
| no bugs that have been introduced as a part of this dependency.
| Essentially, there would be a server farm out there that'd run
| all of the tests so you wouldn't have to deal with prod
| breaking because of some new dependency version not taking
| every possible configuration into account.
|
| Of course, that implies: - that there are
| automated builds in place - that there are automated
| tests in place - that both of the above are run
| regularly, on every dependency update - that both of the
| above are actually meaningful (think close to 100%
| functionality coverage in tests)
|
| The current situation of relying upon dynamically linked code
| is only prevalent because we as an industry have decided not to
| even attempt to do the above. I'd say that it's a matter of not
| being willing to put in effort, not being willing to test our
| software and packages, and not being willing to develop
| software more slowly but thoroughly. Everyone wants "good
| enough" now, rather than "almost perfect" in 10 years,
| regardless of whether we're talking about developing software,
| or the tooling around it.
|
| > But it's worse than that. Because everyone wants to ship a
| statically compiled Go app, nobody packages for the distros.
|
| In my eyes, that just adds to the above - there are so many
| deployment targets out there, all of these different packaging
| systems that are used by different distros and all have
| numerous idiosyncrasies as opposed to a single good package
| manager, that people have been fed up with it and now just
| either ship Docker containers or expect you to download
| binaries.
| ghoward wrote:
| Author here.
|
| I think you are strawmaning my arguments.
|
| Personally, I would include encryption libraries in the "system
| libraries" (I used that term on purpose) that could be
| dynamically linked.
|
| However, even if they are, you _still_ want to recompile every
| user whenever that library is updated because of possible ABI
| and API breaks. And some of those encryption libraries have a
| history of API breaks.
|
| Also, I am building the ideas I laid out in the post, which
| should make it easier to handle dynamically-linked _and_
| statically-linked encryption libraries.
| chrisseaton wrote:
| If you just update an encryption library and ship only that...
| then who's tested all the applications against the new version?
| salawat wrote:
| Not the library author's problem. That's the consumer's
| problem.
|
| I'm not saying it's great, but it's how it works.
| chrisseaton wrote:
| > That's the consumer's problem.
|
| Well this is the explanation for why people are static
| linking! They don't want to run someone else's entirely
| untested code and bear all responsibility for any problems.
| salawat wrote:
| For the record? I'm totally fine with that. As I'm
| basically one of the static linking group, or "use
| dynamic libraries in a way non-distinguishable from
| static linking".
|
| If I'm relying on it, it's getting tested and I'm not
| going to dork with it without good reason once it is.
| Wowfunhappy wrote:
| Are there any Linux distros that actually distribute (most)
| software as (mostly) static binaries? Right now, the only way to
| do it seems to be compiling everything myself...
| hhvn wrote:
| https://sta.li/ Experimental, probably not that usable.
|
| http://sabo.xyz/ More of a complete distro, but the dev himself
| says he still uses a couple of programs dynamically linked.
| ObscureScience wrote:
| I think there are only experiments. From what I know Oasis
| Linux is furthest along https://github.com/oasislinux/oasis
|
| Another resources is https://sta.li/ I thought there was a
| Archlinux-based attempt, but can't find references to that.
| lrem wrote:
| Why would you want that?
| Wowfunhappy wrote:
| Simplicity. If everything is statically linked, then updating
| package A can never break package B.
|
| As I see it, the entire world of Docker containers exists
| primarily to solve the problem that static linking already
| solved.
|
| Something like Nix would work as well, and I think I would
| love Nix if I knew how to use it. But I don't, and I don't
| really have the time to learn.
| lrem wrote:
| I don't think that makes a lot of sense in a distro. If
| package A gets an update, it means a sufficiently bad bug
| was discovered in it. If it happens to be a library, you
| want to make sure you update every package B that uses it.
| Which, conveniently, the distro solves for you too.
| Wowfunhappy wrote:
| This is true, but only if I'm actually getting
| _everything_ from the distro repository and I 'm able to
| keep all of it on the latest version. In practice, I
| sadly tend to run into situations where I need stuff from
| other places.
| lrem wrote:
| But then you don't want the software from the distro to
| be statically linked, but the software from other places.
| In which case it doesn't matter that much what does the
| distro do.
| Wowfunhappy wrote:
| Basically, the distros aren't perfect and I sometimes run
| into conflicts. Also, with static binaries it would be
| much easier to pin specific software at old versions
| without affecting anything else, even just temporarily
| for testing.
|
| IMO, when stuff is self-contained, everything becomes
| easier! This isn't to say some stuff like OpenGL can't be
| dynamically linked, but some day I'd love to use a distro
| with a static-first approach, because right now it simply
| isn't an option.
| Brian_K_White wrote:
| AppImage, Flatpack, Snaps... all seem to be largely
| reinventing the primary qualities, good and bad, of static
| binaries in a new and much more complicated form.
|
| At least a container provides more function than mere
| bundling, but a lot of containers actually get used merely
| for bundling.
| blablabla123 wrote:
| I think programs in /bin used to be usually statically
| linked. When updating the glibc for instance, this makes the
| change far more doable. In general dealing with static
| binaries is much less pain, especially when they haven't been
| compiled locally/by the distributor for this particular
| version of the distribution.
| drivebycomment wrote:
| /sbin used to be statically linked on unix. The entire
| point of /sbin is the independence from almost everything
| else so that you can rely on them even when the system
| boots up in a weird state.
| ghoward wrote:
| Author here. I don't know why my own submission of this [1]
| didn't catch and this one did. Oh well. XD
|
| I'm open to answering questions, and I'll be hanging in the
| thread.
|
| [1]: https://news.ycombinator.com/item?id=28728801
| comex wrote:
| I am extremely skeptical of this calculation by Drew DeVault:
|
| > On average, dynamically linked executables use only 4.6% of
| the symbols on offer from their dependencies. A good linker
| will remove unused symbols.
|
| Percent of exported symbols used is a _terrible_ proxy for
| percent of library functionality used.
|
| As an example, an SDL hello world program [1] uses 3 functions
| out of 750 exported by libSDL2. Does that mean it only uses
| 3/750 = 0.4% of libSDL2's functionality? Of course not; when I
| tried compiling it against a static libSDL2 instead of dynamic,
| the executable's binary size increased by a full 70% of the
| size of libSDL2.so. [2]
|
| Now, on one hand I admit that's an extreme example, as a real
| SDL program would use a higher number of functions. And SDL
| itself is not very amenable to dead code stripping; a good
| chunk of the hello-world program is for things like sound or
| input that the it doesn't actually use.
|
| Still, even the hello-world program legitimately needs a bunch
| of functionality, including support for both X11 and Wayland,
| and setting up hardware accelerated surfaces using OpenGL. And
| a real program would need more. Also, I'm not even counting the
| long list of shared libraries linked by SDL (ranging from
| libX11 to libFLAC); a SDL-based program will use 0% of their
| symbols (since it doesn't them directly), but may need a
| substantial portion of their functionality.
|
| More importantly, this general pattern, where exported APIs are
| just the tip of the iceberg, applies to many libraries.
|
| I tried a similar experiment with zpipe, the official example
| program for zlib - a much smaller and shallower library. It
| ended up using 23% of symbols but 53% of the implementation.
|
| On the other end of the spectrum, the problem would be far
| worse with a real GUI library, where even, say, a simple text
| input box truly requires massive amounts of functionality
| including display, layout, Unicode font rendering, menus,
| internationalization, and keyboard and mouse input. Plus, of
| course, a variety of input method editors if you want to
| support languages like Japanese.
|
| In a statically linked world you would probably want to
| implement most of that as IPC to shared daemons. Such a design
| could work; there isn't much that really needs the performance
| benefits of being in-process. But no major GUI libraries are
| written that way, because in a world with shared libraries
| there's no need.
|
| [1]
| https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b...
|
| [2] Test harness also at [1]. This is with LTO; without LTO the
| percentage is much higher. I also had to disable SDL's "dynapi"
| feature that makes static linking secretly actually dynamically
| link. Which they put in for good reasons, albeit mostly
| applicable only to proprietary software where the user can't
| just recompile/relink.
| lgg wrote:
| Every time I read one of these essays extolling the virtues of
| static linking it makes me pretty sad. Engineering is about trade
| offs, and sometimes it does make sense to statically link, but
| the fact that those trade offs are so lopsided on Linux has very
| little to do with dynamic linking vs static linking as generic
| concepts, and more to do with the fact that the design of ld.so
| is straight out of the 1990s and almost nothing has be done to
| either exploit the benefits dynamic linking brings, nor to
| mitigate the issues it causes.
|
| On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have
| invested heavily in dynamic linking over the last two decades.
| That includes features we use to improve binary compatibility
| (things like two level namespaces (aka Direct Binding in ELF) and
| umbrella frameworks), middleware distribution (through techniques
| like bundling resources with their dylibs into frameworks), and
| mitigate the security issues (through technologies like
| __DATA_CONST).
|
| Meanwhile, we also substantially reduced the cost of dynamic
| linking through things like the dyld shared cache (which makes
| dynamic linking of most system frameworks almost free, and in
| practice it often reduces their startup cost to below the startup
| cost of statically linking them once you include the cost of
| rebasing for PIE), and mitigate much of the rest of the cost of
| dynamic linking through things like pre-calculated launch
| closures. It does not hurt that my team owns both the static and
| dynamic linkers. We have a tight design loop so that when we come
| up with ideas for how to change libraries to make the dynamic
| linker load them faster we have the ability to rapidly deploy
| those changes through the ecosystem.
|
| As I explained in here[1] dynamic linking on macOS is way faster
| than on Linux because we do it completely differently, and
| because of that it is used way more pervasively on our systems (a
| typical command line binary loads 80-150 dylibs, a GUI app around
| 300-800 depending on which of our OS you are talking about). IOW,
| by mitigating the costs we drove up the adoption which amplifies
| the benefits.
|
| And it is not like we have squeezed all the performance out of
| the dynamic linker that we can, not by a long shot. We have more
| ideas than we have time to pursue, as well as additional
| improvements to static linking. If you have any interest in
| either static or dynamic linking we're looking for people to work
| on both[2].
|
| [1]:
| https://www.realworldtech.com/forum/?threadid=197081&curpost...
|
| [2]: https://jobs.apple.com/en-us/details/200235669/systems-
| engin...
|
| (edit: fixed link formatting)
| ghoward wrote:
| Author here.
|
| I'm curious: did you read to the end of the post where I lay
| out ideas about how to get the benefits of both in one?
| lgg wrote:
| Yes. They are all reasonable ideas, and we actually have
| experience with most (all?) of them:
|
| * Distributing IR applications: We do a form of this by
| supporting bitcode for App Store submissions on iOS, tvOS,
| and watchOS. For watchOS it was a great success in the sense
| that it allowed us to transparently migrate all the binaries
| from armv7k (a 32 bit ABI on a 32 bit instruction set) to
| arm64_32 (a 32 bit on a 64 bit instruction set), but we had
| to carefully design both ABIs in parallel in order to allow
| that to be efficient. It also introduces serious burdens on
| developer workflows like crash reporting. Those probably are
| not significant issues for people deploying binaries to
| servers, but it can be pretty difficult for developers trying
| to aggregate crash statistics from apps deployed to consumer
| devices. It also causes security issues with code provenance
| since you have to accept locally signed code, have the
| transforms performed by a centrally trusted entity who signs
| it, or you have to limit to your optimizations to things you
| can verify through a provable chain back to the IR.
|
| * Split stacks: We don't do this per se, but we do something
| semantically equivalent. When we designed the ABI from arm64e
| we used PAC (pointer authentication codes) to sign the return
| addresses on the stack, which means that while you can smash
| the stack and overwrite the return pointer, the
| authentication code won't match any more and you will crash.
|
| * Pre-mapped libraries: We have done this since then Mac OS X
| Developer Releases back in the later 90s, though the
| mechanism has changed a number of times over the years.
| Originally we manually pre-assigned the address ranges of
| every dylib in the system (and in fact we carefully placed
| large gaps between the __TEXT and __DATA of the library and
| have the libraries in overlapping ranges so that we could
| place all the system __TEXT in single adjacent region and all
| the __DATA in a second adjacent region that way we could
| exploit the batch address translation registers on PPC
| processors to avoid polluting the TLBs). When the static
| linker built a dylib we look in a file with the mappings and
| build the dylib with segment base addresses that we looked up
| from the that manually maintained list, and when the OS
| booted we would pre-map all the libraries into their slots so
| every process just had them mapped.
|
| That was a huge pain in the neck to maintain, since it meant
| whenever a library grew too large it was would break the
| optimizations and someone would need update the file and
| rebuild the OS. It also only dealt with rebasing, to deal
| with binding we locally ran update_prebinding which users
| hated because it was slow, sysadmins hated because it means
| we routinely rewrote every binary on the system. That was
| also before anyone had deployed ASLR or codesigning.
|
| Nowadays we use the dyld shared cache to achieve similar
| ends, but it is a far more flexible mechanism. We essentially
| merge almost all the system dynamic libraries at OS build
| time into one mega-dylib that we can pre-bind together and
| sign. We also use VM tricks to allow the system to rebase it
| on page in rather than the dynamic linker doing any work, and
| we pre-map it into every address space by default.
|
| We even perform a number of additional optimizations when we
| build it, such as analyzing the dependencies of every
| executable in the base system and pre-calculating a lot of
| the work dyld and even the dynamic runtimes like ObjC would
| have to do in order to avoid doing them during app launch.
|
| So in short, I think your ideas there have more merit than
| you perhaps suspect, and from experience I can say if/when
| you implement them you implement them you might find your
| views on the trade offs of dynamic vs static linking change.
| ghoward wrote:
| Thank you.
|
| And after reading your response, I might give up on at
| least one of those ideas.
| osigurdson wrote:
| I really dislike the tired re-use of the term "considered
| harmful". A better title may be "I don't like dynamic linking -
| here's why".
| Jach wrote:
| I'd like to see solutions like SDL2's become more widespread:
| https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a...
| At runtime, the first call into SDL sets up a jump table for all
| SDL functions afterwards. When SDL is statically linked, it uses
| those static functions by default, but it's possible to specify
| an environmental variable pointing at an SDL dynamic lib, and the
| jump table will use those functions instead.
| dhsysusbsjsi wrote:
| Personal end user perspective: troubleshooting dylib errors is
| the most common cause of wanting to throw the computer out the
| window and quit programming.
| cratermoon wrote:
| See also: DLL Hell in Windows and, more generally Dependency
| Hell. Dynamic linking errors are a subset of general dependency
| problems.
| spacechild1 wrote:
| As others have already mentioned elsewhere, DLL hell hasn't
| really been a thing anymore on Windows for years. I
| personally can remember it, but that's more than 15 years
| ago.
| eqvinox wrote:
| "end user perspective" + "quit programming"? Do you
| troubleshoot dylib errors on your regular end user software and
| just happen to be a programmer unrelated to that?
|
| (If you're troubleshooting dylib errors on things you're
| developing, that's not an end user perspective?)
| salawat wrote:
| Yes. If you've never had a clashing Debian package build
| before, you've not been paying attention. I've made it my
| mission in life to try to get people more aware of linkers,
| and the fact that when you build something the one way it
| works on your system it doesn't mean it will work that way on
| someone else's.
| eqvinox wrote:
| I've never had a clashing Debian package build before, and
| I build quite a few Debian packages. Some for a wider
| audience and some for my own use.
|
| All packages that leave the confines of my own systems are
| built the "correct" way, in a clean minimal chroot with
| sbuild/schroot/cowbuilder. The docs on how to set that up
| are pretty good, and after you've done it once it's pretty
| easy to repeat for extra target distros (e.g. older Debian,
| Ubuntu.)
|
| What would you point out to me as part of your mission?
|
| [Just to be clear, a package is a package for a particular
| target distro. There's no such thing as "a .deb for
| software XYZ". Only "a buster .deb for software XYZ", or "a
| bullseye .deb for software XYZ", etc.]
| dizzy3gg wrote:
| End user of a linker?
| rewma wrote:
| > End user of a linker?
|
| I'd argue that if you're not packaging your software or
| testing your software's dependencies, either you're doing
| something extremely exotic that lies far outside anyone's
| happy path or "dylib error" should not even be a keyword in
| your vocabulary.
| alisonkisk wrote:
| The windows version, DLL Hell, is well known to non
| programmers.
| pjc50 wrote:
| Roughly the same thing happens on Windows and is called
| "DLL hell", occasionally witnessed by end users.
| rewma wrote:
| DLL Hell ceased to be a practical concern over a decade
| ago, particularly given that Windows provides tight
| control over its dynamic linking search order.
|
| https://docs.microsoft.com/en-
| us/windows/win32/dlls/dynamic-...
|
| DLL Hell is not a linking problem, it's a packaging
| problem.
| gbear605 wrote:
| There's a middle level of "troubleshooting dylib errors on
| open source software I'm installing for personal use because
| I'm a masochist who isn't happy with closed source software"
| eqvinox wrote:
| Not sure that's a solvable problem, or even a problem that
| should be solved. It's the distributions' job to create
| packages to build a working whole system. If you decide to
| take that on yourself... I mean... yeah, you gotta learn
| how to handle DSO issues. Because you decided you want to
| do it yourself.
|
| I don't think I've ever run into DSO problems with any
| reasonable distro packaging (which, in this case, excludes
| old Gentoo -- current Gentoo doesn't have those problems
| anymore either.)
| AnIdiotOnTheNet wrote:
| But if the distro doesn't package software you want to
| use, either because it only has an out of date version,
| modified version, or was just never packaged, then you're
| left dealing with this nonsense.
|
| In saner OS's where they don't rely on unpaid third party
| middlemen between the developer and the user this is
| rarely a problem.
| cratermoon wrote:
| > In saner OS's where they don't rely on unpaid third
| party middlemen between the developer and the user this
| is rarely a problem.
|
| Windows seems to fit your criteria for an OS without
| unpaid third parties, and there still exists a thing
| known as DLL Hell.
| spacechild1 wrote:
| I haven't really encountered DLL hell in Windows for the
| last 15 years or so. Most apps simply bundle all their
| dependencies locally and don't install them system wide.
| KronisLV wrote:
| Why are we still developing software like we used to 40 years
| ago?
|
| Why not just import external dependencies into your project at a
| function/class level rather than at a package one: if you only
| use FooPackage.BarClass and FooPackage.bazMethod(), you should be
| able to choose to make your project depend on just those two
| things, ideally in a completely transparent way by your IDE.
|
| Then having to manage the full scope of packages and their
| versions becomes less relevant, because the IDE can check whether
| a new version has been released in the package manager of choice,
| check whether those two things have changed and if they have,
| then demand that the developer have a look at the code changes to
| refactor code if necessary, otherwise not requiring any action on
| their part. Furthermore, if you ever need to migrate to a
| different package or even rewrite the functionality itself, you
| could just look at the few signatures that you use, rather than
| having to remove the package entirely and see what breaks in your
| IDE. Why can't we have itemized lists of everything that's called
| and everything that we depend on, as opposed to the abstraction
| of entire packages?
|
| Better yet, why even depend on binary blobs or hidden code
| (that's ignored by your IDE) in the first place? Why not just
| download the source for every package that you use and upon
| updates be able to review the code changes to the package much
| like a regular Git diff?
|
| Of course, this would probably require getting rid of reflection
| and other forms of dynamic code, which i fully support, since
| those have never been good for much in the first place and only
| destroy any hopes of determinism and full control flow analysis.
|
| As for the possible counterargument of this being hard to do: it
| wouldn't really be with more granular packages. Instead of trying
| to shove an ecosystem inside of a single package, why not split
| it into 10-20 bits of reusable code instead? Smaller packages,
| which would be easier to review and manage.
|
| Context: i dislike how Spring and Spring Boot in Java force a
| huge ecosystem of fragile dependencies upon you, with their
| reflection which ensures that your apps will break at runtime,
| for example, when moving from Spring Boot 1.5 to 2.0.
| Furthermore, in the JS world, the node_modules folders are
| probably 10-1000x too large for what's actually necessary to
| display a webpage with some interactive behaviour.
|
| Disclaimer: i have no delusions about any of the above being
| technically feasible right now. Perhaps "Why?" would be a good
| question to ask. In my eyes, perhaps the industry is too set in
| its current ways and as long as we don't have languages and
| entire ecosystems that approach established practices from a
| wildly different angle, no one can actually contest the decades
| of already built packages and tools.
|
| On the bright side, WireGuard kind of displaced OpenVPN somewhat
| and there are numerous benefits to it being smaller, which is a
| good example of getting rid of bloated software. Furthermore, Nix
| may or may not do that to alternative approaches to package
| management, but its usage still remains really low.
|
| In my eyes, the only long term solution is to be able to tell
| exactly what's necessary for your code to build and work, and to
| test every dependency update that comes out against this
| automated process. Static dependencies but at the speed of
| updates of dynamic dependencies. Then you'd just have to wait for
| a day until the devs fix the newest regressions of a new
| dependency release before getting a new statically linked app, as
| opposed to using dynamic linking and finding that some dependency
| breaks your app as it happens in prod.
| nemetroid wrote:
| Joe Armstrong had some similar thoughts: "Why do we need
| modules at all?".
|
| https://erlang.org/pipermail/erlang-questions/2011-May/05876...
| KronisLV wrote:
| You know, the "all functions have unique distinct names" idea
| is pretty interesting!
|
| Honestly, that reminds me of the PHP standard library a bit,
| which feels oriented towards procedural programming at times.
| Now, the language and the actual implementation of the
| functions aside, i found that pattern really pleasant to work
| with, especially since it let me work more in line with
| functional programming principles as well - pure functions
| and no global state (even if passing data around was more
| difficult).
|
| Now, i won't say that the approach fits every project or
| domain out there, but at the very least having a clear idea
| of what your code depends on and what's going on inside of
| your code base is an idea that i still stand by. Sadly, the
| only tool that i found that at least tries to make this
| easier was Sourcetrail, which is essentially dead now:
| https://www.sourcetrail.com/
| flohofwoe wrote:
| > if you only use FooPackage.BarClass and
| FooPackage.bazMethod()...
|
| That's how it works already with static linking (just under the
| hood), the linker will discard unused code and data (with LTO
| this happens automatically for all dead code and data,
| otherwise some special compiler/linker/librarian-options may be
| needed).
|
| A DLL on the other hand cannot predict what API functions will
| actually be called on it, so everything must be included.
| Brian_K_White wrote:
| As an end user, I would LOVE to be able to run a calculator app
| without having to pull in 500M of kde libraries and even
| daemons.
|
| But this is pretty much what a static binary already does, did,
| 40 years ago. The developer has access to the full universe,
| but the only thing that ends up in the binary are the bits they
| actually used. Problem solved, 40 years ago.
| PaulDavisThe1st wrote:
| > Why not just import external dependencies into your project
| at a function/class level rather than at a package one: if you
| only use FooPackage.BarClass and FooPackage.bazMethod(), you
| should be able to choose to make your project depend on just
| those two things, ideally in a completely transparent way by
| your IDE.
|
| TFA is about programs that require a linker at either compile
| and/or run time. The references in the rest of your comment to
| Java and JS suggest that your experience with programming in
| languages that require linking against compiled libraries might
| be limited.
|
| What you describe is _precisely_ what the linker is for! It
| looks at the code you wrote (and compiled), sees that it calls
| FooPackage.bazMethod(), and then goes and looks for that call
| along the linker search path using a combination of implicit
| and explicit library names. If it 's the static linker, it
| imports the code for that call, and then recurses to check the
| dependencies of FooPackage.bazMethod(). If it's the dynamic
| linker, it does the same, except differently (runtime pointer
| swizzling etc. etc)
|
| When you link your program against libfootastic.a, the linker
| is doing exactly what you suggest, and in most IDEs that step
| is going to happen automatically. When you link your program
| against libfootastic.so, the linker will do exactly what you
| suggest, but at runtime.
|
| > Better yet, why even depend on binary blobs or hidden code
| (that's ignored by your IDE) in the first place? Why not just
| download the source for every package that you use and upon
| updates be able to review the code changes to the package much
| like a regular Git diff?
|
| Many large end-user desktop applications do precisely this. For
| Ardour (I'm the lead dev), our build system downloads 80+
| source code packages and builds them locally to create a 2GB
| installed dependency tree of compiled libraries and ancillary
| files. There are a lot of arguments against doing this, and
| some arguments in favor.
|
| However, we can't compile each file from each dependency as
| part of the Ardour build because the build systems for those
| dependencies are complex and independent. There's no way to
| force the developers of libfftw (Fastest Fourier Transform in
| the West) to somehow prepare their code structures so that we
| can trivially replicate their entire build system as part of
| ours. If there was only a single build system for EVERYTHING,
| sure, maybe this would be feasible. But there isn't, and I'm
| completely certain that there never will be.
| ensiferum wrote:
| Yep wrangling the dependencies really becomes a tremendous
| pain at scale. Chromium for example does the same thing and
| they have enough engineering backing to be able to throw
| enough manpower into it so that that can get each and every
| single dependency in the same build set using the same build
| files and tools (gyp/gn/ninja). While feasible for an org
| such as Google this is totally unfeasible for a small shop
| with Limited engineering man hours. Really wish there was an
| easier way to deal with all of this. Sigh
| 3np wrote:
| ""Static Linking Considered Harmful" Considered Harmful"
| Considered Harmful:
|
| This is the blog version of Re: Re: Fwd: Re: in email and has to
| stop here.
|
| See also https://meyerweb.com/eric/comment/chech.html
| Brian_K_White wrote:
| Incorrect.
|
| Ok fine, explaination: "Re:" are not art. "Re:" are not added
| deliberately and for intentional artistic effect, and are not
| references (despite the literal) or homages.
| ghoward wrote:
| Author here.
|
| See my reply to https://meyerweb.com/eric/comment/chech.html at
| https://news.ycombinator.com/item?id=28737279 .
| rascul wrote:
| According to Linus, "Shared libraries are not a good thing in
| general".
|
| https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
| ghoward wrote:
| Author here.
|
| I quote that email twice in my post. I decided not to quote
| that part of the email just in case it might inflame the
| debate.
|
| But yes, he did say that.
___________________________________________________________________
(page generated 2021-10-03 23:01 UTC)