[HN Gopher] How Swift achieved dynamic linking where Rust couldn...
___________________________________________________________________
How Swift achieved dynamic linking where Rust couldn't (2019)
Author : zdw
Score : 117 points
Date : 2021-02-20 17:28 UTC (5 hours ago)
(HTM) web link (gankra.github.io)
(TXT) w3m dump (gankra.github.io)
| mark_l_watson wrote:
| There is a lot of great tech behind Swift and LLVM. I really
| enjoyed learning a bit (no pun intended) of Rust a few years ago,
| but it didn't stick for what I need. I really like Swift, but I
| have only done one SwiftUI app, all the Swift stuff I do is on
| the command line with swift build/test/run with a light weight
| editor. Apple's pre-trained deep learning models are very useful,
| and an edit/build/run iteration is just a second or two.
|
| I wish Rust had been around 30 years ago when I loved doing low
| level programming. Between my C++ books and gigs, C++ was good
| financially but Rust is just better (apologies to people who love
| C++).
| onei wrote:
| While many of the ideas in Rust are not particularly radical in
| terms of compiler research, I think part of its success is
| because it had the past 30 years of lessons. We all stand on
| the shoulders of giants. Maybe in another 30 years we'll see a
| new language that learns from Rust's inevitable mistakes.
| RcouF1uZ4gsC wrote:
| > I wish Rust had been around 30 years ago when I loved doing
| low level programming.
|
| I think Rust's killer feature is lifetime analysis by the
| compiler. In 1991, you had 33 MHz 80486 with around 4 MB of
| RAM. I don't think the average developer had the computing
| power to do Rust's lifetime in any reasonable amount of time.
| zozbot234 wrote:
| To be extra clear about this, Rust _does_ support dynamic
| linking; it 's just limited to interfacing via the stable C ABI.
| The ABI-resilience conerns described in the article are
| ultimately punted to the users and/or the dev ecosystem, albeit
| some facilities (namely, bindgen/cbindgen) are also provided to
| ease the task.
| glandium wrote:
| With a big caveat: you can't have a dependency on a crate that
| is a dylib/cdylib or even staticlib in Cargo.toml. Your only
| way out is to have a build.rs that somehow builds the
| dependency with cargo. Which, if you're using a workspace, will
| deadlock. https://github.com/rust-lang/cargo/issues/8938
| pjmlp wrote:
| Usually when a language supports dynamic loading we refer to
| the language itself, as integration with OS ABI dynamic loader
| is a given for any language worth using.
| zamalek wrote:
| In addition, Rust doesn't have to monomorphize generics. The
| choice is left to the developer (`Box<dyn Trait>` is how it is
| done).
| steveklabnik wrote:
| _technically_ speaking, you can dynamic link regardless of the
| abi or not, using a stable ABI just provides you degrees of
| freedom with which compiler version compiles what. They 're
| orthogonal concepts, even if they're most useful when combined.
| cakoose wrote:
| Some similar work:
|
| The GCC compiler for Java (GCJ) started out doing whole-program
| compilation, but that couldn't handle Java's flexible ABI and
| dynamic loading. In 2004 they introduced a compilation mode with
| an additional layer of indirection similar to Swift's:
| <ftp://gcc.gnu.org/pub/gcc/summit/2004/GCJ%20New%20ABI.pdf>. They
| didn't have to deal with monomorphizing, though.
|
| Go's does something similar to Swift to avoid code bloat in their
| built-in polymorphic map:
| <https://dave.cheney.net/2018/05/29/how-the-go-runtime-
| implem...>. It's just a one-off manual thing, though.
| RcouF1uZ4gsC wrote:
| To be honest, I am actually glad about this. Dynamic linking is
| the cause of a lot of complexity. First there is DLL hell on
| Windows and its equivalent on Linux. Second, having to maintain a
| stable ABI is a huge drain on language evolution. In C++, there
| have been many features that have been vetoed by the
| compiler/standard library implementers because it would cause an
| ABI break. Because of this ABI concerns, C++ is stuck with
| overheads for unique_ptr, map, unordered_map, and regex classes
| that are inefficient.
|
| Secondly, a lot of the benefits of dynamic linking don't make as
| much sense now.
|
| First with regards to space savings. With modern disk capacity
| having extra copies of executable code is probably not that big
| of a deal. In addition with Link Time Optimization, you actually
| may not be saving space with dynamic linking since the linker can
| pretty aggressively throw away code that it knows is ever called.
|
| Second, with regards to security. One thing to consider is that
| dynamic linking is so complex that it is one of the reasons why
| people use docker, to make sure their binary and its dependencies
| are stable. Once you have something in a docker, it has a lot of
| the same update issues as a static linked binary, only less
| efficient.
|
| In addition, with security, Rust unlike C and C++ is a memory
| safe language. Given the vast majority of security issues are
| memory safety issues, I would expect Rust to have a lot fewer
| CVE's. I think the experience with Go is likely enlightening in
| this regard. I am not aware of a huge issue of lack of security
| because it didn't do dynamic linking and you couldn't do security
| updates as easily.
|
| Finally, Rust as a new language is embracing the new way of
| programming and deploying. We are shifting away from using binary
| artifacts (often closed source) that once built were rarely
| changed, to build from source and continuously build/deploy.
| Cargo makes it relatively easy to do this. In that model,
| updating a binary for security is just a subset of the normal
| building and updating of a binary that is done daily.
|
| By avoiding dynamic linking, I think, Rust positions itself best
| as the no-compromise, high performance, modern, systems
| programming language.
| millstone wrote:
| Rust is certainly made simpler by not sweating ABI stability.
| But the main advantage of dynamic linking is to enable
| yesterday's app to run on tomorrow's OS (and vice-versa). It's
| why your apps keep working when you update your phone.
|
| Without a dynamic linking story, Rust is just not viable for
| writing UIKit, Android's frameworks, etc. Which is OK, it's a
| reasonable choice, but it limits Rust's scope.
| dralley wrote:
| Zig is in a really good position here. Its featureset sticks
| closer to C, and being able to automatically generate C
| headers from Zig code and automatically consume C headers
| from Zig code is really convenient.
| pjmlp wrote:
| Not until it sorts out its use-after-free story.
| millstone wrote:
| Does Zig really solve this problem? For example, if I add a
| field to a Zig struct, can existing code still use the
| struct without needing to be recompiled?
| dralley wrote:
| No, but what I'm saying is, it makes dynamic linking a
| lot easier in general than Rust does.
| pjmlp wrote:
| >. One thing to consider is that dynamic linking is so complex
| that it is one of the reasons why people use docker, to make
| sure their binary and its dependencies are stable.
|
| Coding for almost 40 years and I am yet to use Docker to sort
| out this kind of issues.
|
| > I think the experience with Go is likely enlightening in this
| regard. I am not aware of a huge issue of lack of security
| because it didn't do dynamic linking and you couldn't do
| security updates as easily.
|
| When I started programming, dynamic linking was only available
| on big iron machines filling computer rooms, we did not need Go
| for knowing what static linking entails.
|
| > Finally, Rust as a new language is embracing the new way of
| programming and deploying. We are shifting away from using
| binary artifacts (often closed source) that once built were
| rarely changed, to build from source and continuously
| build/deploy. Cargo makes it relatively easy to do this. In
| that model, updating a binary for security is just a subset of
| the normal building and updating of a binary that is done
| daily.
|
| If Rust wants to succeed in replacing C and C++ in typical big
| corp, cargo better support binary libraries eventually.
| gpm wrote:
| You must work in very different big corps than I, I have yet
| to see a binary .so we linked to but didn't have the source
| to.
| pjmlp wrote:
| The kind of corps where Oracle and SQL Server licenses are
| tiny drops on project expenses, do Windows, macOS, iOS,
| Android and UNIX in general.
|
| I am quite sure WebSphere, just to give an example, isn't
| releasing the source code for the .so that come along its
| Java implementation.
| ChrisMarshallNY wrote:
| If I use SPM, I seldom do dylibs. Since I wrote almost every
| package I consume, and the ones I didn't write are quite small;
| that works fine, for me.
| MR4D wrote:
| > has a lot of the same update issues as a static linked
| binary, only less efficient.
|
| I think this is an insightful comment. Obviously there are
| other reasons to use docker, but too often
| (cough...Python...cough) I see virtualization used for exactly
| that.
| kaishin wrote:
| Swift has unfortunately no choice but to solve this problem if
| it is to be used to build native software for Apple platforms.
| Animats wrote:
| I tend to agree. Dynamic linking is only a win on space when 1)
| you have multiple programs running which share the same
| library, 2) they're not all identical programs which would
| share code, and 3) the programs use a significant fraction of
| what's in each dynamic library they load.
|
| With static linking, you get to prune at the function level at
| link time. DLLs can't do that.
| coliveira wrote:
| A major use of dynamic linking is loading extensions. Without
| DL you'll have a hard time using languages such as Python, or
| creating extensions to applications.
| sgtnoodle wrote:
| Dynamic linking and dynamic loading aren't necessarily the
| same thing, are they? I imagine one could statically link
| in some C code to a rust program that uses dlopen, for
| example, or even just do the syscall directly in rust
| (assuming that's possible).
| pjmlp wrote:
| Yes they are, exactly the same OS infrastructure.
|
| It doesn't matter if it is the kernel or your application
| that calls dlopen().
|
| How do imagine that something like ld.so gets
| implemented?
| sgtnoodle wrote:
| The difference, I think, is that dynamic linking
| necessarily binds local symbols to the dynamically loaded
| foreign code, while dynamic loading simply loads the
| foreign code into the same address space and provides an
| API for looking up symbols' addresses by name. There's no
| inherent need for the ABI to match, as long as you don't
| directly try to call into a function; you can write
| adapters to explicitly handle the dynamically loaded
| code's ABI, like python does with ctypes.
| pjmlp wrote:
| Nope, that is the same infrastructure, just the details
| change across how they get exposed across OSes, specially
| non-POSIX ones.
| sgtnoodle wrote:
| At the end of the day it's all machine code running on
| silicon. What point are you trying to make? I was
| responding to the claim that a lack of good support for
| dynamic linking in a language prevents it from using
| shared objects for plugins and modules. My point was that
| dynamic loading is an operating system feature
| implemented through system calls, and can therefore be
| done in any language that can do a system call, whether
| or not the language makes it easy for you. I also claim
| that dynamic linking is not equivalent to loading, but I
| agree with you that they are related to each other and
| linking is indeed implemented on top of the system calls
| that provide dynamic loading. Dynamic linking
| specifically implies (for me) that non-static application
| symbols are resolved one way or another before the
| program counter jumps to main().
| coliveira wrote:
| In abstract you could separate these things, but in
| modern OSs there is little to no difference between these
| concepts. Dynamic loading is implemented using dynamic
| linking and if you don't want to use dynamic linking then
| you have to reinvent the wheel down to the OS level. You
| will have a very hard time to make dynamic loading
| available in any architecture following this route.
| pjmlp wrote:
| And you are forced to run multiple processes with OS IPC, as
| means for application extensions, which uses a lot of more
| CPU resources to get going.
| zozbot234 wrote:
| But then you can simply deploy these multiple processes on
| separate machines and connect them via a network, and you
| basically get microservices!
| pjmlp wrote:
| Only when they make use of IPC APIs that allows for that.
|
| And even then good luck getting everything to work
| without issues.
|
| Ah the wonders of having a language server microservice
| running across the network, or a image conversion plugin
| or audio DAW.
| randomNumber7 wrote:
| > By avoiding dynamic linking, I think, Rust positions itself
| best as the no-compromise, high performance, modern, systems
| programming language.
|
| You can use static linking in c++ if you want to. There is 0
| advantage in not having the option to use dynamic linking if
| you want to. It might be even more secure if e.g. the program
| links dynamically against a commonly used library and this
| library is updated regularly.
| [deleted]
| RcouF1uZ4gsC wrote:
| > You can use static linking in c++ if you want to. There is
| 0 advantage in not having the option to use dynamic linking
| if you want to.
|
| While you can do static linking in C++ if you want to, you
| still pay the cost for support for dynamic linking because of
| ABI issues. You can static link, but you still lost out on
| new language features and changes that break ABI and so you
| are still stuck with sub-optimal standard library code. For
| example, for years, until GCC 5, GCC std lib had a non-
| standard, slower copy on write implementation of std::string,
| that stuck around for so long because it would break ABI to
| change it. Even now, in MSVC STL, there are a lot of
| performance optimizations which the maintainer acknowledges,
| but will have to wait because they would break ABI. So even
| if you just statically link and continuously build in C++,
| you are still paying the price for support of dynamic
| linking.
| pjmlp wrote:
| Microsoft breaks the ABI all the time, this is why there is
| usually several MSVCrt.dll versions installed.
|
| Visual Studio 2017 - 2019 were the first time they ever
| bothered with it, plus it is planned that VS vNext will
| break it again.
|
| This not counting the other C++ compilers on Windows with
| their own standard libraries, like C++ Builder and Intel
| C++.
|
| There is always COM/WinRT if I want an OOP ABI that doesn't
| break as easily.
| coliveira wrote:
| First of all, C/C++ can do static linking as well as dynamic
| linking. If static linking is preferred, people will do it.
| Second, dynamic linking is necessary because of how modern OSs
| are designed. Most GUIs are gigantic libraries that cannot be
| statically linked. Similarly for network code and other areas
| of modern OSs.
| kortex wrote:
| I largely agree and personally hate working with dynamic
| linking, but ultimately it depends on your architecture model -
| to share memory, or not to share memory. If you've already
| decided you don't want to share memory, and you don't want to
| use system IPC, that leaves you with sockets
| (network/unix/fifo), which are all just byte streams. If you
| have byte streams, then you gotta use (de)serialization (serde)
| on either end, so effectively there is one ABI: {uint8[],
| size_t}. This is the microservices/docker/cloud model, and it's
| great. For many purposes.
|
| But if you are dealing with OS level stuff, talking directly to
| hardware, in the embedded space, etc, now you care about
| different ABIs for performance reason. Now you decide, do I
| want static linking or dynamic linking? The only place I see
| dynamic linking really being a feature is when you have to link
| against the kernel, drivers, or proprietary subsystems (e.g.
| CUDA). IMHO, most userspace application layer stuff shouldn't
| be sharing memory, it should be communicating, again using byte
| streams and serde.
|
| So that basically leaves the hardware/software interface:
| kernel APIs, drivers, etc. If you're in this space, great, it
| makes sense to use dynamic linking with very stable ABIs with
| no generic shenanigans, you don't need them. I think a lot of
| pain of DLL hell comes from using dynamic libs when you really
| should be statically compiling if it's a library, and IPC if
| you need to communicate.
| rrll22 wrote:
| So no surprise Rust is faster.
|
| edit nvm I thought dynamic linking slowed things down since
| static linking means code fits better in the cache. Looking at
| the points, it seems that either dyanmic linking makes code
| faster or people prefer slower dynamic code.
| auggierose wrote:
| And I say you are a low down rusty coder.
| kungito wrote:
| From reading the first few pages I get triggered because the
| whole "where Rust couldn't" feels like mentioning Rust in the
| title to piggy back on Rust's rising popularity. I personally
| don't find Swift interesting in any way while Rust revolutionized
| GCless (and GCless parallelized) typed programming.
| pjmlp wrote:
| If by GCless you mean affine types, the only thing that Rust
| did was bringing affine types to the masses, which is already
| an achievement, agreed-
|
| However in the big context of application programming, and how
| Swift is used on Apple stack, any form of garbage collection
| alongside memory ownership is much more productive.
|
| Long term adoption for Rust will be places like where MISRA-C
| and SPARK are used nowadays.
|
| If I have to spray my code with Arc, Rc and RefCell, I rather
| let the compiler type them for me.
| JoshTriplett wrote:
| I'm one of the leads of the Rust language team, and I love
| seeing writeups like this. When we're designing Rust, we look
| carefully at other languages, to see examples of precedent,
| different trade-offs, and experiences; we don't design in a
| vacuum.
|
| This is an extremely clear explanation of Swift making a
| different set of trade-offs to achieve a different goal, and
| it's worthy of consideration and evaluation. (I read it when it
| was first published.)
|
| There are no attacks against Rust here. This is a professional
| write-up discussing two languages, by someone experienced with
| both.
| gpm wrote:
| Gankra was _heavily_ involved in low level rust plumbing such
| as linking, ABIs, unsafe code, and so on. Comparing to rust is
| both speaking to the other similar language she has experience
| with, and speaking to many of the people who were likely to
| read this article (those people being from the rust community).
| This article also really does achieve the goal in the third
| paragraph (at least when read by the right audience)
|
| > Also some folks like to complain that Rust doesn't bother
| with ABI stability, and I think looking at how Swift does helps
| elucidate why that is.
| millstone wrote:
| Note Gankra is "she" not "he"
| tux3 wrote:
| I care about Rust (mostly due to using it a lot), so I really
| hope more people spend the time to criticize it, and dig up the
| areas that need love and care compared to other languages.
|
| But in this particular case, you should probably know that the
| author is not some nobody trying to piggyback on Rust's
| popularity :) They're a pretty well known contributor, in fact.
| steveklabnik wrote:
| Gankra worked on both Rust and Swift, and additionally wrote
| several very beloved Rust documentation resources, such as
| "Learning Rust With Entirely Too Many Linked Lists" and "The
| Rustinomicon."
|
| This is an extremely accomplished person writing about
| something she is possibly more qualified to than any other
| person on earth.
| saghm wrote:
| I agree that the author is extremely qualified to write about
| this, but personally I also find the title a little jarring,
| mostly due to the word "couldn't". It seems more that there
| are number of tradeoffs where allowing slight runtime costs
| by default (like reference counting) unlocks a lot of stuff
| that's needed to be able to make dynamic linking easier, and
| Rust chose to go a different route than Swift. I understand
| that changing the title to read "where Rust chose to optimize
| for different priorities" won't get as many clicks, but the
| title to me paints the lack of dynamic linking support in
| Rust as some sort of failing rather than an explicit design
| choice. Realistically, I think the fact that Rust and Swift
| picked different routes here is a good thing! Having
| languages which optimize for different things gives people
| more options to pick one that suits their needs better; in a
| world with both nails and screws, it's better to have both
| hammers and screwdrivers available.
| kungito wrote:
| I totally agree. I have a problem only with the wording of
| the title and it shouldn't matter who wrote it. It's a
| matter of integrity and we as software engineers should set
| higher standards to how we write professional articles
| steveklabnik wrote:
| Sure, that the author is an expert does not mean that she
| is infallible, or there's nothing to discuss. That's not
| what this sub-thread is about. This thread is about some
| sort of insinuation that this is born out of some sort of
| weird attempt at Swift to use Rust to gain prominence, or
| something.
| kortex wrote:
| Those were some of the most memorable sections for me. I
| never really thought about the other side of
| monomorphizing. It made it really obvious where "zero-cost
| abstractions" start to fall short.
| moldavi wrote:
| I don't see the problem. Rust made some explicit choices
| that meant it couldn't do something. Doesn't change the
| fact that it couldn't do something. No need to get so
| defensive, lots of languages have things they can't do.
| lehi wrote:
| The experience from a few years ago: _" Then we hit a wall with
| the dynamic linker. At the time you could only link Swift
| libraries dynamically. Unfortunately the linker executed in
| polynomial time so Apple's recommend maximum number of libraries
| in a single binary was 6. We had 92 and counting. As a result It
| took 8-12 seconds after tapping the app icon before main was even
| called. Our shinny new app was slower than the old clunky one.
| Then the binary size problem hit."_ -
| https://twitter.com/StanTwinB/status/1336890442768547845
| judge2020 wrote:
| https://threadreaderapp.com/thread/1336890442768547845.html
| dmit wrote:
| Why post this link? I only have first-hand experience reading
| Twitter threads on PC and phone (official app and web view),
| but in all cases it's just as easy to follow as the page you
| linked. (Except for the Georgia font - the typography is
| consistently better on the official sources.)
|
| The site you linked served me a huge ad. Will the author of
| the tweets ever see a single cent of the revenue my
| hypothetical ad click would have earned them? Of course not.
|
| Let's say I am an investigative reporter, looking for the
| original source of information that I saw in a tweet. Forget
| it, I'm not. Let's say instead that I sometimes like to check
| if I can poke holes in a news story I see during my morning
| scan of HN headlines. I'm sorry, I can't set the bar any
| lower. I think you need to be a corporation to be able to set
| bars lower than this. To my surprise, the site you linked
| _did_ have a direct link to the first tweet of the thread it
| was displaying. It was embedded in the timestamp, styled to
| be shown in a small font, faint gray, no underline. The
| embodiment of "fine print". I was looking specifically for
| it, and it still took a spelunking session in the HTML code
| to figure out where the link was.
|
| The site you linked is bad. Unless you have good reasons to,
| please stop posting links to it. Thanks.
| zapzupnz wrote:
| Such an overreaction. Some people prefer not to read
| Twitter threads as individual posts. This is merely an
| option provided for those people.
|
| No need to get your knickers in a twist over it. The
| original link remains in the original post, as
| clickable/tappable as ever it was.
| dmit wrote:
| > Some people prefer not to read Twitter threads as
| individual posts.
|
| That's great, but Twitter already does that by default.
| Is this about padding between paragraphs (in the CSS
| sense)?
| pengaru wrote:
| The twitter link doesn't work with js disabled unless you
| turn your browser into a bot via user agent switching.
|
| But the thereaderapp link works perfectly fine without any
| js while simultaneously stripping out all the other garbage
| inherent on twitter like advertising, "discussion", or
| "related" content.
| dmit wrote:
| > The twitter link doesn't work with js disabled unless
| you turn your browser into a bot via user agent
| switching.
|
| So you want to read the content. But you can't because
| you disabled JavaScript. But you still can if you change
| the user agent config. But you won't, because ...?
|
| What?
|
| > stripping out all the other garbage inherent on twitter
| like advertising
|
| It still has advertising! Except it advertises for the
| proxy and not for the original source! You've got an ad
| blocker, good for you. Don't go praising one ad-supported
| service ahead of another just because you don't see ads
| on either.
| pengaru wrote:
| I don't have an ad blocker, I simply have js disabled.
| mhh__ wrote:
| If I've read this correctly, D has been able to dynamic link
| properly for years, against it's own ABI, the C ABI, or the C++
| ABI (e.g. templates, vtables up to single inheritance)
| millstone wrote:
| Does D have an ABI spec? I imagine the garbage collector
| complicates it considerably.
| mumblemumble wrote:
| https://dlang.org/spec/abi.html
| mhh__ wrote:
| Its both a blessing and a curse that the garbage collector
| doesn't complicate it all that much
| gok wrote:
| Previously https://news.ycombinator.com/item?id=21488415
___________________________________________________________________
(page generated 2021-02-20 23:01 UTC)