[HN Gopher] Rust's dependencies are starting to worry me
___________________________________________________________________
Rust's dependencies are starting to worry me
Author : chaosprint
Score : 124 points
Date : 2025-05-09 09:11 UTC (13 hours ago)
(HTM) web link (vincents.dev)
(TXT) w3m dump (vincents.dev)
| palata wrote:
| Similar feeling here.
|
| Cargo makes it so simple to add tons of dependencies that it is
| really hard not to do it. But that does not stop here: even if
| _I_ try to be careful with adding dependencies, a couple
| dependencies are likely to pull tens of transitive dependencies
| each.
|
| "Then don't depend on them", you say. Sure, but that means I
| won't write my project, because I won't write those things from
| scratch. I could probably audit the dependency (if it wasn't
| pulling 50 packages itself), but I can't reasonably write it
| myself.
|
| It is different with C++: I can often find dependencies that
| don't pull tens of transitive dependencies in C++. Maybe because
| it's harder to add dependencies, maybe because the ecosystem is
| more mature, I don't know.
|
| But it feels like the philosophy in Rust is to pull many small
| packages, so it doesn't seem like it will change. And that's a
| pity, because I like Rust-the-language better than C++-the-
| language. It just feels like I trade "it's not memory-safe" for
| "you have to pull tons of random code from the Internet".
| jampekka wrote:
| I take bit less unstable dependencies over the total mess of
| C++ dependencies with CMake, shared libraries, version
| conflicts etc any time. There's probably also a bit of an
| illusion about C++ transitive dependencies due to them usually
| being precompiled (because compiling them is such pain).
| ChocolateGod wrote:
| The whole pkgconfig, cmake, autotools etc ecosystem is insane
| compared to how Rust and Go do things.
|
| It's part of the reason why software distribution on Linux
| has been pushed to using containers, removing the point of
| having shared libraries. I think Google with it's C++
| replacement (Carbon) plans on doing it's own system.
| skydhash wrote:
| From my point of view, the issue stems from developers
| wanting to control distribution. Fine if it's for your own
| usage, not really if you're planning for others to use it.
| You will find the most convoluted build system just because
| they have a pet platform they want to specially support
| making it hell to do anything on others.
|
| It could be better, but the current solutions (npm, go,
| python,...) favor only the developers, not the maintainers
| and packagers.
| ChocolateGod wrote:
| There's examples of maintainers/packagers effectively
| sabotaging other peoples projects when making packages
| for distros, whether that's shipping them broken, ancient
| versions etc.
|
| e.g. Bottles, WebkitGTK (distros liked keeping this one
| held back even though doing so is a security risk)
|
| IMHO it shouldn't be the responsibility of the OS vendor
| to package third party applications.
| palata wrote:
| > It's part of the reason why software distribution on
| Linux has been pushed to using containers
|
| My understanding of people distributing their software in
| containers is that they can't be arsed to learn how to do
| it properly. They would install their software and ship the
| entire computer if that was cost effective.
| jampekka wrote:
| What needs to be "learned properly" is sadly a huge pile
| of incoherent legacy cruft that ideally wouldn't be there
| at all.
|
| This is not to denigrate the huge and critical effort
| that makes current computing possible, and that is likely
| unavoidable in the real world. But software distribution
| needs to evolve.
| antonvs wrote:
| > Sure, but that means I won't write my project, because I
| won't write those things from scratch.
|
| You need to think a bit harder about that, to help you decide
| whether your position is rational.
| MeetingsBrowser wrote:
| This confuses me as well. Is the implied solution to choose a
| language where you are forced to write those things from
| scratch?
| palata wrote:
| My point is that if, in the language, everybody is
| incentivise to use fewer dependencies, then a random
| library that I would not write myself (because it is an
| entire project in itself) would have fewer dependencies.
| Because it is not the case, either I take that library and
| accept its transitive dependencies, or I don't have a
| library at all.
|
| In Rust, I'm sometimes actually tempted to wrap a C/C++
| library (and its few dependencies) instead of getting the
| Rust alternative (and its gazillion dependencies).
| palata wrote:
| And you need to think a bit about that (probably not very
| hard), to help you decide whether I'm irrational or whether
| you may not have totally understood my point.
| perrygeo wrote:
| > the philosophy in Rust is to pull many small package
|
| I'm not sure it's a philosophy, more a pragmatic consideration
| for compilation speeds. Anyone who's done a non-trivial amount
| of Rust knows that moment when the project gets too big and
| needs to split into separate crates. It's kinda sad that you
| can't organize code according to proper abstractions, many
| times I feel forced to refactor for compiler performance.
| X0Refraction wrote:
| This was linked from the top comment on the Rust subreddit:
| https://wiki.alopex.li/LetsBeRealAboutDependencies
|
| I think it makes a good point that some of the difference here
| is just perception due to dependencies in C/C++ being less
| immediately visible since they're dynamically loaded. To some
| degree that is a plus though as you likely trust the
| maintainers of your OS distribution to provide stable,
| supported libraries.
|
| As other commenters have said, perhaps this is an area where
| the Rust maintainers could provide some kind of extended
| standard library where they don't guarantee backwards
| compatibility forever, but do provide guarantees about ongoing
| fixes for security issues.
| palata wrote:
| > some of the difference here is just perception due to
| dependencies in C/C++ being less immediately visible since
| they're dynamically loaded.
|
| Not in my case. I manually compile all the dependencies
| (either because I need to cross-compile, or because I may
| need to patch them, etc). So I clearly see _all_ the
| transitive dependencies I need in C++. And I need _a lot
| less_ than in Rust, by a long shot.
| imtringued wrote:
| I have been wasting 6 hours yesterday on getting the bullet
| examples to compile outside of bullet itself with no success.
| It's more likely that a lot of software simply doesn't get
| written because C++ and CMake are a pain in the ass.
| palata wrote:
| I find CMake pretty easy, and I only use a few core features
| from it. Usually the pain comes from completely wrong setups
| by people who didn't learn the basic. But it's true of
| everything, I think.
| stefanos82 wrote:
| Existing discussion in
| https://news.ycombinator.com/item?id=43930640
| demarq wrote:
| > Many call for adding more to the rust standard library much
| like Go
|
| This is the way.
| dralley wrote:
| There should be a second stdlib with relaxed stability
| guarantees. Don't fill the normal stdlib full of cruft that can
| never be changed again.
| demarq wrote:
| Yeah, I agree. Something like the Boost lib for C++
| jerf wrote:
| A strong advantage of that approach is that you don't need
| to be the core Rust team to do it. Anyone who wants to do
| this can just start doing it now.
| shepmaster wrote:
| I agree. Unfortunately, I think that a lot of the people
| who ask for a bigger standard library really just want
| (a) someone else to do the work (b) someone they can
| trust.
|
| The people working on Rust are a finite (probably
| overextended!) set of people and you can't just add more
| work to their plate. "Just" making the standard library
| bigger is probably a non-starter.
|
| I think it'd be great if some group of people took up the
| very hard work to curate a set of crates that everyone
| would use and provide a nice facade to them, completely
| outside of the Rust team umbrella. Then people can start
| using this Katamari crate to prove out the usefulness of
| it.
|
| However, many people wouldn't use it. I wouldn't because
| I simply don't care and am happy adding my dependencies
| one-by-one with minimal feature sets. Others wouldn't
| because it doesn't have the mystical blessing/seal-of-
| approval of the Rust team.
| demarq wrote:
| lets put a price on it
| imtringued wrote:
| This is only an advantage if the core Rust team is
| uncooperative, which is sad rather than something to be
| happy about.
| jerf wrote:
| The "Rust core team" should be working on the "Rust
| core", not every little thing that someone somewhere
| thinks should go in a standard library. It is part of the
| job of a "core team" to say "no".
|
| A _lot_.
|
| Like, a _lot_ a lot a lot. Browse through any programming
| language that has an open issue tracker for all the
| closed proposals sometime. Individually, perhaps a whole
| bunch of good ideas. The union of them? Not so much.
| wofo wrote:
| Actually, a proposal for exactly this was published
| yesterday: https://github.com/rust-lang/rfcs/pull/3810
|
| It's unfortunate that the response so far hasn't been very
| positive
| tadfisher wrote:
| That proposal is not exactly this; that seems to propose a
| "blessed crates" namespace which includes popular open-
| source libraries. I read this proposal as a Python-style
| batteries-included stdlib.
| wofo wrote:
| What the OP proposes is not exactly a bigger stdlib,
| because they mention it should have "relaxed stability
| guarantees". Or is python allowed to change their stdlib
| in backwards-incompatible ways?
| zanecodes wrote:
| The non-standard library, if you will.
| infogulch wrote:
| This is obviously the best solution for Rust. A 'metalibrary'
| library type would add a lot of value to the ecosystem as a
| nexus: - All included crates can be tested
| for inter-compatibility - Release all included crates
| under a single version, simplifying upgrades - Sample
| projects as living documentation to demo integrations and
| upgrades - Breaking changes can be held until all
| affected crates are fixed, then bump all at once - An
| achievable, valuable, local goal for code review / crev
| coverage metrics
|
| There could be general "everything and the kitchen sink"
| metalibraries, metalibraries targeted at particular domains
| or industries, metalibraries with different standards for
| stability or code review, etc. It might even be valuable
| enough to sell support and consulting...
| Alupis wrote:
| So we reinvent Java's bloated SDK again, with all of the
| "javax" packages. What's old is new?
| int_19h wrote:
| Well, it turned out that the alternative is even worse,
| so... let's chalk it down to learning experience.
| SkiFire13 wrote:
| The issue with that is how to get everyone to agree on how that
| would work, e.g. what the criteria for this extension would be,
| what's the policy for future changes, who will maintain all of
| this, etc etc.
| echelon wrote:
| No way. I'd much prefer we have a constellation of core
| companion libraries like Google's Guava.
|
| We do not need to saddle Rust with garbage that will feel dated
| like Python's standard library. Cargo does the job just fine.
| We just need some high quality optional batteries.
|
| Embedded projects are unlikely to need standard library bloat.
| No_std should be top of mind for everyone.
|
| Something that might make additional libraries feel more first
| class: if cargo finally got namespaces and if the Rust project
| took on "@rust/" as the org name to launch officially
| sanctioned and maintained packages.
| pjmlp wrote:
| Python's garbage works everywhere there is a full CPython
| implementation, I see that as an advantage.
| echelon wrote:
| I develop for Linux, Mac, and Windows. Multiple
| architectures and OSes. I rarely see platform issues with
| Rust. It's typically only stuff at the edge, like CUDA
| libraries, that trip up cross-platform builds.
|
| Rust, as a systems language, is quite good at working on a
| variety of systems.
| pjmlp wrote:
| Starts already that Rust won't support architectures not
| available on LLVM, but on GCC, otherwise having a Rust
| frontend project for GCC wouldn't be a thing.
|
| And the systems language remark, I am still looking
| forward when sorting ABI issues for binary libraries is
| finally something that doesn't need to go through
| solutions designed for C and C++.
| bigstrat2003 wrote:
| > We do not need to saddle Rust with garbage that will feel
| dated like Python's standard library.
|
| Python's standard library is a strength, not a weakness. Rust
| should be so lucky. It's wonderful to have basic
| functionality which is guaranteed to be there no matter what.
| Many people work in environments where they can't just YOLO
| download packages from the Internet, so they have to make do
| with whatever is in the stdlib or what they can write
| themselves.
| echelon wrote:
| > Python's standard library is a strength, not a weakness.
| Rust should be so lucky.
|
| Rust is luckier. It has the correct approach. You can find
| every battery you need in crates.io.
|
| Python has had monstrosities like urllib, urllib2, http,
| etc. All pretty much ignored in favor of the external
| requests library and its kin. The standard library also has
| inconsistencies in calling conventions and naming
| conventions and it has to support those *FOREVER*.
|
| The core language should be pristine. Rust is doing it
| right. Everything else you need is within grasp.
| wolvesechoes wrote:
| "Rust is doing it right."
|
| Standard response every time there is some criticism of
| Rust.
| echelon wrote:
| bigstrat2003's argument is approximately "Python is
| batteries included"
|
| My counter argument is that the "batteries included"
| approach tends to atrophy and become dead weight.
|
| Your counter seems to be "that's not an argument, that's
| just Rust hype."
|
| Am I interpreting you correctly? Because I think my
| argument is salient and correct. I don't want to be stuck
| with dated APIs from 20 years of cruft in the standard
| library.
|
| The Python standard library is where modules go to die.
| It has two test frameworks nobody uses anymore, and how
| many XML libraries? Seven? (The correct answer is "four",
| I think. And that's four too many.) The Python standard
| library has so much junk inside, and it can't be safely
| removed or cleaned up.
|
| A standard library should be data structure/collections,
| filesystem/os libraries, and maybe network libraries.
| That's it. Everything else changes with too much
| regularity to be packed in.
| Daishiman wrote:
| Your critique doesn't match the reality of Python users.
|
| There is a single datetime library. It covers 98% of use
| cases. If you want the final 2% with all the bells and
| whistles you can download it if you wish. There is a
| single JSON library. It's fast enough for almost anything
| you want. If you want faster libraries with different
| usability tradeoffs you can use one but I have never felt
| compelled to do so.
|
| Same thing with CSV, filesystem access, DB api, etc.
| They're not _the best_ libraries at the time of any
| script you 're writing, but the reality is that you never
| really need the best, most ergonomic library ever to get
| you through a task.
|
| Because of this, many big complex packages like Django
| have hardly any external dependencies.
|
| If anything you're not the one getting stuck with date
| APIs; it's the Python core devs. Maintainers of other
| packages are always free to choose other dependencies,
| but they almost invariably find that the Python stdlib is
| good enough for everything.
| jaas wrote:
| I don't think an additional standard library layer, whatever
| you call it, has to have the same tight controls on backwards
| compatibility and evolution that the actual standard library
| has. IMO the goal of creating it should be to improve supply
| chain security, not to provide an extremely stable API, which
| might be more of a priority at lower levels but chokes off
| the kind of evolution that will be needed.
|
| I think what you're suggesting is a great idea for a new
| standard library layer, you're just not using that label. A
| set of packages in a Rust namespace, maintained by the same
| community of folks but under policies that comply with best
| practices for security and some additional support to meet
| those best practices. The crates shouldn't be required, so
| no_std should work just as it would prior to such a
| collection.
| procaryote wrote:
| Python's standard library is the main reason python is
| usable.
|
| Python packaging is somehow a 30 year train crash that keeps
| going, but the standard library is good enough that I can do
| most things without dependencies or with very small number of
| them.
| pjmlp wrote:
| Indeed, yes sometimes this brings cruft into the mix.
|
| However I rather have cruft that works everywhere the toolchain
| is fully implemented, instead of playing whack-a-mole with
| third party libraries when only some platforms are supported.
| bigstrat2003 wrote:
| I think that the bare bones stdlib is a huge mistake in Rust. I
| would love to see that rectified. Unfortunately, approximately
| 5 other people share that view. The Rust community as a whole
| is _very_ opposed to adding functionality to std.
| morganherlocker wrote:
| Now instead of seeing millions of lines of inscrutable code in
| _your_ program bloating binary sizes, you can see it in _every_
| program (that doesn 't disable stdlib).
| lantastic wrote:
| In every program that uses a particular feature from the
| stdlib. Given the same feature, I tend to trust stdlib more
| than some rando project. And if you don't trust the stdlib,
| why would you trust the compiler?
| constantcrying wrote:
| That is a serious burden on the maintainers, it creates all
| kinds of different problems, especially if the functionality of
| the libraries assumes a certain execution environment. Rust
| doesn't just target x86 desktops.
| klooney wrote:
| > dotenv is unmaintained.
|
| How much maintenance could you possibly need to load secrets from
| .env into the environment.
| iammrpayments wrote:
| I find hilarious when people judge the quality of a repository
| by how many commits it has, as if 10.000 commits means the code
| is better.
| shepmaster wrote:
| I agree with your general point, but for this specific
| functionality, I'll point out that setting environment
| variables of the current process is unsafe. It took us a long
| time to realize it so the function wasn't actually marked as
| unsafe until the Rust 2024 edition.
|
| What this means in practice is that the call to invoke dotenv
| should also be marked as unsafe so that the invoker can ensure
| safety by placing it at the right place.
|
| If no one is maintaining the crate, that won't happen and
| someone might try to load environment variables at a bad time.
| andy_xor_andrew wrote:
| ok, I'm hooked - how is setting an env var in the current
| process unsafe? My gut says it's not unsafe in a memory-
| ownership sense, but rather in a race condition sense?
|
| whatever the issue is, "setting an env var is unsafe" is so
| interesting to me that I'm now craving a blog post explaining
| this
| Orangeair wrote:
| It's a long standing bug, setenv and unsetenv are not
| thread-safe
|
| https://www.evanjones.ca/setenv-is-not-thread-safe.html
| robertlagrant wrote:
| I honestly think using setenv is just a terrible idea.
| estebank wrote:
| https://doc.rust-lang.org/std/env/fn.set_var.html#safety
| csomar wrote:
| On the other hand loading .env from the environment is critical
| (since you are usually passing secrets through .env). I
| wouldn't want to maintain that myself and not share it with a
| xxK other projects in case there is a vulnerability.
| prophesi wrote:
| The maintainers themselves give this warning in the repo's
| README, so even if it were maintained, it still wouldn't be
| production ready.
|
| > Achtung! This is a v0.* version! Expect bugs and issues all
| around. Submitting pull requests and issues is highly
| encouraged!
|
| https://github.com/dotenv-rs/dotenv
| 0cf8612b2e1e wrote:
| That is an escape hatch that is seemingly used everywhere.
| Nobody wants to release a 1.0 with backwards compatibility
| guarantees.
|
| ZeroVer https://0ver.org/
| im3w1l wrote:
| Ironically a project that hasn't been changed in a while
| "unmaintained" is a good candidate for bumping to v1, while a
| project with new breaking commits every day is a bad
| candidate.
| XxiXx wrote:
| I think it's a "cultural" thing. With Go you often find
| developers/projects proudly mentioning that any or just a few
| non-std dependencies are used. Coming from Go it really feels
| strange when you see pages of dependencies scrolling over your
| screen when you build a Rust project.
| sophacles wrote:
| I have yet to come across a go project that doesn't pull in
| tons of 3rd party code as well. It seems like maybe you're
| over-stating the "culture" a bit.
| meling wrote:
| Yeah, while I've seen some great libraries that follow the
| practice of minimizing their dependencies, I'm a bit annoyed
| with the amount of dependencies that docker will bring along
| [1]. I've been on the lookout for alternatives for my docker
| needs, but the state of podman, buildah and some others that
| I checked is similar. They all bring in roughly the same
| number of dependencies... if anyone knows of a stripped down
| Go lib that can be used to build from a Dockerfile, pull, and
| run a container, I would be grateful for any suggestions.
| Heck docker / moby isn't even using go.mod proper.
|
| [1] https://github.com/moby/moby/blob/master/vendor.mod
| lantastic wrote:
| Wow, that's massive. I guess it's inevitable that a popular
| piece of open-source software for end-users will be
| compelled to accrue dependencies due to popular demand for
| features that require them.
|
| I feel Telegraf made a good compromise: out of the box, it
| comes with a _ton_ of stuff[1] to monitor everything, but
| they make it possible to build only with pieces that you
| need via build tags, and even provide a tool to extract
| said tags from your telegraf config[2]. But lots of supply-
| chain security stuff assume everything in go.mod is used,
| so that can results in a lot of noise.
|
| [1]
| https://github.com/influxdata/telegraf/blob/master/go.mod
| [2] https://github.com/influxdata/telegraf/tree/master/tool
| s/cus...
| hu3 wrote:
| > I have yet to come across a go project that doesn't pull in
| tons of 3rd party code as well.
|
| These have Zero dependencies. It's not rare in Go land.
|
| - https://github.com/go-chi/chi 19k stars
|
| - https://github.com/julienschmidt/httprouter 16k stars
|
| - https://github.com/gorilla/mux 21k stars
|
| - https://github.com/spf13/pflag 2.6k stars
|
| - https://github.com/google/uuid 5.6k starts
|
| Many others have just a few dependencies.
| api wrote:
| Go has a fatter standard library and a "fat" runtime with
| built-in green threads (an asynchronous runtime basically) and
| garbage collection, so you get more out of the box and thus end
| up using fewer dependencies.
| neilv wrote:
| In the past (not in Rust, but other languages), for important
| systems, I've instituted policies of minimizing dependencies from
| these language-specific package repositories, and for the ones
| you do use, having to copy it to our own repos and audit each
| update before use.
|
| But that's not practical for all situations. For example, Web
| frontend developer culture might be the worst environment, to the
| point you often can't get many things done in feasible time, if
| you don't adopt the same reckless practices.
|
| I'm also seeing it now with the cargo-culting of opaque self-
| hosted AI tools and models. For learning and experimenting, I'd
| spend more time sufficiently compartmentalizing an individual
| tool than with using it.
|
| This weekend, I'm dusting off my Rust skills, for a small open
| source employability project (so I can't invest in expensive
| dependency management on this one). The main thing thing
| bothering me isn't allocation management, but the sinking feeling
| when I watch the cast-of-thousands explosion of transitive
| dependencies for the UI and async libraries that I want to use.
| It's only a matter of time before one of those is compromised, if
| not already, and one is all it takes.
| wofo wrote:
| There are some voices trying to address this security risk
| (e.g. the proponents of this new RFC: https://github.com/rust-
| lang/rfcs/pull/3810). However, for some reason (probably
| culture) there isn't much momentum yet to change the status
| quo.
| cogman10 wrote:
| The rust RFC process has, frankly, become somewhat of a CF.
|
| There's literally 1000s of RFCs for rust with only a small
| handful that are integrated. Having this forest, IMO, makes
| it hard for any given proposal to really stand out. Further,
| it makes duplicate effort almost inevitable.
|
| Rust's RFC process is effectively a dead letter box for most.
| geodel wrote:
| I think they can constitute committee for RFC review
| process(in case there is none today) and based on
| recommendation multiple domain specific teams/ groups can
| be created to review RFCs in timely manner.
| dathinab wrote:
| > isn't much momentum yet to change the status quo.
|
| it's complex problem with tons of partial solutions which
| each have tons of ways to implement them with often their no
| being a clear winner
|
| i.e. it's the kind of hard to solve by consensus problem
|
| e.g. the idea of a extended standard library is old (around
| since the beginning of rust) but for years it was believed
| it's probably the best to make it a separate independent
| project/library for various reason. One being that the saying
| "the standard library is the place where code goes to die"
| has been quite true for multiple ecosystems (most noticeably
| python)
|
| as a side note ESL wouldn't reduce the LOC count it would
| increase it as long as you fully measure LOCs and not "skip"
| over some dependencies
| pjmlp wrote:
| Best way is to have CI/CD systems only connected to the
| official internal repos.
|
| Devs can add whatever they feel like on their workstations but
| it will be a sad build server if they get pushed without
| permission.
| dsr_ wrote:
| s/Best way/The only safe way/
|
| Anything else will get abused in the name of expediency and
| just-this-one-time.
|
| Also, the process for adding a crate/gem/module/library needs
| to be the same as anything else: license review, code review,
| subscription to the appropriate mailing list or other
| announce channel, and assignment of responsibility. All of
| these except code review can be really, really fast once you
| have the process going.
|
| All problems are, at least in part, dependency chain
| management problems.
| sunrunner wrote:
| I agree that some amount of friction when including third
| party dependencies is a vital thing to push people to
| consider the value versus cost of dependencies (and license
| review, code review, channel subscriptions are all
| incredibily important and almost always overlooked),
| however how should this work for transitive
| dependendencies? And the dependencies of _those_
| dependencies?
|
| The dependency trees for most interpreted or source-
| distributed languages are ridiculous, and review of even a
| few of those seems practically impossible in a lot of
| development environments.
| MeetingsBrowser wrote:
| > Devs can add whatever they feel like on their workstations
|
| A compromised dev machine is also a problem.
| pjmlp wrote:
| True, hence we can go next level and also deal with limited
| accounts for developers, and I can tell you most folks on
| HN would hate to work in such corporate environments.
| eddd-ddde wrote:
| The cool thing about rust is you can implement async yourself.
| You aren't tied to any specific implementation.
| dboreham wrote:
| Or not use async at all.
| pjmlp wrote:
| Same in C++, partially true in .NET/C# and F#.
| thrance wrote:
| I had the same concerns when I started using Rust, but then I
| eventually embraced it, for better or worse. Cargo makes it so
| your build almost never breaks (it's happened maybe twice for the
| 8 years I've been doing Rust). Plus there are still way less
| vulnerabilities with Rust projects than non-Rust projects, in
| spite of the crazy number of dependencies.
|
| If I was to design a Rust 2.0, I'd make it so dependencies need
| permissions to access IO, or unsafe code, etc.
| csomar wrote:
| > when checking a rust security advisory mentioning that dotenv
| is unmaintained
|
| This is a problem with all languages and actually an area where
| Rust shines (due to editions). Your pulled in packages will
| compile as they previously did. This is not true for _garbage
| collected_ languages (pun intended).
|
| > Out of curiosity I ran toeki a tool for counting lines of code,
| and found a staggering 3.6 million lines of rust .... How could I
| ever audit all of that code?
|
| Again, another area where Rust shines. You _can_ audit and most
| importantly modify the code. This is not that easy if you were
| using Nodejs where the runtimes are behind node /v8 or whatever.
| You compile these things (including TLS) yourself and have full
| control over them. That's why Tokio is huge.
| lolinder wrote:
| > This is not true for garbage collected languages
|
| JavaScript is backwards compatible going back effectively
| forever, as is Java. Rust's unique system is having a way to
| _make_ breaking changes to the language without breaking old
| code, not that they prioritize supporting old code
| indefinitely.
|
| The _libraries_ are a different story--you 're likely to have
| things break under you that rely on older versions of libraries
| when you update--but I don't see Rust actually having solved
| that.
|
| > You can audit and most importantly modify the code. This is
| not that easy if you were using Nodejs where the runtimes are
| behind node/v8 or whatever.
|
| Node and V8 are open source, which makes the code just as
| auditable and modifiable as the 3.6 million lines of Rust.
| Which is to say, both are equally unapproachable.
| csomar wrote:
| > The libraries are a different story--you're likely to have
| things break under you that rely on older versions of
| libraries when you update--but I don't see Rust actually
| having solved that.
|
| No language can fix that. However, I've lost count of the
| times my Python/JavaScript interpretation fails because of
| something in one of the dependencies. Usually, it's not a
| JS/Python problem but rather has to do with a Node/Python
| version update. It always boils down to the "core" issue
| which is the runtime. That's why I like that Rust give me a
| "fixed" runtime that I download/compile/package with my
| program.
|
| > Node and V8 are open source, which makes the code just as
| auditable and modifiable as the 3.6 million lines of Rust.
| Which is to say, both are equally unapproachable.
|
| I've recently patched a weird bug under Tokio/Otel and can't
| imagine doing that with Node/V8 without it being a major
| hassle. It is relatively straightforward in Rust though
| requires maintaining your own fork of _only_ the dependency
| /branch in question.
| jerf wrote:
| A true enough statement, but "Rust" is unnecessarily specific.
| Dependencies are getting scary in general. Supply chain attacks
| are no longer hypothetical, they're here and have been for a
| while.
|
| If I were designing a new language I think I'd be very interested
| in putting some sort of capability system in so I can confine
| entire library trees safely, and libraries can volunteer somehow
| what capabilities they need/offer. I think it would need to be a
| new language if for no other reason than ecosystems will need to
| be written with the concept in them from the beginning.
|
| For instance, consider an "image loading library". In most modern
| languages such libraries almost invariably support loading images
| from a file, directly, for convenience if nothing else. In a
| language that supported this concept of capabilities it would be
| necessary to support loading them from a stream, so either the
| image library would need you to supply it a stream
| unconditionally, or if the capability support is more rich, you
| could say "I don't want you to be able to load files" in your
| manifest or something and the compiler would block the
| "LoadFromFile(filename)" function at compile time. Multiply that
| out over an entire ecosystem and I think this would be hard to
| retrofit. It's hugely backwards incompatible if it is done
| correctly, it would be a _de facto_ fork of the entire ecosystem.
|
| I honestly don't see any other solution to this in the long term,
| except to create a world where the vast majority of libraries
| become untargetable in supply chain attacks because they can't
| open sockets or read files and are thus useless to attackers, and
| we can reduce our attack surface to just the libraries that truly
| need the deep access. And I think if a language came out with
| this design, you'd be surprised at how few things _need_ the
| dangerous permissions.
|
| Even a culture of minimizing dependencies is just delaying the
| inevitable. We've been seeing Go packages getting supply-chain-
| attacked and it getting into people's real code bases, and that
| community is about as hostile to large dependency trees as any
| can be and still function. It's not good enough.
| wofo wrote:
| I've thought about this (albeit not for that long) and it seems
| like you'd need a non-trivial revamp of how we communicate with
| the operating system. For instance, allowing a library to "read
| from a stream" sounds safe until you realize they might be
| using the same syscalls as reading from a file!
| assassinator42 wrote:
| Java and the .NET Framework had partial trust/capabilities
| mechanisms decades ago. No one really used them and they were
| deprecated/removed.
| pjmlp wrote:
| It was more like no one used them _correctly_.
| eikenberry wrote:
| Wouldn't that mean they were poorly implemented. If no one
| uses something correctly, seems like that isn't a problem
| with the people but the thing.
| palata wrote:
| I don't think so. Software is maybe the only
| "engineering" discipline where it is considered okay to
| use mainstream tools incorrectly and then blame the
| tools.
| voxgen wrote:
| I don't think retrofitting existing languages/ecosystems is
| necessarily a lost cause. Static enforcement requires rewrites,
| but runtime enforcement gets you most of the benefit at a much
| lower cost.
|
| As long as all library code is compiled/run from source, a
| compiler/runtime can replace system calls with wrappers that
| check caller-specific permissions, and it can refuse to compile
| or insert runtime panics if the language's escape hatches would
| be used. It can be as safe as the language is safe, so long as
| you're ok with panics when the rules are broken.
|
| It'd take some work to document and distribute capability
| profiles for libraries that don't care to support it, but a
| similar effort was proven possible with TypeScript.
| 0cf8612b2e1e wrote:
| Is there anything in existence which has a version of this
| idea? It makes a ton of sense to me, but you are right that it
| would be practically impossible to do in a current language.
| Smaug123 wrote:
| Austral, for example? https://austral-
| lang.org/spec/spec.html#rationale-cap
| kibwen wrote:
| Yes, but you can't enforce this at the language level if your
| objective is security (at least not for natively-compiled
| languages). You need OS-level support for capabilities, which
| some OSes do provide (SeL4, Fuchsia). But if you're in a VM
| rather than native code then you can enforce capabilities,
| which is what Wasm does with WASI.
| metaltyphoon wrote:
| .NET Framework, windows only, (non .NET, aka .NET Core)
| martsa1 wrote:
| Wasm + wasi let you define hard boundaries between components
| with explicit interfaces, might be loosely along these lines?
| cyberax wrote:
| TypeScript ecosystem supports this! An environment without e.g.
| file operations will simply miss classes that are needed for
| it, and your compilation will fail.
| eikenberry wrote:
| > I think it would need to be a new language [..]
|
| Language _s_ (plural) ... no single language will work for
| everyone.
| srikanth767 wrote:
| True
| aliceryhl wrote:
| I'm quite careful to tightly control the dependencies of Tokio.
| All dependencies are under control by members of the Tokio team
| or others that I trust.
| schmichael wrote:
| We need a term like "Mature" or similar for dependencies that are
| done. Mature dependencies have two characteristics:
|
| 1. Well defined scope
|
| 2. Infrequent changes
|
| Nomad has many of these (msgpack, envparse, cli, etc). These
| dependencies go years without changing so the dependency
| management burden rapidly approaches zero. This is an especially
| useful property for "leaf" dependencies with no dependencies of
| their own.
|
| I wish libraries could advertise their intent to be Mature. I'd
| choose a Mature protobuf library over one that constantly tweaked
| its ergonomics and performance. Continual iterative improvement
| is often a boon, but sometimes it's not worth the cost.
| delusional wrote:
| I have a lot of sympathy for this viewpoint, but I also ask
| that we try to remind ourselves. We are asking for
| professionalism from hobby projects.
|
| If you want a mature protobuf implementation you should
| probably buy one. Expecting some guy/gal on the internet to
| maintain one for your for free seems ill advised.
| schmichael wrote:
| A great point! All of the libraries I mentioned are created
| and maintained by corporations. Hobbyists, as always, are
| free to do as they please without judgement from me. :)
|
| I will say I get great satisfaction from the little envparse
| library I wrote needing near-0 maintenance. It's a rare treat
| to be able to consider any project truly done.
| procaryote wrote:
| Isn't that an argument _for_ having a "mature" label? To
| avoid the hobbyists who have no intention to maintain their
| thing?
|
| Also there are lots of lovely projects maintained at high
| levels by hobbyists, and plenty of abandonware that was at
| some point paid for
| pclmulqdq wrote:
| > I have a lot of sympathy for this viewpoint, but I also ask
| that we try to remind ourselves. We are asking for
| professionalism from hobby projects.
|
| Nobody is asking for professional quality standards from
| hobby projects. At best, they are asking for hobby projects
| to advertise themselves as such, and not as "this is a
| library for [x] that you can use in your stuff with the
| expectations of
| [maintenance/performance/compatibility/etc.]."
|
| Resume-driven development seems to cause people to oversell
| their hobby projects as software that is ready to have
| external users.
|
| > If you want a mature protobuf implementation you should
| probably buy one
|
| No software is ever developed this way. For some reason,
| libraries are always free. Approximately nobody will buy paid
| libraries.
| procaryote wrote:
| Java did this sometimes by essentially adding slightly tidied
| up versions of whatever was the de-facto standard to the
| standard library. Java 1.3 didn't have regexes but most people
| were using the same apache commons thing, so java 1.4 added
| regexes that looked exactly like that. Java's date handling was
| a pain so people mostly used joda-date; a later java version
| added something that mostly works like jodadate. Etc.
|
| It is an easy way to get a somewhat OK standard library as the
| things you add became popular on their own merits at some
| point.
|
| Once added, the lowest friction path is to just use the
| standard library; and as it is the standard library you have a
| slightly better hope someone will care to maintain it. You can
| still build a better one if needed for your use-case, but the
| batteries are included for basic usage
| nemothekid wrote:
| I feel like leftpad has given package managers a very bad name. I
| understand the OP's hesitation, but it feels a little ridiculous
| to me.
|
| tokio is a work-stealing, asynchronous runtime. This is a feature
| that would be an _entire language_. Does OP consider it
| reasonable to audit the entire Go language? or the V8 engine for
| Node? v8 is ~10x more lines than tokio.
|
| If Cloudflare uses Node, would you expect Cloudflare to audit v8
| quarterly?
| timewizard wrote:
| If two different dependencies use a different version of some
| other dependency between them does cargo still include both
| versions by default?
|
| This is something I've only ever seen cargo do.
| metaltyphoon wrote:
| > If two different dependencies use a different version of
| some other dependency between them does cargo still include
| both versions by default?
|
| No, cargo will resolve using sem ver compatibility and pick
| the best version. Nuget, for C# does something very similar.
| conradludgate wrote:
| And for what it's worth, people do audit tokio. I have audited
| tokio. Many times in fact. Sure, not everyone will, but someone
| will :)
| righthand wrote:
| Everyone is in such a rush to get their project out the door, no
| one has time to generate a key and properly code sign releases
| and begin developing a more secure chain. Now we have JS package
| "whatever code" ecosystem but for Rust. As if we haven't watched
| NPM get hacked many times over the last decade or so.
| mcflubbins wrote:
| > Everyone is in such a rush to get their project out the door
|
| This is the cause of so many issues.
|
| And its not like we're at war or trying to cure the next
| pandemic, we're writing CRUD apps and trying to convince people
| to click on adds for crap they don't need.
| gxt wrote:
| You can audit your dependencies for crates with security
| vulnerabilities reported to the RustSec Advisory Database, also
| block unmaintained crates, and enforce your license requirements
| using SPDX expressions with cargo-audit and cargo-deny.
|
| You can ensure that third-party Rust dependencies have been
| audited by a trusted entity with cargo-vet.
|
| And you should have taken a look at where those 3M locs come
| from, it's usually from Microsoft's windows-rs crates that are
| transitively included in your dependencies through default
| features and build targets of crates built to run on windows.
| 1vuio0pswjnm7 wrote:
| "Not thinking about package management careful makes me sloppy."
|
| Isn't the point of a memory safe language to allow programmers to
| be sloppy without repercussions, i.e., to not think about
| managing memory and even to not understand how memory works.
|
| Would managing dependencies be any different. Does Rust allow
| programmers to avoid thinking carefully about selecting
| dependencies.
| sophacles wrote:
| > Isn't the point of a memory safe language to allow
| programmers to be sloppy without repercussions, i.e., to not
| think about managing memory and even to not understand how
| memory works
|
| No. The point is even the best programmers of unsafe languages
| regularly introduce both simple and subtle bugs into codebases
| while being careful about handling memory correctly, and
| therefore we should use languages that don't even allow those
| bugs for most every use case. Using these languages still
| allows crap programmers to waste GBs of correctly allocated and
| handled memory, and good programmers to write tight, resouce-
| sipping code.
|
| Dependencies are orthogonal to this.
| 1vuio0pswjnm7 wrote:
| If careful programmers who can manage memory should use the
| same language as careless ones who cannot, then does this mean
| both should also automatically use third party libraries by
| default.
|
| Are there systems languages that provide memory management but
| do not default to using third party libraries. If yes, then do
| these languages make it easier for programmers to avoid
| dependencies.
| empath75 wrote:
| No, the point is to stop you from being sloppy. The code won't
| compile if you're sloppy with memory management.
|
| You can be _relatively_ sure that you're not introducing memory
| unsafety by adding a dependency, but you can't be sure that it
| isn't malware unless you audit it.
| timewizard wrote:
| > to be sloppy without repercussions
|
| It's the difference between a wet mess and a dry one. Rust
| creates dry messes. It's still a mess.
| rs186 wrote:
| I once wanted to contribute to the popular swc project
| (https://github.com/swc-project/swc). I cloned the repo, ran
| build, and a whooping 20GB was gone from my disk. The parser
| itself (https://github.com/swc-
| project/swc/blob/main/crates/swc_ecma...) has over a dozen
| dependencies, including serde.
|
| Meanwhile, the heaviest JavaScript parser implemented in
| JavaScript is more lightweight.
|
| I decided that I should leave this project alone and spend my
| time elsewhere.
| MeetingsBrowser wrote:
| I agree that relying on unknown dependencies is a risk, but
| this misses the point IMO. Number of dependencies and disk
| space are kind of arbitrary.
|
| > Meanwhile, the heaviest JavaScript parser implemented in
| JavaScript is more lightweight.
|
| The lightest weight javascript program relies on V8 to run,
| which has multiple orders of magnitude more dependencies. Most
| of which you have never heard of.
|
| At least cargo makes it easier to get a clearer picture of what
| the dependencies are for a program.
| munificent wrote:
| _> relies on V8 to run, which has multiple orders of
| magnitude more dependencies._
|
| Actually, this isn't true. (Or at least wasn't a while back.)
| I used to work with a bunch of ex-V8 folks and they really
| despised third-party dependencies and didn't trust any code
| they didn't write. They used a few third-party libs but for
| them most part, they tried to own everything themselves.
| pixl97 wrote:
| Number of dependencies isn't exactly arbitrary...
|
| If you have one huge dep it's easier to keep track you're on
| the latest update, also it's much less likely you'll fat
| finger it and import something typosquatting.
|
| Also if you're in enterprise you'll have less 100 page SBOM
| reports.
| rs186 wrote:
| No, it has very little to do with v8 or any runtime. Those
| parsers run on any decent and recent enough runtime,
| including browsers and Node.js. If you look at the actual
| code, they use basic APIs in the JavaScript language that you
| can find in almost any other language.
| constantcrying wrote:
| I am counting 13 dependencies, the rest are internal ones. Are
| any of these superfluous or only needed for small edge cases?
| Serde seems exactly a case where you absolutely should use an
| external dependency.
|
| Also, repository size seems an extremely irrelevant metric.
| rs186 wrote:
| 13 > 12 so over a dozen dependencies. If you look at acorn or
| babel/parser, they barely have any dependency.
|
| Repository size is directly related to how long it takes to
| run a build, which is extremely important if I were to
| contribute to the project.
|
| > Serde seems exactly a case where you absolutely should use
| an external dependency.
|
| I can't see any reason a parser has a hard dependency on a
| serialization library.
| constantcrying wrote:
| >13 > 12 so over a dozen dependencies. If you look at acorn
| or babel/parser, they barely have any dependency.
|
| Which ones are superfluous?
|
| There are good reasons to use dependencies. If someone has
| solved a problem you need to solve as well it is pointless
| to duplicate the effort.
|
| >Repository size is directly related to how long it takes
| to run a build, which is extremely important if I were to
| contribute to the project.
|
| Totally false. There is zero inherent relation.
|
| >I can't see any reason a parser has a hard dependency on a
| serialization library.
|
| And because you can't see a reason there is none?
|
| It is totally meaningless to talk about any of this if you
| can not point out why this is superfluous.
| rs186 wrote:
| I don't think there is any point in debating this,
| because apparently you are in the camp of "dependencies
| are ok", with or without a good reason, when a different
| camp is "avoid dependencies unless you really have to".
| You just provided an example of why dependencies explode
| like this.
|
| > And because you can't see a reason there is none?
|
| Somehow every other JS based parser doesn't do fancy
| serialization, as far as I can tell. You can come up with
| reasons of why one might need it, but as a user of the
| parser, I want the footprint to be small, and that's a
| requirement. In fact, that's one of the reasons I never
| used swc parser in my serious projects.
| Orangeair wrote:
| I think that https://blessed.rs does a pretty good job of
| providing recommendations for things that probably can't be
| crammed into the standard library, but which you'll almost
| certainly end up needing at one point or another. I honestly like
| that system a lot, it makes it so that the only packages you need
| to worry much about are usually doing something rather specific.
| kion wrote:
| IMO any system where taking a dependency is "easy" and there is
| no penalty for size or cost is going to eventually lead to a
| dependency problem. That's essentially where we are today both in
| language repositories for OSS languages and private monorepos.
|
| This is partly due to how we've distributed software over the
| last 40 years. In the 80s the idea of a library of functionality
| was something you paid for, and painstakingly included parts of
| into your size constrained environment (fit it on a floppy). You
| probably picked apart that library and pulled the bits you
| needed, integrating them into your builds to be as small as
| possible.
|
| Today we pile libraries on top of libraries on top of libraries.
| Its super easy to say `import foolib`, then call
| `foolib.do_thing()` and just start running. Who knows or cares
| what all 'foolib' contains.
|
| At each level a caller might need 5% of the functionality of any
| given dependency. The deeper the dependency tree gets the more
| waste piles on. Eventually you end up in a world where your
| simple binary is 500 MiB of code you never actually call, but all
| you did was take that one dependency to format a number.
|
| In some cases the languages make this worse. Go and Rust, for
| example, encourage everything for a single package/mod to go in
| the same file. Adding optional functionality can get ugly when it
| would require creating new modules, but if you only want to use a
| tiny part of the module, what do you do?
|
| The only real solution I can think of to deal with this long term
| is ultra-fine-grained symbols and dependencies. Every function,
| type, and other top-level language construct needs to declare the
| set of things it needs to run (other functions, symbols, types,
| etc). When you depend on that one symbol it can construct, on
| demand, the exact graph of symbols it needs and dump the rest for
| any given library. You end up with the minimal set of code for
| the functionality you need.
|
| Its a terrible idea and I'd hate it, but how else do you address
| the current setup of effectively building the whole universe of
| code branching from your dependencies and then dragging it around
| like a boat anchor of dead code.
| nicoburns wrote:
| As far as I'm aware, LTO completely solves this from a binary
| size perspective. It will optimise out anything unused. You can
| still get hit from a build time perspective though.
| kion wrote:
| LTO only gets you so far, but IMO its more kicking the can
| down the road.
|
| The analogy I use is cooking a huge dinner, then throwing out
| everything but the one side dish you wanted. If you want just
| the side-dish you should be able to cook just the side-dish.
| 01HNNWZ0MV43FF wrote:
| Then another group of armchair programmers will bitch you
| out for using small dependencies
|
| I just don't listen. Things should be easy. Rust is easy.
| Don't overthink it
| floating-io wrote:
| I see it more as having a sizable array of ingredients in
| the pantry, and using only what you need or want for a
| given meal.
| samus wrote:
| It's certainly better than in Java where LTO is simply not
| possible due to reflection. The more interesting question is
| which code effectively gets compiled so you know what has to
| be audited. That is, without disassembling the binary. Maybe
| debug information can help?
| metaltyphoon wrote:
| Doesn't Java offer some sort of trimming like C#? I know he
| won't remove everything but at least they can trim down a
| lot of things.
| pjmlp wrote:
| Yes, jlink, code guard, R8/D8 on Android, if you want to
| stay at the bytecode level, plus all the commercial AOT
| compilers and the free beer ones, offer similar
| capabilities at the binary level.
| pjmlp wrote:
| Not only it is possible, it has been available for decades
| on commercial AOT compilers like Aonix, Excelsior JET, PTC,
| Aicas.
|
| It is also done on the cousin Android, and available as
| free beer on GraalVM and OpenJ9.
| 0x696C6961 wrote:
| In Go, the symbol table contains enough information to
| figure this out. This is how
| https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck is
| able to limit vulnerabilities to those that are actually
| reachable in your code.
| poincaredisk wrote:
| "completely solves" is a bit of an overstatement. Imagine a
| curl-like library that allows you to make requests by URL.
| You may only ever use HTTP urls, but code for all the other
| schemas (like HTTPS, FTP, Gopher) needs to be compiled in as
| well.
|
| This is an extreme example, but the same thing happens very
| often at a smaller scale. Optional functionality can't always
| be removed statically.
| api wrote:
| That's a consequence of crufty complicated protocols and
| standards that require a ton of support for different
| transports and backward compatibility. It's hard to avoid
| if you want to interoperate with the whole world.
| dathinab wrote:
| yes, it's not a issue of code size but a issue of supply
| chain security/reviewability
|
| it's also not always a fair comparison, if you include tokio
| in LOC counting then you surely would also include V8 LOC
| when counting for node, or JRE for Java projects (but not
| JDK) etc.
| xlii wrote:
| > Go and Rust, for example, encourage everything for a single
| package/mod to go in the same file.
|
| Clarification: Go allows for a very simple multi-file. It's one
| feature I really like, because it allows splitting otherwise
| coherent module into logical parts.
| dcow wrote:
| Further: I've never seen rust encourage anything of the sort.
| Module directory with a mod.rs and any number of files works
| just fine.
| kion wrote:
| I probably mischaracterized this as its been a while since
| I did more than trivial Rust. AFAIK its not possible to
| depend on only a part of a module in Rust though right? (At
| least without an external build system)
|
| For example, you can't split up a module into foo.rs
| containing `Foo` and bar.rs containing `Bar`, both in
| module 'mymod' in such a way that you can `use mymod::Bar
| and foo.rs is never built/linked.
|
| My point is the granularity of the package/mod encourages
| course-grained deps, which I argue is a problem.
| eddd-ddde wrote:
| You'd use feature flags to enable certain parts of the
| library.
| dathinab wrote:
| > not possible to depend on only a part of a module in
| Rust though right
|
| yesn't, you can use feature flags similar to `#if` in C
|
| but it's also not really a needed feature as dead code
| elimination will prune out all code functions, types,
| etc. you don't use. Non of it will end up in the produced
| binary.
| tialaramex wrote:
| Yeah, likewise Rust is completely fine after you say `mod
| foo` and have a file named foo.rs, if you also make a foo/
| directory and put foo/whatever.rs and foo/something_else.rs
| that those are all part of the foo module.
|
| Historically Rust wanted that foo.rs to be renamed foo/mod.rs
| but that's no longer idiomatic although of course it still
| works if you do that.
| dathinab wrote:
| to extend on this:
|
| in rust crates are semantically one compilation unit (where
| in C oversimplified it's a .h/.c pair, and practically
| rustc will try to split it in some more units to speed up
| build time).
|
| the reason I'm pointing this out is because many sources of
| "splitting a module across files" come from situations
| where 1 file is one compilation unit so you needed to have
| a way to split it (for organization) without splitting it
| (for compilation) in some sitation
| dietr1ch wrote:
| I don't think libraries are the problem, but we don't have a
| lot of visibility after we add a new dependency. You either
| take the time to look into it, or just add it and then forget
| about the problem (which is kind of the point of having small
| libraries).
|
| It should be easy to build and deploy profiling-aware builds
| (PGO/BOLT) and to get good feedback around time/instructions
| spent per package, as well as a measure of the ratio of each
| library that's cold or thrown away at build time.
| taeric wrote:
| I agree that I don't like thinking of libraries as the
| problem. But they do seem to be the easiest area to point at
| for a lot of modern development hell. Is kind of crazy.
|
| I'll note that it isn't just PGO/BOLT style optimizations.
| Largely, it is not that at all, oddly.
|
| Instead, the problem is one of stability. In a "foundation
| that doesn't move and cause you to fall over" sense of the
| word. Consider if people made a house where every room had a
| different substructure under it. That, largely, seems to be
| the general approach we use to building software. The idea
| being that you can namespace a room away from other rooms and
| not have any care on what happens there.
|
| This gets equally frustrating when our metrics for
| determining the safety of something largely discourages
| inaction on any dependencies. They have to add to it, or
| people think it is abandoned and not usable.
|
| Note that this isn't unique to software, mind. Hardware can
| and does go through massive changes over the years. They have
| obvious limitations that slow down how rapidly they can
| change, of course.
| throwaway462663 wrote:
| > It's a terrible idea...
|
| It's a terrible idea because you're trying to reinvent section
| splitting + `--gc-sections` at link time, which rust (which the
| article is about) already does by default.
| kion wrote:
| The article is about Rust, but I was commenting on
| dependencies in general.
|
| Things like --gc-sections feels like a band-aid, a very
| practical and useful band-aid, but a band-aid none the less.
| You're building a bunch of things you don't need, then
| selectively throwing away parts (or selectively keeping
| parts).
|
| IMO it all boils down to the granularity. The granularity of
| text source files, the granularity of units of distribution
| for libraries. It all contributes to a problem of large
| unwieldy dependency growth.
|
| I don't have any great solutions here, its just observations
| of the general problem from the horrifying things that happen
| when dependencies grow uncontrolled.
| kibwen wrote:
| _> In some cases the languages make this worse. Go and Rust,
| for example, encourage everything for a single package /mod to
| go in the same file._
|
| What? I don't know about Go, but this certainly isn't true in
| Rust. Rust has great support for fine-grained imports via
| Cargo's ability to split up an API via crate features.
| KennyBlanken wrote:
| What did you expect from someone who thinks that merely
| including a bunch of library header files but only calling
| one small function results in hundreds of megabytes of
| compiled binary code for functions that are never used, and
| all the functions those functions use?
| SamuelAdams wrote:
| This idea is already implemented in Dotnet, with Trimming and
| now ahead of time compilation (AOT). Maybe other languages can
| learn from dotnet?
|
| https://learn.microsoft.com/en-us/dotnet/core/deploying/trim...
|
| https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...
| CBLT wrote:
| Those are done at compile time. Many languages (including
| Rust, which this story is about) also remove unused symbols
| at compile time.
|
| The comment you're replying to is talking about not pulling
| in dependencies at all, before compiling, if they would not
| be needed.
| dathinab wrote:
| dead code elimination is a very old shoe
|
| which get reinvented all the time, like in dotnet with
| "trimming" or in JS with "tree-shaking".
|
| C/C++ compiler have been doing that since before dot net was
| a thing, same for rust which does that since it's 1.0 release
| (because it's done by LLVM ;) )
|
| The reason it gets reinvented all the time is because while
| it's often quite straight forward in statically compiled
| languages it isn't for dynamic languages as finding out what
| actually is unused is hard (for fine grained code
| elimination) or at lest unreliable (pruning submodules). Even
| worse for scripting languages.
|
| Which also brings use to one area where it's not out of the
| box, if you build .dll/.so in one build process and then use
| them in another. Here additional tooling is needed to prune
| the dynamic linked libraries. But luckily it's not a common
| problem to run into in Rust.
|
| In general most code size problems in Rust aren't caused by
| too huge LOC of dependencies but by an overuse of
| monopolization. The problem of tons of LOC in dependencies is
| one of supply chain trust and review ability more then
| anything else.
| zozbot234 wrote:
| > In the 80s the idea of a library of functionality was
| something you paid for, and painstakingly included parts of
| into your size constrained environment (fit it on a floppy).
| You probably picked apart that library and pulled the bits you
| needed, integrating them into your builds to be as small as
| possible.
|
| If anything, the 1980s is when the idea of fully reusable,
| separately-developed software components first became
| practical, with Objective-C and the like. In fact it's a
| significant success story of Rust that this sort of pervasive
| software componentry has now been widely adopted as part of a
| systems programming language.
| jiggawatts wrote:
| A really important consideration that is often overlooked is
| that the waste accumulates _exponentially!_
|
| If each layer of "package abstraction" is only 50% utilised,
| then each layer multiplies the total size by 2x over what is
| actually required by the end application.
|
| Three layers -- apps that pull in packages pulling in packages
| -- already gets you to 88% bloat! (Or just 12% useful code)
|
| An example of this is the new Windows 11 calculator that can
| take several seconds to start because it loads junk like the
| Windows 10 Hello for Business account recovery helper library!
|
| Why? Because it has currency conversion, which uses a HTTP
| library, which has corporate web proxy support, which needs
| authentication, which needs WH4B account support, which can get
| locked out, which needs a recovery helper UI...
|
| ...in a calculator. That you can't launch unless _you have
| already logged in successfully_ and is definitely not the
| "right place" for account recovery workflows to be kicked off.
|
| But... you see... it's just _easier_ to package up these things
| and include them with a single line in the code somewhere.
| aeonik wrote:
| if only we had a system that we could all operate on with a
| standard set of tools that would take care of shared resource
| access like this.
| KennyBlanken wrote:
| I can't remember the last time I saw someone so conclusively
| demonstrate they know nothing about the basics of how
| libraries, compilers, and linkers work.
| ruraljuror wrote:
| The actual behavior of go seems much closer to your ideal
| scenario than what you attribute to it. Although it is more
| nuanced, so both are true. In go, a module is a collection of
| packages. When you go get a module, the entire module is pulled
| onto the host, but when you vendor only the packages you use
| (and i believe only the symbols used from that package, but am
| not certain) are vendored to your module as dependencies.
| conradludgate wrote:
| As a fellow rust developer, I love our dependencies but I put a
| lot of effort into pruning the ones I want to use. If I see a
| crate using too many I might contribute to it or find a
| replacement.
|
| If you want to use dependencies, I wouldn't be surprised when you
| realise they also want to use dependencies. But you can put your
| money/time in the right places. Invest in the dependencies that
| do things well.
| zaptheimpaler wrote:
| This is just a modern problem in all software development,
| regardless of language. We are doing more complex things, we have
| a much bigger library of existing code to draw from and there are
| many reasons to use it. Ultimately a dependency is untrusted
| code, and there's a long road to go in hardening entire systems
| to make running arbitrary dependencies safe (if its even
| possible).
|
| In the absence of a technical solution, all others basically
| involve someone else having to audit and constantly maintain all
| that code and social/legal systems of trust. If it was pulled
| into Rust stdlib, that team would be stuck handling it, and
| making changes to any of that code becomes more difficult.
| harha_ wrote:
| Regardless of language, really? I highly doubt that, you don't
| generally see such problems with C or even C++ because
| dependencies are more cumbersome to add, especially in a way
| that's cross-platform.
| zaptheimpaler wrote:
| Because most dependencies are either manually installed by
| the user, or are dynamic libraries that are provided and
| audited by the distro maintainers. The dependencies are
| there, they're just harder to see -
| https://wiki.alopex.li/LetsBeRealAboutDependencies
| harha_ wrote:
| Sure, there are various dependencies, but it's nothing like
| "cargo install crate-name". Cargo makes it so effortless to
| joink the dumbest dependency for the simplest thing.
| zaptheimpaler wrote:
| Sure, I think software that's easy to use is a good thing
| and Rust dependency management is 100x nicer to work with
| than C++.
| pjmlp wrote:
| Kind of true, when not using vcpkg/conan.
| zeroxfe wrote:
| > If it was pulled into Rust stdlib, that team would be stuck
| handling it, and making changes to any of that code becomes
| more difficult.
|
| I think Rust really needs to do more of this. I work with both
| Go and Rust daily at work, Go has its library game down -- the
| standard library is fantastic. With Rust it's really painful to
| find the right library and keep up for a lot of simple things
| (web, tls, x509, base64 encoding, heck even generating random
| numbers.)
| kdps wrote:
| I'd argue that the severity varies between languages, despite
| the core problem being universal. Languages with comprehensive
| standard libraries have an advantage over those with minimal
| built-in functionality, where people rely on external
| dependencies even for the most basic things (e.g. see Java/.NET
| vs JS/Node). Lightweight is not always better.
| jongjong wrote:
| Yes, but a lot of the complexity is unnecessary bloat. Almost
| every project I've ever seen or worked on was full of
| unnecessary complexity. People naturally tend to over-
| complicate things, all the programming books, including
| software design books focus on unimportant aspects and miss all
| the important ones. It's incredibly frustrating.
|
| Yet, if someone were to write a book which explained things
| properly (probably a 3000 word article would suffice to turn
| anyone into a 10x dev), nobody would buy it. This industry is
| cooked.
| QuadmasterXLII wrote:
| I wonder how much good a "dependency depth" label on packages
| would do, at the crates.io level. Like, a package can only depend
| on a package with a lower declared dependency depth than it, and
| packages compete to have a low dependency depth as a badge.
| amelius wrote:
| I think the main problem is that you should be able to run
| dependencies inside their own sandbox, and the language focuses
| only on memory safety within a monolithic program.
| colanderman wrote:
| Rust at least has a partial remedy to this problem: feature
| flags. Many libraries use them to gate features which would
| otherwise pull in extra dependencies. (In fact I believe there is
| specific support for flags which correspond to dependency names.)
___________________________________________________________________
(page generated 2025-05-09 23:00 UTC)