[HN Gopher] The birth of a package manager
___________________________________________________________________
The birth of a package manager
Author : wofo
Score : 57 points
Date : 2023-03-16 14:56 UTC (8 hours ago)
(HTM) web link (ochagavia.nl)
(TXT) w3m dump (ochagavia.nl)
| calvinmorrison wrote:
| I thought I'd get started on Rust last night with a sort of
| simple getopt's CLI tool. I immediately ran into some whacky
| dependency/library issue. Google revealed an open bug report.
|
| I put rust down and maybe I'm stupid but, I was following a 'get
| started with rust' type of tutorial.
|
| I think I will not look again for some years.
| brabel wrote:
| That's surprising, the Rust ecosystem is already getting quite
| mature for most things. Can you share the bug you ran into?
| epage wrote:
| Was glad to see this blurb
|
| > One interesting avenue of future work is to try to replace
| libsolv by a solver written in Rust, such as PubGrub. That way we
| could get rid of a bunch of unsafe code we are using to interface
| with libsolv through Rust's FFI.
|
| For people who would be interested in learning more, check out
| this conference presentation: https://youtu.be/XyJpYFnl18U
|
| Using pubgrub in cargo is something we talk about on a regularly
| basis. Its our equivalent of rustc chalk or polonius: we keep
| limping along, always finding a new hack to extend the life of
| the current package resolver due to the large lift to change out
| the resolver but also knowing how much the current one slows us
| down and holds us back.
|
| I'm excited for any forward movement on pubgrub in any related
| project.
| munificent wrote:
| If you prefer text over video, here's Natalie's original
| article about PubGrub:
|
| https://nex3.medium.com/pubgrub-2fb6470504f
|
| She invented this for Dart's package manager (pub), which is
| where the name comes from.
| droelf wrote:
| Another rattler author here! Indeed, pubgrub-rs looks like one
| hell of a good alternative to libsolv. We've already tried
| quite a bit to use it in rattler, but the many "variant"
| packages in the conda ecosystem gave it a bit of a hard time.
| We still think that we can make it work.
|
| And thanks for linking a PackagingCon talk - I am actually the
| an organizer of that conference and happy to report that it's
| coming back this year in October (https://packaging-con.org has
| the details).
| epage wrote:
| Oo, don't want to miss it next year! Do you have a mastadon
| account or an email list? I'm trying to keep to my "no
| twitter" streak.
| droelf wrote:
| I just made a mastodon account for PackagingCon on
| Fosstodon, will update you here once it's approved.
| _frkl wrote:
| Thanks for sharing, very cool! There doesn't seem to be a whole
| lot of documentation on how to use it as a library, but looking
| at the code all seems to be nice and tidy, and it doesn't look
| too difficult to adapt to a use-case or two I have in mind.
|
| Any plans to publish a release version to crates.io, or is it
| mainly intended as an internal tool for prefix.dev?
| wofo wrote:
| As far as I know the plan is indeed to publish a version to
| crates.io, and the only reason it hasn't been done yet is that
| it is still early days. You might want to jump in their discord
| server to ask, though, if you want to be sure (or maybe you can
| even create an issue on GitHub)
| baszalmstra wrote:
| Rattler author here! I just published v0.1.0 to crates.
| _frkl wrote:
| Yay! :) Thanks heaps! It's funny, 'rattler' is exactly what
| I need at the moment. I was about to write a rust app that
| downloads 'micromamba' and runs it as a subprocess. But
| this is so much better for what I try to do, being able to
| include everything in a single binary and calling rust code
| directly...
| baszalmstra wrote:
| Great to hear! You peaked my interest now! I would love
| to know more about your usecase! Would you be open to
| join our discord to talk about it?
| wofo wrote:
| OP here, happy to see this reached the frontpage! For those like
| me who always look at the comments to decide if the article is
| worth their time: this is about an open source package manager
| written in Rust, which I had the pleasure to work on during my
| last contract. There were a bunch of interesting technical
| challenges (mostly around performance), which required some
| creativity to solve. If you like that kind of stuff, you might
| enjoy reading the article ;)
| ris wrote:
| Ah yes if there's one thing the world needs it's another package
| manager to fight with the other seventeen on your system.
| pxc wrote:
| The caching trick for libsolv here could be a game changer for
| performance in a lot of Linux distro package managers, where
| libsolv's completeness comes at a substantial performance cost.
|
| I've never looked too deeply into how conda works because I'm not
| that into Python, but I guess I should. It seems there are some
| valuable general lessons here-- I wish that more package
| managers, and in particular, Nix, maintained more version
| combinations in their repos. Searching through Nixpkgs commits to
| find compatible versions of things because you have to rely on
| the implicit versioning of the monorepo sucks.
|
| Adding dependency resolution to Nix as it stands would normally
| add a big performance cost. Something like this would help with
| that problem a lot, and Nix already usually runs with a
| persistent daemon. Maybe a similar technique could fit in well
| here.
| chriswarbo wrote:
| I'm not sure how 'dependency resolution' would work in a system
| like Nix; since Nix has no notion of "package", "dependency",
| "version", etc.
|
| > Searching through Nixpkgs commits to find compatible versions
| of things because you have to rely on the implicit versioning
| of the monorepo sucks.
|
| I get what you're saying, but I gave up on thinking of
| Nix/Nixpkgs in terms of "package management". It's much closer
| to a build system, like GNU Make; rather than, say, APT/Yum.
|
| When it comes to versions of things in Nixpkgs: most things are
| defined as _a function of other things_ ; i.e. the arguments to
| that function are (roughly) its "dependencies". If we want some
| different version, we can call those functions with different
| arguments (or use the various `override` helpers included in
| their return values). Once we've overridden the things we care
| about, we can usually take the fixed-point/closure of that
| package set (whether it's the whole of Nixpkgs, or just e.g.
| python3Packages, or whatever); that way, our changes get
| propagated through all of the function calls to ensure
| everything's referencing our versions. Admittedly, this can be
| quite tedious!
| pxc wrote:
| I'm aware of how Nixpkgs currently works. :)
|
| It's a huge limitation for many reasonable ways that people
| want to use Nix. It's a shortcut that Nix has gotten away
| with, not a superior approach. Look at the machinery mach-nix
| implements with a cached database of PyPi and how it has to
| scan across different Nixpkgs revisions just to automatically
| choose compatible versions of deps when it does its code
| generation for a clear illustration.
|
| There are lots of source-based package management systems
| which rely on a mix of implicit versioning based on what
| happens to be exist under an unversioned name in the monorepo
| and explicit versioning, Gentoo Portage for example.
|
| There's no reason that we couldn't have either
| a. a complement to Nixpkgs which serves as a record of which
| build recipes have successfully built what versions, and what
| versions their deps were at b. richer metadata in
| Nixpkgs about versions that can be used to support multiple
| versions of same-named packages in the repository, and a
| smarter callPackage that uses those version constraints to
| choose, when necessary, which versions of dependencies to put
| into scope when calling a package (yes, I know there is
| ongoing work to remove callPackage, or at least explicit
| invocations of it)
|
| and both would be really useful for tools that autogenerate
| Nix code for packaging purposes.
|
| Hacking things together with nix-index just to get a little
| automagical code generation where users can specify package
| versions rather than Nixpkgs commits is a kludge. It's
| cumbersome and rightly turns some developers off from using
| Nix. Nixpkgs should be enriched or complemented in order to
| obviate that approach, and the Nix ecosystem will need
| tooling that has an actual dependency resolver to work with
| it-- even if the only maintained combination of versions for
| distribution is pretty much singular, and implicitly encoded
| in Nixpkgs just as it is today.
| dralley wrote:
| .solv cache files are a libsolv thing that has existed for a
| long time, the only "trick" here is using them. The package
| managers you refer to already do as far as I'm aware, the
| slowness is for other reasons.
| pxc wrote:
| Doesn't the persistent daemon/server that keeps them pre-
| loaded in RAM make a difference here over just repeatedly
| loading something that's cached on disk?
| droelf wrote:
| Yes, we definitely believe and rely a lot on proper "SAT"
| solving in the conda ecosystem (old versions are also never
| deleted, for reproducibility reasons, which makes this problem
| actually quite interesting and large).
|
| We are also quite curious to learn more about the Nix approach.
| Building packages is still a bit too painful / annoying in the
| conda ecosystem, which is something we really want to fix, and
| I think we can learn a lesson or two in the Nix ecosystem.
|
| And by the way, the conda ecosystem is not at all Python
| specific. It's a common misconception, but they are all general
| binary packages and can be used for Ruby, R, Rust, C++, Zig,
| ... whatever - just like Nix. Rattler, mamba and conda all
| handle them the same nice way.
| pxc wrote:
| I knew that conda's connection to Python was mostly
| historical rather than technological, since it handled native
| deps and so it had to be able to work with C as well. I
| didn't realize so much work had been done packaging all of
| those other language ecosystems, though!
___________________________________________________________________
(page generated 2023-03-16 23:02 UTC)