[HN Gopher] The magic of dependency resolution
___________________________________________________________________
The magic of dependency resolution
Author : willm
Score : 31 points
Date : 2023-07-11 16:03 UTC (6 hours ago)
(HTM) web link (ochagavia.nl)
(TXT) w3m dump (ochagavia.nl)
| Gunax wrote:
| Never thought of dependencies as a SAT issue before... Makes
| sense once I think about it.
|
| I always imagined dependencies as a tree.
|
| If I have dependency x which uses foo 1.0 and dependency y which
| uses foo 2.0, isn't it possible to get an unsatisfiable rule?
|
| Why do we need a universal version of foo? Cant we just use both
| foo 1.0 and 2.0? When being used by x, we use foo1.0 and when
| being used by y, use foo 2.0
| OscarCunningham wrote:
| Ideally, one would also prefer higher versions of each package.
| How much longer would it take to find a solution with the
| guarantee that there's no other solution with all packages at
| least as good?
| droelf wrote:
| Unfortunately, it is sometimes not even possible to say which
| solution should be preferred because different branches lead to
| different sets of packages having higher versions. There is not
| really an "ultimate" heuristic to determine which of the two
| sets is better.
|
| However, users are always encouraged to add more constraints to
| shape the solution, and packagers encouraged to take good care
| of the metadata.
|
| At prefix we are quite interested in figuring out how we can
| help packagers to determine e.g. compatibility ranges more
| correctly by doing static analysis and such things.
| OscarCunningham wrote:
| Right, but we could still ask for Pareto optimality: no way
| to make one package better, while keeping the rest at least
| as good.
| crabbone wrote:
| Ideally, there should be a choice that allowed to look for the
| minimal and the maximal versions that satisfy given
| requirement.
|
| Developers usually want the newest, but maintainers usually
| want the oldest.
| JohnFen wrote:
| > Ideally, one would also prefer higher versions of each
| package
|
| Ideally, yes. In practice? I know that I rather strongly prefer
| not to use the latest version of any piece of software.
| Dudester230602 wrote:
| _" Dependency resolution is something programmers usually take
| for granted. Be it cargo, npm, or whatever package manager you
| use, no one is actually surprised when this black-box figures
| out, all by itself, the specific set of packages that should be
| installed."_
|
| I am personally still surprised npm/node etc. both work and are
| used unsarcastically.
| crabbone wrote:
| Oh, this is about Conda?..
|
| I think, this is going to be the third go at the problem? The
| first two were hands-down awful leading to creation of Mamba. But
| speeding things up is only part of the solution here.
|
| Conda package ecosystem has a bunch of issues that are related to
| the format and the repositories providing packages as well as on
| the policy for accepting packages into those repositories. They
| tried to solve too many problems at once, probably, w/o even
| realizing what kind of problem they are attacking.
|
| Here's a major problem with Conda ecosystem, the way I see it:
| package developers are encouraged to produce very "precise"
| dependency requirements because this makes installation process
| go faster and ensures less wiggle room for untested dependency
| combination. This, in turn, results in proliferation of package
| versions, all of which declare themselves as incompatible and
| non-interchangeable with many others.
|
| On top of this, beside providing just the Python packages, Conda,
| ambitiously, wants to also provide shared libraries, which blows
| up the minimum installation to an enormous size, including the
| number of packages that need installing.
|
| So, being a Conda package maintainer, you will face a dilemma: do
| you want your users to be able to install the package in
| acceptable time (at least under 10 minutes...) or work towards
| compatibility with multiple different versions of other packages
| the users might need for other reasons in the same environment
| (and your installation times start shooting through the roof). In
| my experience, the longest _successful_ Conda install took about
| a day (i.e. close to 24 hours). It may take few days for Conda to
| fail to install (due to imaginary or real conflict).
|
| Now, add to this the existence of multiple sources for different
| packages (i.e. "channels"). Different channels may, and actually
| quite often do provide different unrelated other packages in
| their channel, but the users cannot specify the channel when
| requesting to install a package _on per-package basis_. So, the
| solver needs to guess somehow which channel to prefer as a source
| of any given package. The problem is that while the package
| version found in that channel might be the same, the requirements
| for that package might differ between channels.
|
| So, while aiming at providing a lot of possibly useful features,
| Conda programmers programmed themselves into a corner with no
| good way out.
|
| Several things might help, but none will ultimately solve the
| situation.
|
| * Package bundles (or meta-packages, like they are called in some
| Linux distributions) will cut significantly on the work a solver
| needs to do. A lot of the basic (shared libraries) packages are,
| essentially, always the same in each install.
|
| * LTS curated channel snapshots. I.e. provide a very large
| collection of packages that are verified to all work together,
| but unlike the meta-packages need not be installed together.
|
| * Incorporate channel into version specification, especially when
| specifying package dependencies so that maintainers were required
| to specify the source of their dependencies when submitting their
| package.
|
| * There are a bunch of (very) infrequently used options in the
| current solver that can be safely removed, but they add to the
| complexity and the time the solver spends in various "impossible"
| scenarios.
| droelf wrote:
| Let me try to reply to you (as one of the authors of mamba and
| as a conda-forge core dev):
|
| Mamba has solved (seemingly successfuly) many problems of conda
| being slow. It's used by default in conda-forge, the largest
| conda repository out there. Asking packagers to add upper
| bounds is less of a speed-hack vs. just the correct thing to
| do. What's interesting about conda-forge is that upper bounds
| can also be added later on via a "repodata patch". They really
| just serve to get users packages that are actually compatible.
|
| The kind of "freak solves" are not a thing anymore with mamba.
|
| I agree with you that multiple channels present problems.
| Channels that explicitly inherit from each other work quite
| well together (e.g. bioconda and robostack channels extend
| conda-forge), but the Anaconda main (defaults) channel and
| conda-forge do not work well together - on the metadata and ABI
| level. For this reason we encourage users to never mix those
| two.
|
| At prefix.dev we do want to make it easy to build "on top of"
| conda-forge in the future.
|
| I think most issues on your list that follows at the end are
| non-issues once you start to use mamba. And I would encourage
| you wholeheartedly to give micromamba a try (for a really fast,
| single-binary experience with no base env and slim
| installation) or pixi (again, no base env, just a Rust binary).
| javajosh wrote:
| Props to the author for thanking their tester for breaking their
| stuff until it worked! Good testers are worth their weight in
| gold.
| JohnFen wrote:
| I'll second this. Good testers are at least as valuable and
| essential as good devs.
___________________________________________________________________
(page generated 2023-07-11 23:02 UTC)