[HN Gopher] Constraints Are Good: Python's Metadata Dilemma
___________________________________________________________________
Constraints Are Good: Python's Metadata Dilemma
Author : ingve
Score : 59 points
Date : 2024-11-27 07:09 UTC (9 days ago)
(HTM) web link (lucumr.pocoo.org)
(TXT) w3m dump (lucumr.pocoo.org)
| PaulHoule wrote:
| Eggs are dying out, pointed out by this 2 year old blog post:
|
| https://about.scarf.sh/post/python-wheels-vs-eggs
|
| The metadata problem is related to the problem that pip had an
| unsound resolution algorithm based on "try to resolve something
| optimistically and hope it works when you get stuck and _try_ to
| backtrack ".
|
| I did a lot of research along the line that led to uv 5 years ago
| and came to the conclusion that installing out of wheels you can
| set up a SMT problem the same way maven does and solve it right
| the first time. They had a PEP to publish metadata files for
| wheels in PyPi but I'd built something before that could suck the
| metadata out of a wheel with just 3 http range requests. I
| believed that any given project might depend on a legacy egg and
| in those cases you can build that egg into a wheel via a special
| process and store it in a private repo (a must for the perfect
| Python build system)
| the_mitsuhiko wrote:
| The metadata problem is unrelated to eggs. Eggs haven't played
| much of a role in a long time but the metadata system still
| exists.
|
| Range requests are used by both uv and pip if the index
| supports it, but they have to make educated guesses about how
| reliable that metadata is.
|
| The main problem are local packages during development and
| source distributions.
| PaulHoule wrote:
| Back in the case of eggs you couldn't count on having the
| metadata until you ran setup.py which forced pip to be
| unreliable because so much stuff got installed and
| uninstalled in the process of a build.
|
| There is a need for a complete answer for dev and private
| builds, I'll grant that. Private repos like we are used to in
| maven would help.
| the_mitsuhiko wrote:
| Eggs did not contain setup.py files. The metadata like for
| wheels was embedded in the egg (in the EGG-INFO folder)
| from my recollection. Eggs were zip importable after all.
| PaulHoule wrote:
| It looks like you can embed dependency data in an egg but
| it is also true that (1) a lot of eggs have an internal
| setup.py that does things like compile C code, and (2)
| it's a hassle for developers on some platforms who might
| not have the right C installed, and (3) eggs reserve the
| right to decide what dependencies they include when they
| are installed based on the environment and such.
| taeric wrote:
| I am curious how Python got into this situation. Was it largely
| taking the path of least resistance to more and more adoption?
|
| I get that Python is, strictly speaking, an older language. But,
| it isn't like these are at all new considerations.
| the_mitsuhiko wrote:
| Classic case of lack of constraints early on. Once people use
| all that power you end up with a mess.
| lyu07282 wrote:
| > The challenge with dynamic metadata in Python is vast, but
| unless you are writing a resolver or packaging tool, you're not
| going to experience the pain as much.
|
| But that is by choice, I as a user, am forced to debug this pile
| of garbage whenever things go wrong, so in a way it's even worse
| for users. It's a running joke in the machine learning community
| that the hard part about machine learning is having to deal with
| python packages.
| pdonis wrote:
| A lot of the problem seems to be driven by a desire to have
| editable installs. I personally have never understood why having
| editable installs is such an important need. When I'm working on
| a Python package and need to test something, I just run
|
| python -m pip install --user <package_name>
|
| and I now have a local installation that I can use for testing.
| the_mitsuhiko wrote:
| That would you require to make re-installations if your local
| app you develop against after every code change. Very few
| people will want to do that and it's potentially very slow.
|
| It's also a step not needed by most other ecosystems.
| pdonis wrote:
| _> it's potentially very slow._
|
| Potentially, perhaps. But it's certainly not for the cases
| where I use it: a pure python package, whose dependencies are
| already installed and are not changing (only the package
| itself is). Under those conditions, the command line I gave
| takes a couple of seconds to run.
| paulddraper wrote:
| I.e. orders of magnitude longer
| pdonis wrote:
| _> It's also a step not needed by most other ecosystems._
|
| From what I can gather, most other ecosystems don't even have
| the problem under discussion.
| raoulj wrote:
| Yeah. It's too slow. Editable installs make application
| development much faster.
___________________________________________________________________
(page generated 2024-12-06 23:01 UTC)