[HN Gopher] Rye and Uv: August Is Harvest Season for Python Pack...
___________________________________________________________________
Rye and Uv: August Is Harvest Season for Python Packaging
Author : keybits
Score : 134 points
Date : 2024-08-21 11:12 UTC (11 hours ago)
(HTM) web link (lucumr.pocoo.org)
(TXT) w3m dump (lucumr.pocoo.org)
| keybits wrote:
| Uv's latest release was discussed yesterday:
| https://news.ycombinator.com/item?id=41302475
|
| The linked post is the author of Rye's take on that.
| bananapub wrote:
| I really like this framing - lots of incremental work by lots of
| people over time got us to the point where ~a few people at one
| company can radically improve the situation with a medium amount
| of work.
| dirtbag__dad wrote:
| TY to the astral team for making my quality of life so much
| better and Armin for being brave enough to pass the torch.
|
| Strong +1 on a one tool wins approach - I am so tired of burning
| time on local dev setup, everything from managing a monorepo
| (many packages that import into each other) to virtual
| environments and PYTHONPATH (I've been at it for like 8 years now
| and I still can't grok how to "just avoid" issues like those
| across all pkg managers, woof!)
|
| I am really excited to see what's next. Especially looking for a
| mypy replacement and perhaps something that gives compiling
| python a "native" feeling thing we can do
| alanfranz wrote:
| There's one problem left with those tools: authority. They're not
| pypa endorsed, that's what makes those different from cargo.
|
| At the same time pypa wasn't able to provide a comprehensive
| solution over the years, and python packaging and development
| tools multiplied - just 3-4 years ago poetry and pipenv seemed to
| solve python packaging problems in a way that pip+virtualenv
| couldn't.
|
| We need pypa to now jump on the astral.sh ship - but will they do
| that without a certain amount of control?
| the_mitsuhiko wrote:
| As a somewhat outside observer I don't quite understand what
| the PyPA actually is. In some sense there is an unclear number
| of participants and it's not even sure to which degree the PyPA
| relates to core python or the PSF.
|
| I think the real endorsement that could help would be the core
| Python project itself. In a perfect world the official Python
| tutorial would start with "here is how you install python" and
| it starts by installing uv, the same way as the official Rust
| docs point to rustup and cargo.
|
| I hope strongly that the PSF will manage to establish some sort
| of relationship with Astral which would enable that to
| eventually be a reality.
| crabbone wrote:
| Here are few things that... well, won't really work well:
|
| * Python package installation, package format and loading of
| modules are defective. The design is bad. It means that no
| implementation blessed by PSF or not isn't going to solve the
| packaging problem. So, there's no point to ask PSF or PyPA to
| adopt any external tool. If the external tools are better
| than pip in some way, then it could be the speed or memory
| footprint etc. They will not solve the conceptual problems,
| because they don't have the authority to do that.
|
| * PyPA and PSF, but maybe to a lesser extent, are populated
| today by delusional mediocre coders who have no idea where
| the ship is going or how to stir it. They completely lack
| vision and understanding of the problems they are to deal
| with. They add "features", but they don't know if those
| features are needed, and, in most cases it's just noise and
| bloat. From a perspective of someone who has to deal with
| fruits of their labor, they just ensure that my job of
| "someone who fixes Python packaging issues" will never go
| away.
|
| So... as "someone who fixes Python packaging issues" I kind
| of welcome the new level of hell coming from Astral. From
| where I stand, it is pouring more gas into a big dumpster
| fire. Just one more tool written in a non-mainstream, non-
| standardized, quickly evolving language, impossible to debug
| without a ton of instrumentation, with the source code hard
| to write and hard to understand. It's just another deposit
| towards my job security.
| simonw wrote:
| "Python package installation, package format and loading of
| modules are defective"
|
| How are they defective?
| simonw wrote:
| I learned recently that the "authority" in the name PyPA -
| Python Packaging Authority - was originally intended as a joke:
| https://discuss.python.org/t/remove-the-authority-from-packa...
| kstrauser wrote:
| I stopped worrying about what PyPA said when they endorsed
| Pipenv above other options, largely due to what seemed to me to
| be personal relationship reasons. Pipenv was disastrous at that
| time. It _meant_ well, but at work we found ourselves waiting
| an hour for it to update its lockfile, and we had a relatively
| small number of widely-used dependencies in each repo. It just
| didn 't work.
|
| As a practical matter, I hugely appreciate the hard technical
| work PyPA does. However, I don't concern myself much with which
| toolsets they're recommending at the moment. Use what the
| community uses and don't worry about the "official"
| suggestions.
| fvdessen wrote:
| I was looking this morning at migrating our software from poetry
| to uv at my company, due to poetry's slowness. And so far i've
| been reading a lot of doc and not getting a lot things done. I
| did the previous migration to poetry as well which was vastly
| simpler. So far it seems that poetry tried to make a simple
| package manager that works like any other, while uv is keeping
| quite a bit of the python package insanity around.
| burntsushi wrote:
| Are there any specifics you can share about what you found
| difficult to do in uv?
| bvrmn wrote:
| At least uv doesn't lead to an absolute clusterfuck of poetry
| conflicting with virtualenv. Or broken package.toml format with
| minor poetry changes. Or amazingly dumb "sources" which doesn't
| work for transitive dependencies and leads to even longer
| resolve times in case of multiple indexes.
| orbisvicis wrote:
| To which insanities are you referring?
| yoyohello13 wrote:
| We just recently migrated to uv from pip-tools and it was
| pretty seamless. uv pretty much just plugs into the standard
| python tooling workflow.
| lmeyerov wrote:
| After the whole npm VC rugpull + Microsoft acquisition, and
| OpenAI showing legal non-profit status is toothless marketing to
| VC-path-entangled leaders, I'm reluctant to cede critical path
| language infra to these kinds of organizations. Individual
| contributors to these are individually great (and often
| exceptional!), but financial alignment at the organizational
| level is corrupted out of the gate. Fast forward 1-4 years, and
| the organization is what matters. "Die a hero or live long enough
| to become the villain."
|
| So fast lint, type checking, code scans, PR assistants, yes, we
| can swap these whenever. But install flow & package repo, no.
|
| That is unfortunate given the state of pip and conda... But here
| we are.
| crabbone wrote:
| That battle has already been lost for Python. Microsoft owns
| Python, they just don't make it public.
|
| This is how I came to believe this is the case:
|
| Few years ago I wanted to write Python bindings to kubectl. I
| discovered that in order for that to work cross-platform, I
| need to make CGo use the same compiler on all platforms as does
| Python. Unfortunately, on MS Windows, CGO uses MINGW while
| Python uses MSVC. I wrote to Python dev. mailing list (which
| still existed at that time) and asked why did they choose to
| use a proprietary compiler for their "open-source" project. The
| answer I received in a round-about way was that MSVC was a
| historical choice, which cannot be presently changed because MS
| provides Python Foundation with free infrastructure to run CI
| and builds, and it also provides developers to work on Python
| (i.e. MS employees get paid by MS to work on Python
| interpreter). And that they are under orders not to drop MS
| tools from the toolchain.
|
| Year after year the situation was getting worse. Like in a lot
| of similar projects, success created a lot of ground for
| mediocre nobodies to reach positions of power. Python
| foundation and satellite projects like PyPA started to be
| populated by people whose way into these positions was not
| through contributing any useful code, but rather writing pages
| of code of conduct. This code of conduct and never-ending
| skirmishes around controlling positions eventually led to some
| old-timers leaving or being outright kicked out (latest such
| event was the ban of Tim, the guy who, beside other things,
| wrote Tim sort, which is a somewhat famous feature of Python).
|
| Year after year MS was pushing its usual agenda they do in
| every project they get their hands on: add crapload of useless
| features for the sake of advertising. Make the project swing
| every way possible, but mostly follow the fashion trends as
| hard as possible. This is how Python is now devoted to adding
| as much of ML-style types as possible (in the language with a
| completely different type system...), AoT compilation and JIT
| (in the language that's half of the time used to dynamically
| glue native libraries...) and so on. Essentially, making it a
| C#, but without curly braces.
|
| MS is smart enough to understand that publicly announcing their
| ownership of Python will scare a lot of people away from the
| technology, so they don't advertise it much. But they keep
| working on ensuring developers' dependency on their tooling,
| and eventually they will come to collect on their investment.
| cdchn wrote:
| Sounds more like Microsoft owns Windows rather than Python.
| ris wrote:
| I wouldn't blame people if they sat this round out and waited for
| 2026's iteration of "Python Package Managers: we've really solved
| it this time!"
|
| (still a happy Nix user)
| nurettin wrote:
| Should have ended with "I use Nix, btw"
| zokier wrote:
| I know python packaging has it's issues, but so far I personally
| have gotten pretty far with plain pip. The biggest shift to me
| was switching from original virtualenv to built-in venv module.
| On the other hand, if I wanted to be really serious about
| dependency management, I'd steal a page from FAANG, and build a
| monorepo and avoid all this hassle with package managers.
| drawnwren wrote:
| I would really encourage you to try your hand at at a monorepo.
| I manage a python monorepo in prod and dependency management is
| hell. Poetry has some newer features that I am looking at
| trying to implement, but the state of the ecosystem wrt big
| monorepos is horrible.
| crabbone wrote:
| > I manage a python monorepo in prod and dependency
| management is hell
|
| You are either not very competent at what you do, or you
| wrote the opposite of what you wanted to write.
|
| With monorepo you don't have dependencies. Everything is in
| the repository. The reason to have monorepo is to avoid
| dealing with dependencies...
| miohtama wrote:
| Generally all Python software use third party packages
| which are fetched from PyPi package site. These packages
| are not stored in monorepo.
| dudus wrote:
| You still have dependencies. But they are included in the
| repo.
|
| What happens when you have 2 different apps in your
| monorepo but one uses an older version of Django and wasn't
| upgraded yet. The monorepo doesn't handle that
| automatically, you need tooling. It's not as black and
| white as you say.
| drawnwren wrote:
| So, you have two choices for directory layout in mono
|
| either:
|
| root/ requirements.txt / pyproject.toml
| src/
|
| ---
|
| in which case you need to be careful about when your
| dependencies get installed during deployment because ie
| pytorch can be ~500mb
|
| your other option is
|
| root/ src/ A/
| requirements.txt / pyproject.toml B/
| requirements.txt / pyproject.toml
|
| which means you need to come up with a bespoke dependency
| management solution. in either case, you're doing
| dependency management.
|
| I encourage you to google "monorepo {python, poetry, pip}"
| and you're going to land on many multiple page blog posts
| describe arcane dependency management solutions.
| animal_spirits wrote:
| So this example is a single repo with many services? I
| don't think OP was talking about that. Why would you have
| a single repo with different requirements? At that point
| they should be independent projects
| gen220 wrote:
| The second option is traditionally what people mean by
| "monorepo" (multiple executables in a repo with a
| dependency tree). First one you'd call a "monolith"
| (single executable in a repo).
|
| The second one you'd want to manage with a tool like
| Bazel, which will use whatever plugins are appropriate
| for the language (pip in Python's case).
|
| It's a legitimate ambition to try and build a tool within
| a single language's ecosystem to manage the second
| option, but it's a really hard problem that Bazel (and
| others) have already solved well, so you might as well
| use them instead.
| akdor1154 wrote:
| I'm in a similar boat - uv's workspaces look tantalising.
| kapilvt wrote:
| yeah. its not great, we had to build out a poetry plugin that
| worked for our cases to support a mono repo,
| https://github.com/cloud-custodian/poetry-plugin-
| freeze?tab=...
|
| the uv support on workspaces (virtual and concrete) has me
| intrigued.
| PaulHoule wrote:
| My take is that this one of the most interesting topics there is
| right now but a below average article about that topic.
| simonw wrote:
| Are you saying that because this was more of an opinion piece?
|
| As the author of Rye, a package that was heavily influential on
| the development of uv, Armin is very well positioned to share
| his opinions about this.
| orbisvicis wrote:
| With declarative tooling - unlike setup.py - we've lost the
| ability to install to locations not managed by Python. Most of my
| projects used to have a python-code config file that I could
| include both in my code and in setup.py. Last I checked none of
| that is now possible. Want to install a system binary - ship an
| RPM or a flatpak.
|
| I don't understand how we could lose so much flexibility and yet
| gain so little in return.
|
| P.S I've only ever encountered minor dependency issues in my
| admittedly small projects using just pip and venv.
| OJFord wrote:
| That's entirely appropriate imo - system packager should be
| running pip or whatever and then putting the output where it
| wants, it's not for setuptools or whatever to declare that.
|
| What path would you put in setup.py anyway? A different one for
| different distro preferences, a different one again for
| Windows, for macOS?
| heavyset_go wrote:
| You can still use the setuptools backend when using declarative
| tooling, I use it in projects I can't fully port to
| pyproject.toml.
| bkor wrote:
| For those interested in uv (instead of pip), uv massively sped up
| the release process for Home Assistant. The time needed to make a
| release went down from ~2.5 hours to ~20 minutes. See
| https://developers.home-assistant.io/blog/2024/04/03/build-i...
| for details. I'm just a HA user btw.
___________________________________________________________________
(page generated 2024-08-21 23:00 UTC)