[HN Gopher] Venvstacks: Virtual Environment Stacks for Python
___________________________________________________________________
Venvstacks: Virtual Environment Stacks for Python
Author : ingve
Score : 91 points
Date : 2024-11-02 12:19 UTC (1 days ago)
(HTM) web link (lmstudio.ai)
(TXT) w3m dump (lmstudio.ai)
| incognito124 wrote:
| So, Docker container image, without OverlayFS benefits?
| zeotroph wrote:
| And with just 3 layers: Runtime, Framework, Application. But at
| least you are not switching tools, and it presumably would
| prevent you from installing LARGE_v1.1, and then installing
| TINY_v2.2 in a later layer, which however upgrades LARGE to
| v1.2 and your docker images are now twice the size.
| davedx wrote:
| Python's Cambrian explosion of dependency management continues.
| Maybe one day via some strange machine form of genetic
| algorithms a canonical system will emerge.
| xelamonster wrote:
| I was expecting this to be containers under the hood, instead
| it seems the idea is basically to bundle your virtualenv and
| ship it to prod? I think I'll stick with Docker...
| thangngoc89 wrote:
| I'm using poetry because of the default lock file usage (there
| isn't a global, mutable state, all changes required to update the
| lock file). I wish that this function could be incorporated to
| poetry directly
| Terretta wrote:
| For a variety of reasons this season is a good time to look at
| `uv`.
|
| See this by the author of Rye:
|
| https://lucumr.pocoo.org/2024/8/21/harvest-season/
|
| "Unified" packaging:
|
| https://astral.sh/blog/uv-unified-python-packaging
| trallnag wrote:
| Poetry works so well for me at the moment, I prefer to let uv
| cook
| thangngoc89 wrote:
| I just take a look at the uv feature list and the feature I
| wanted most is Python version management. I'm using
| micromamba to install all interested Python versions (from
| 3.8 - 3.12) and set tell poetry about the python location.
|
| But like you said, poetry is working so well so I'll wait a
| little bit longer before jumping the ship.
| xrd wrote:
| Since you mentioned uv and the topic is virtual
| environments...
|
| I am using uv and it seems great.
|
| I don't understand the difference between using "python -m
| venv .venv; source .venv/bin/activate" and creating a venv
| with uv and then running the same source command. What does
| uv offer/integrate that's not present already in python venv
| module?
| thangngoc89 wrote:
| It replaces pip and uses venv module under the hood if I
| understand correctly
| xrd wrote:
| Got it. I'm just unsure whether I can use the other uv
| features like pin with it? That feature feels like it was
| added and the benefits or trade-offs weren't documented.
| At least I don't see it.
| zeotroph wrote:
| Ah, more complex than I thought: "venvstacks allows you to
| package Python applications and all their dependencies into a
| portable, deterministic format, without needing to include copies
| of these large Python frameworks in every application archive.",
| and in "Caveats and Limitations" (please, all projects should
| have one): "does NOT support combining arbitrary virtual
| environments with each other".
|
| _Is_ there a helper to merge venv1 and venv2, or create venv2
| which uses venv1 dependencies and on load both are merged?
| HappMacDonald wrote:
| My understanding is that the entire reason that venv exists is
| because python's library system is nothing but dependency
| spaghetti: whatever is needed by one project conflicts with
| whatever is needed by another so you have to give them bespoke
| library environments where those conflicts won't interfere with
| one another.
|
| In that perspective "merging" them directly defeats the
| purpose. What is needed is a better library ecosystem.
| zahlman wrote:
| venvs are used to isolate groups of dependencies, but it's
| not just about conflicts. Many other languages expect you to
| do the same thing; people complain less because either the
| language statically resolves imports and can support multiple
| versions of a library in the same environment; and/or because
| the ecosystem has some conventions that allow the tooling to
| _detect_ the "current environment" better, and/or because
| the standard installer isn't itself a library that defaults
| to appearing in every environment and installing specifically
| for "its own" environment; and/or (possibly the biggest one)
| they don't have to worry about building and installing
| complex multi-language projects where the one they're using
| is just providing a binding.
|
| An important reason for using them is to _test deployment_ :
| if your code works in a venv that only has specific things
| installed (not just some default "sandbox" where you install
| everything), then you can be sure that you didn't _forget to
| list_ a dependency in the metadata. Plus you can test with
| ranges of versions of your dependencies and confirm which
| ones your library will work with.
|
| They're also convenient for making neat, isolated
| installations for applications. Pipx wraps pip and venv to do
| this, and as I understand it there's similarly uvx for uv.
| This is largely about making sure you avoid "interference",
| but also about pinning specific versions of dependencies. It
| also lowers the bar somewhat for your users: they still need
| to have a basic idea of what Python is and know that they
| have a suitable version of Python installed, but you can tell
| them to install Pipx if they don't have it and run a single
| install command, instead of having to wrestle with both pip
| and venv and then also know how to access the venv's console
| scripts that might not have been put on PATH.
| zahlman wrote:
| The hard part is figuring out what "merge" means for your use
| case. If there's a definite set of packages that should be in
| the environment that are all already at definite locations on
| the local drive, there are many possible approaches (copying,
| `.pth` files, hard links, and symlinks _should_ also work) to
| stitching together the venv you want. But you can 't just feed
| the individual package paths to Python, because `sys.path`
| entries are places that Python will look _for_ the top-level
| package folders (and top-level module `.py` files - which
| explains why), not the paths _to_ individual importable things.
|
| More importantly, at runtime you can only have one version of a
| given package, because the imports are resolved at runtime. Pip
| won't put multiple versions of the same library into the same
| environment normally; you can possibly force it to (or more
| likely, explicitly do it yourself) but then everyone that wants
| that library will find whichever version gets `import`ed and
| cached first, which will generally be whichever one is found
| first on `sys.path` when the first `import` statement is
| reached at runtime. (Yes, the problem is the same if you only
| have one venv in the first place, in the sense that your
| dependency graph could be unsolvable. But naively merging venvs
| could mean not noticing the problem until, at runtime,
| something tries to import something else, gets an incompatible
| version, and fails.)
| d0mine wrote:
| To create venv2 that uses venv1, define pep-735 dependencies
| groups in your pyproject.toml Specifically, a group can include
| other groups
|
| uv supports groups and can create venv with the desired group
| set
| https://docs.astral.sh/uv/concepts/dependencies/#development...
|
| For example, there can be "dev" group that includes "test",
| "mkdocs", "nuitka" groups (nuitka wants to be run with venv it
| builds binary for, so to keep venv minimal, it is in a separate
| group)
| nmstoker wrote:
| I must be missing something because I understood that pip cached
| packages already.
|
| Perhaps it still creates a copy of the package files in the
| virtual environment, thus the cache only saves repeated downloads
| and not local disk space. If that's the case then this does look
| really useful.
| zahlman wrote:
| Pip caches downloads, but in the sort of filesystem database
| that git uses. You can't even directly pull .whl files (nor
| unzipped folders) out of it. Every virtual environment gets its
| own, unzipped copy of the wheels represented in that database.
| So yes, it's only "saving repeated downloads". (And it will
| still talk to PyPI by default to see if there's a newer
| version. There's an outstanding issue to add a proper "offline
| mode": https://github.com/pypa/pip/issues/8057)
|
| It doesn't need to work like that. A lot of libraries would
| work fine directly from the wheel, because they're essentially
| renamed zip files and Python knows how to import from the
| archive contents directly. But this doesn't work if the package
| is supposed to come with any mutable data, nor if it tries to
| use ordinary file I/O for its immutable data (you're supposed
| to use a standard library helper for that, but awareness is
| poor and it's a hassle anyway). Long ago, the "egg" format
| expected you to set a "zip-safe" flag (via Setuptools, back
| when it was your actual packaging tool rather than just a
| behind-the-scenes helper wrapped in multiple backwards-
| compatibility layers) so that installers could choose to leave
| the archive zipped. But I don't know that Pip ever actually
| used that information, and it was easy to get wrong.
|
| But more importantly, the contents of the virtual environment
| could be referenced from a cache that contained actual wheels
| (packed or unpacked) by hard links, `.pth` files (with a slight
| performance hit) or symlinks (I'm pretty sure; haven't tested).
| yags wrote:
| Couple more resources:
|
| Github Repo: https://github.com/lmstudio-ai/venvstacks
|
| Blog post: https://lmstudio.ai/blog/venvstacks
|
| Docs: https://venvstacks.lmstudio.ai/
| dang wrote:
| Changed to the blog post from
| https://pypi.org/project/venvstacks/, since it gives more
| background. Thanks!
| Vaslo wrote:
| Why would this be an advantage to Astral's uv?
| woadwarrior01 wrote:
| None. FWIW, this exists because they wanted to ship a whole
| Python environment with an electron app.
| ewuhic wrote:
| Python venv tools are the new JS frameworks: every day some other
| ridiculous tool is born.
|
| And yes, everything is already solved with: due diligence (non-
| existent in scientific community) and nix.
| batch12 wrote:
| Can you clarify your second sentence? I'm having a hard time
| understanding the point you're making.
| ewuhic wrote:
| https://news.ycombinator.com/item?id=42032174
| angra_mainyu wrote:
| nix mentioned let's go
|
| In all seriousness, it is a bit tiresome when you come back to
| the python world and see a completely fragmented ecosystem
| (poetry, pdm, pip-tools, uv, and the older ways).
| whinvik wrote:
| But can it do GDAL.
|
| (This is the Doom equivalent joke for Python environments!)
| superkuh wrote:
| venv is admitting that there no longer is a Python language, only
| multitudes of pythons, one for each application you want to run.
| seamossfet wrote:
| It's crazy to me that in 2025 we still haven't figured out python
| dependency management
| kevin_thibedeau wrote:
| Debian had it figured out, then they had to go and ruin it all
| by hopping on the venv bandwagon.
| biorach wrote:
| In fairness, this is a response to a need which became common
| only relatively recently
| wizrrd wrote:
| How do venvstacks differ from PyInstaller and other bundlers?
| marban wrote:
| What's wrong with Pyenv & Poetry?
| dagw wrote:
| Nothing per se. But there are a few different workflows and
| project types they either don't support or make very difficult.
|
| If pyenv and poetry solve all your problems then it's a
| perfectly fine setup.
| v3ss0n wrote:
| Why not just use PDM
| skissane wrote:
| I watch my CI pipeline spend a minute or two creating the
| virtualenv and pip install into it several times a day, despite
| the fact that nothing has changed (no change to requirements.txt,
| etc), and I wish there was a well-supported way to package up a
| venv into some kind of artifact (keyed by Python version, host
| platform and package versions) so we only had to rebuild it if
| something changed. This sounds a bit like that, but also sounds
| like something more complicated than what I was looking for.
| nerdbaggy wrote:
| Gitlab CI can do that.
| https://docs.gitlab.com/ee/ci/caching/#compute-the-cache-key...
| smetj wrote:
| Put it in a container?
___________________________________________________________________
(page generated 2024-11-03 23:01 UTC)