[HN Gopher] Classifying Python virtual environment workflows
___________________________________________________________________
Classifying Python virtual environment workflows
Author : ingve
Score : 76 points
Date : 2023-01-07 12:29 UTC (10 hours ago)
(HTM) web link (snarky.ca)
(TXT) w3m dump (snarky.ca)
| computerfriend wrote:
| I hope the convention becomes to also consider ./venv as another
| default location.
| IanOzsvald wrote:
| Can other data scientists comment?
|
| I'm 15 years in with python and scientific work. For a lot of
| years I liked conda but then it got crazy slow. Next I started
| making conda environments and installing packages with pip. Now
| I'm experimenting with mamba ("fast conda") and that's pretty
| good.
|
| Conda envs mean I can experiment with different versions of
| Python (I'm a co author for O'Reilly's High Performance Python so
| eg 3.11 and 3.12 are pretty interesting right now). Conda
| "should" also make identical teaching environments (I teach my
| own courses). Pip was a pragmatic choice to get installations in
| minutes not hours in the years when conda was silly-slow.
|
| The above is also all for short -lived research work (my typical
| client mode for scientific work), so it is probably different to
| anyone doing long-run dev work, production deploys, or for those
| not needing non-Python binary support (eg GPU/C/Fortan lib
| support).
| mark_l_watson wrote:
| I am a Lisp programmer who has mostly used Python for deep
| learning. I found the long runtimes for some conda operations
| worth while in order to have several very different
| environments setup. I now try to use Google Colab for as much
| of my deep learning work as possible, so I have thought of
| dropping conda and try an alternative setup because I am
| writing a Python book that covers deep learning but also covers
| a lot of use cases that I usually use Common Lisp for. I will
| probably take the time to try most of the options in this blog
| article.
| kherrg wrote:
| Mamba is great! You won't find many self-promotional blog posts
| about it because it just works.
| KRAKRISMOTT wrote:
| Mamba doesn't work with optional dependencies (those in
| square braces e.g. pytorch[gpu])
| cinntaile wrote:
| > Conda "should" also make identical teaching environments (I
| teach my own courses).
|
| It's probably easier for your students if you provide
| containers for them, then everyone has an identical
| environment.
| longevyti wrote:
| > Now I'm experimenting with mamba ("fast conda") and that's
| pretty good.
|
| +1 for this. I really like Conda because I use a lot of
| packages that cannot be installed with only pip (E.g. GDAL).
| Conda would sometimes take 10 minutes to make an environment.
|
| Mamba has been an absolute game changer. It has rewritten parts
| of Conda in C++ and added multithreading, so every part of the
| process - the solving, download, and the extraction - is all
| Lightning quick in comparison. A 10 minute process with Conda
| can now take 2.5 mins with Mamba. It also helps that it is also
| prettier to look at!
| chazeon wrote:
| Agreed. I recently saw on a Mamba team's blog post saying
| that conda is also starting to incorporating libmamba, so
| probably the issue could be resolved someday.
| plonk wrote:
| I found that the best way to install GDAL was:
|
| - on Windows, using Christoph Gohlke's wheels,
|
| - on Linux, installing GDAL with the OS' package manager,
| then the Python bindings by fixing the version to $(gdal-
| config ---version) and setting C_INCLUDE_DIR and
| CPP_INCLUDE_DIR.
|
| It's a bit more involved, but the speed of pip and conda's
| multiple channels and dependency resolution issues make this
| worth it.
| Zizizizz wrote:
| My flow that I have settled on as being the most predicable (even
| though PDM allows for installing to a local virtualenv as well)
| is (with pyenv): mkdir newproject && cd
| newproject echo 3.11.0 > .python-version # (If different
| python version) python -m venv .venv # Make the venv
|
| And I have this in my `.zshrc` so whenever I `cd` to that
| directory, it auto-activates the virtual environment. Having the
| local directory in the project also makes it really easy for my
| editor (neovim or VS Code) to pick up export
| PYENV_ROOT="$HOME/.pyenv" command -v pyenv >/dev/null ||
| export PATH="$PYENV_ROOT/bin:$PATH" # Custom key-
| bindings python_venv() { MYVENV=./.venv # when
| you cd into a folder that contains $MYVENV [[ -d $MYVENV ]]
| && [[ ! $VIRTUAL_ENV ]] && source $MYVENV/bin/activate #
| when you cd into a folder that doesn't [[ ! -d $MYVENV ]] &&
| [[ $VIRTUAL_ENV ]] && deactivate } autoload -U add-
| zsh-hook add-zsh-hook chpwd python_venv python_venv
| giancarlostoro wrote:
| I really just run the same python module as you directly, its
| the most predictable across platforms.
| dmurray wrote:
| > # when you cd into a folder that doesn't
|
| Is this the behaviour you want, for subdirectories of your
| project root? Other tools (e.g. git, but also version managers
| like asdf) tend to look in the current directory, and then
| recursively try the parent directory to find what the current
| configuration should be.
| Zizizizz wrote:
| I don't really know the last time I cd'd to a subdirectory in
| a project, I typically just use fzf or nnn for any of those
| interactions. Not a bad idea though, thanks!
| truculent wrote:
| Do you have any issues with many virtualenvs/packages taking up
| a lot of disk space?
|
| For instance, numerical packages (scipy, torch, etc) can add up
| if they're duplicated across many projects, and, as I
| understand it, virtualenv doesn't do any caching/symlinking.
| fbdab103 wrote:
| >...virtualenv doesn't do any caching/symlinking.
|
| Well, one of us is incorrect then. I had been thinking for
| years that one of the key benefits was that virtualenv would
| not do a blind copy of files (except on Windows...sigh). The
| `venv` command even has a `symlinks` vs `copies` option.
|
| `python3 -m venv --help`
| Zizizizz wrote:
| I don't work on too many projects at a time and tend to
| archive and remove the virtualenv when I finish working on
| something for the foreseeable future. I can see why that
| would be a problem for some though.
| fake-name wrote:
| One option I never see mentioned in any of these discussions
| about virtualenvs/stuff is my approach to python environments:
| Each project gets it's own VM/Container.
|
| You then have one python version in the container, and no
| virtualenvs at all. All libraries get installed right into the
| main python libraries folder.
|
| I've been using this approach for 5+ years now, and I've yet to
| hit issues. I read people lamenting issues with python's
| packaging infrastructure, and with the approach I've never run
| into any of the issues they talk about.
|
| Pip complains a lot about being told to install stuff globally,
| but from what I can tell as long as you are careful enough to
| _never_ use distro-sourced python packages (they 're almost
| always out of date anyways, at least on ubuntu), it's never
| caused problems for me.
|
| For web-service type things, this has a lot of knock on effects
| that are beneficial. Each project winds up with it's own IP, if
| you need to restart things you can often just reboot the
| VM/container (assuming you have init scripts), process/library
| isolation issues are effectively impossible, etc...
| leksak wrote:
| Do you store your requirements using pip freeze then?
| fake-name wrote:
| Yep.
| methou wrote:
| same.
|
| but I'm still wishing for a native container support from
| pipenv/pyenv/poetry etc.
| mattbillenstein wrote:
| Separation of concerns...
| fbdab103 wrote:
| Looking for a suggestion from the audience here.
|
| I lean on Makefile for automation wherever possible, even if it
| is just a centralized trigger command to run a packaged script
| (eg scripts/dothething.py). One of my sticking points is how to
| ensure the Python virtual environment is installed and up to date
| before triggering any scripts. That is, I clone a new project, cd
| into it, and want to run some package command inside the expected
| environment (eg `poetry run scripts/dothething.py`). This will
| fail if the environment has not previously installed all of the
| defined dependencies. My annoyance is that requiring `poetry
| install` to run prior to every command is quite slow, and rarely
| has anything to do.
|
| Does anyone have a suggestion as to how to ensure the right thing
| is always done without the slow Poetry resolution? My dream is
| that in all of my projects you could, say `make test` and Python
| would optionally create the environment, install the
| dependencies, and run pytest in the fresh case. The typical
| development case would confirm that the environment exists and
| just run pytest.
| plonk wrote:
| As long as a venv exists and is activated, your Makefile can
| simply "pip install -r ./requirements.txt" and pip should make
| sure that all constraints in the requirements are met.
|
| Make sure that the package management tools are up-to-date
| first with "python -m pip install--upgrade pip setuptools
| wheel".
|
| Pip is very fast now and shouldn't take more than a second if
| all dependencies are already installed. The only part that's
| too slow is installing a package from source.
| 323 wrote:
| You can check that a venv exists by looking for
| .venv/pyvenv.cfg. Is that not enough? Unless you also need to
| check that all packages are installed and up to date.
| fbdab103 wrote:
| 1) Confirming that the environment is in sync with my
| definition file (pyproject.toml or lockfile) is my true
| objective. If I suddenly add libraryFoo to pyproject.toml,
| but I forget to run `poetry install`, I want my `make test`
| command to note the discrepancy and correct the situation.
|
| 2) I think that has some brittleness of detecting where the
| environment lives (as per the author's post). For my use
| case, I would be fine assuming the defaults, but it is
| something else to go wrong
|
| Edit: Thinking out loud, I guess the quickest solution is to
| checksum the pyproject.toml file and store that somewhere. If
| current hash matches previous, do not run `poetry install`.
| While that would get me there, I am hoping for a first-party,
| standard solution.
| 323 wrote:
| > I guess the quickest solution is to checksum the
| pyproject.toml
|
| This is the kind of stuff bazel/ninja do.
|
| https://bazel.build
|
| https://ninja-build.org
| molly0 wrote:
| Only reason I use virtual environments instead of just building
| everything inside Docker containers is the debug experience,
| which is not good enough inside a Docker container.
| cryvate1284 wrote:
| In my view, you'd normally still want to use a virtual
| environment or ignore the system site packages.
|
| The system will have a bunch a bunch of (site) packages already
| installed, and hence you lose control.
| TekMol wrote:
| How is debugging inside a container different than on the host
| os?
| molly0 wrote:
| Well to use something like "debugpy" It's the port mapping
| that just dosn't work straight out of the box, but needs
| custom settings inside my editor (and often times additional
| code inside the startup script) to get it working between the
| container and my local machine.
| TekMol wrote:
| It sounds like you want to do the debugging outside of the
| container?
|
| I just do it inside the container with pudb.
| Zizizizz wrote:
| How does your LSP pick up everything like imports if they
| only live in a docker container?
| 323 wrote:
| You run the LSP also inside the container and connect to it
| remotely.
|
| https://code.visualstudio.com/docs/devcontainers/containers
| Zizizizz wrote:
| Okay that makes sense, I don't really see how it's
| simpler but I completely see the use case. I was unclear
| on what he/she meant by the editor "just working" without
| having some config or plugin to point to the installed
| packages
| TekMol wrote:
| LSP?
| Zizizizz wrote:
| https://code.visualstudio.com/api/language-
| extensions/langua... Basically just your auto-completion,
| linting, quick-actions, etc...
| TekMol wrote:
| Why does it care if the application runs in a container?
|
| When you run a project in a container, you mount the
| projects directory into the container.
|
| That is nothing my editor cares about. It does not even
| know it. It works on the files locally on the host os and
| that's it. The the files are executed somewhere inside a
| container is none of the editors business.
| contravariant wrote:
| It doesn't technically care, but if your container is
| reporting that it has hit a breakpoint in file
| /mnt/some_module/script.py, then your LSP / debugger /
| editor is not going to know what it is talking about
| without some more info.
| TekMol wrote:
| I do debugging inside the container with pudb.
| TekMol wrote:
| Are virtual environments still needed now that we have
| containers?
|
| I just run everything in a container and never need a virtual
| environment. Am I missing out on anything?
| z3c0 wrote:
| I use both. Containers for server code, environments for
| scripting. Containerizing scripts is a bit excessive,
| especially if it's only going to run on your machine.
| tsbischof wrote:
| Not all Python users run containers, especially in the
| sciences. When you are teaching basics of data manipulation to
| people that have barely touched any programming language
| before, getting people to grok even virtual environments is not
| trivial.
| TekMol wrote:
| To me, containers are simpler than virtual environments.
|
| Just do "docker run --rm -it debian:11" and you are ready to
| go.
| alisonatwork wrote:
| This is the version of containers where they are most
| useful - a clean, throwaway environment with exactly the
| one application or userspace you want to test inside it.
|
| Unfortunately as a developer it's more fiddly than that.
|
| You need to mount your code into it. Then if you try commit
| changes to your code, you're a different user, so the
| commit is now messed up. Also you lost your dotfiles, so
| none of your aliases or other shortcuts work when you're in
| the container. You might need to connect to other services
| which now need to run in their own containers, and then
| they all need to talk to each other. Before you know it
| there's a whole litany of commands you need to run and
| configuration files you need to put together just to get
| started in the morning. Imo using containers on development
| machines is a big step backwards for developer
| productivity.
|
| Virtual environment is just... cd in the directory. Run
| magic incantation to set that directory as the canonical
| one for your current shell. The end. IDEs will pick it up
| automatically.
|
| The Python virtual environment and package management
| experience is still worse than pretty much any other
| programming language, but for me it's far and away better
| than using containers on a development machine.
| TekMol wrote:
| You make it sound like you have to edit your code inside
| the container.
|
| I don't do that. I edit it on the host OS.
|
| I just _run_ the code inside the container.
| zzzeek wrote:
| Do you have to mount the location of the code as a volume
| in the container also?
|
| I use containers regularly but adding them to a
| development workflow for a standalone python application
| I could just as easily run directly without having the
| additional complexity of navigating container boundaries
| always mystifies me.
| TekMol wrote:
| Yes, I have one terminal window in which I run / use the
| code. I start it like this:
|
| docker run --rm -it -v $(pwd):/some/where debian:11
|
| Most of the time I don't use plain Debian, but an image
| based on Debian which I have peppered with some
| convenience stuff.
| fbdab103 wrote:
| At $WORK, I more or less can only develop on a Windows machine.
| Getting Docker approval is needlessly painful. Until Docker and
| WSL are Microsoft standard tools, then yes, these solutions are
| required.
| Kwpolska wrote:
| How is WSL not a Microsoft standard tool? What makes it
| different from Visual Studio or VSCode or whatever, which is
| also never going to be built into Windows? Why can you get
| Python installed but not WSL?
| 323 wrote:
| WSL is a full Linux/Ubuntu?
|
| So you can install whatever you can apt-get/wget into it?
|
| I can see that being a problem in restricted environments.
| Kwpolska wrote:
| For a software developer, who can run any software they
| want (because they need to be able to run the software
| they're paid to work on), including software that could
| subvert all the restrictions anyway?
| fbdab103 wrote:
| Clearly you are not familiar with corporate IT and
| "security" compliance.
| Kwpolska wrote:
| I kinda am (though luckily not subject to it), and I find
| it laughable.
| molly0 wrote:
| For me it's the debug experience that is lacking when only
| using containers, I have some work arounds but they are still
| not good enough.
| TekMol wrote:
| Can you give an example?
|
| I don't experience any difference, whether I am inside a
| container or on the host os.
| pnt12 wrote:
| Can you debug in a container, eg VS code debugger with
| breakpoints and steps?
| crucialfelix wrote:
| yes, devcontainers work very well. Full debugger. It's
| also possible to run the container in the cloud and still
| get a full local debugger.
|
| I did have some issues with some projects and want to
| revisit this soon.
| TekMol wrote:
| I have no experience with that as I use pudb for
| debugging, which I simply run inside the container.
| 0x008 wrote:
| Yea it is a supported use case. VS code remote extension
| can work with every container just like remote ssh
| development.
| marban wrote:
| I've settled with: Pyenv > python venv > Poetry
| nszceta wrote:
| CFLAGS="-march=native -mtune=native" CONFIGURE_OPTS="--enable-
| optimizations --with-lto=full" pyenv install 3.11.1 --verbose
| thenipper wrote:
| Same. Though I'm debating swapping out poetry with pip tools.
| tomwojcik wrote:
| I started using poetry for my OSS projects and I'm happy with
| it, but I'd never use it for closed source projects. Poetry
| simplifies deps, config, pkg build etc as it's stored in a
| single place, but it's way to opinionated. For managing
| dependencies _only_ , pip tools is easier to use and
| customize. There's no breaking changes, it's a finished
| product.
|
| Also, I don't want my CI to fail randomly.
| https://github.com/python-poetry/poetry/pull/6297
|
| Look at the number of issues in poetry. Some things are not
| backwards compat, some are broken. If you want reliable, KISS
| with pip tools.
|
| That's my opinion.
| plonk wrote:
| Agreed. We tried using poetry in a monorepo that contains
| dozens of projects and it seemed annoying to get right.
| Poetry seems to want only one main project folder?
| Zizizizz wrote:
| Have you used pip-tools before? I have used PDM instead of
| Poetry and was very impressed
|
| https://github.com/pdm-project/pdm
| thenipper wrote:
| I have. At my last gig at Shopify we used it to manage our
| requirements since poetry didn't play nice with some
| internal dev tools.
|
| I've tried PDM as well. It's definitely nice. Part of me
| just wants to avoid "all encompassing" tools and use a
| specific tool for a specific job.
| jessekv wrote:
| For what its worth, I evaluated pdm, poetry, and pyenv
| before settling on pip-tools for my workflow. pdm is the
| only one I would consider trying again (once it is more
| mature).
| marius_k wrote:
| My flow/boilerplate for apps: ./scripts/create-
| venv.sh # create venv using specific python version,
| update pip, wheel, setuptools, install requirements.txt
| ./scripts/freeze-requirements.sh # create tmp venv with
| requirements-base.txt and pip freeze > requirements.txt
| ./requirements-base.txt # dependencies with loose
| versions ./requirements.txt # frozen/locked
| dependencies
| brundolf wrote:
| Thinking about how python is the only ecosystem where you need to
| deal with any of this, and how many thousands of developer-hours
| are spent on it
| maliker wrote:
| It's extremely annoying. The reality is, python can deploy or
| isolate any way you'd like, but because there isn't one
| canonical way to do it, everyone comes up with their own
| system. My best guess about why this is the case is because
| python is over 30 years old, so they've reinvented the
| packaging wheel (ha, .whl) four of five times, leading to
| massive confusion.
| chazeon wrote:
| Well python is also the only language have a full set of
| numerical / scientific / data-science software that is well-
| maintained. Seeing the Gigs of storage used by Rust / node
| without building any of more complex libraries, I cannot
| imagine how much more they will use when doing scientific /
| data stuff...
| arturkane7 wrote:
| Good point. I am wondering what are the specific reasons say
| Java or JS don't suffer from this, and hoping for rant-free
| responses that don't turn into language wars.
| brundolf wrote:
| A key thing for Node is that NPM (in addition to being first-
| party and having a standard manifest format) installs all
| dependencies in the _project_ directory, under node_modules
|
| People complain about node_modules, but the benefit is that
| every Node project on a system is isolated automatically and
| can re-download all of its dependencies trivially (after
| cleaning them, or after being checked out onto a new machine)
|
| A Node project's system-wide dependencies are:
|
| 1. A new-enough Node installation, and
|
| ...that's it
|
| The same is true for Go and Rust and other modern languages.
| Python is the odd one out.
| melony wrote:
| Go, Rust, and JS are unique in having package management
| solutions that prevents the diamond dependency problem.
| Most other languages suffer from the same problems as
| python in that they cannot have more than one version of
| the same package globally. The dependency resolution
| algorithms for Ruby/Dart/Julia are all NP class and require
| constraint solving which often fails to find a solution if
| your dependencies are complex.
| nonethewiser wrote:
| For JS, which I'm intepreti as node because I think that's
| the real parallel, the answer is NVN and NPM. Or alternatives
| such as yarn or pnpm. But even the default package manager is
| great compared to python.
|
| I havent tried it but my understanding is poetry is a bit
| more like npm.
| 63 wrote:
| To be fair, trying to figure out Maven and proper Java IDE
| integration (which is essential) can also be pretty
| miserable. Even though there aren't as many tool choices to
| deal with, the ones that exist can get really complicated and
| have poor documentation.
| IgorPartola wrote:
| In part because Python is often used as an OS language
| (Debian has it in the base system and a bunch of OS
| subsystems use it) so if you upgrade some library at the OS
| level it affects more than just your own code.
|
| In part it is because Python has modules, not libraries.
| Modules can have executables bundled with them.
|
| In part, it's because Python modules packages don't exactly
| care about backwards compatibility. By and large, people who
| put out packages will keep a changelog but not keep the same
| API going for years. So slightly different version of the
| same package can give you different results.
|
| Lastly it is because Python does not have a clear definition
| of a project. If at the language level I could say "this is
| the root of my project. Do not use any OS-level libraries.
| Here are my library dependencies and they are to be stores in
| $PROJECT_ROOT/libs/" then none of this would be required. I
| have rarely found that the version of Python itself was a big
| problem for me, but having the ability for multiple versions
| to be installed at the OS level combined with the above
| mechanism would entirely eliminate the need for virtual envs.
| Kwpolska wrote:
| I would say that one important factor is the tooling. Instead
| of having one or two standard tools, as seen in Java
| (maven/gradle), Node (npm/yarn), .NET (dotnet CLI and
| MSBuild/maybe FAKE), Python has a plethora of opinionated
| tools. venv+pip is built-in, but doesn't handle any of the
| "where to put the venv" parts, and doesn't handle making
| packages installable (and the classic solution is
| setuptools). There are many competing projects, like pipenv
| (which does not work at all for libraries), poetry (which
| uses a non-standard way of specifying metadata and
| dependencies, and isn't too friendly with the rest of the
| packaging world), flit, hatch, etc.
|
| Why are there so many competing tools? Why can't the
| community just pick one and make it a good tool for _all_
| use-cases? There's an organization called "Python Packaging
| Authority", and it maintains almost all tools mentioned in
| the previous paragraph (except the stdlib venv, and poetry).
| If they're an authority, they should just say "___ is the way
| to go, we'll add the missing use-cases to it, here's a tool
| to migrate everything else to ___". Instead, they have done
| things that support the proliferation of tools, like the PEP
| 517 (which is a standard API for package managers to talk to
| build tools).
|
| Another deficiency in the Python package management system is
| the fact that you need virtual environments to get things
| done, and that they are often finicky. Some of the modern
| tools hide the venv from you, but this gets less practical if
| you're trying to use things from within scripts, or trying to
| point your WSGI server at the venv, or in multi-user
| scenarios. Some of the modern tools put it in .venv and
| manage it for you, but venvs can randomly break in cases the
| tool might not be aware of (eg. in some system package
| managers, upgrading Python to a new minor version would cause
| venvs to break, because symlink targets moved.)
|
| Node solves this by having a `node_modules` directory. .NET
| does stuff at build-time that most people don't need to care
| about, and then just puts the .dll files for your
| dependencies next to the .dll with your thing, and the loader
| looks in the directory your code came from. Python has a
| proposal for `__pypackages__` [0], but it's been there since
| May 2018 with not much progress.
|
| [0] https://peps.python.org/pep-0582/
| chazeon wrote:
| For scientific calculation I just use mamba (sometimes
| micromamba) + managed environment in home folder. It is
| impossible to keep an environment for each project due to the
| library size and storage limitation. These I keep one copy each
| machine (including on HPCs and local machines). For bootstraping,
| I keep a copy of the environment.yaml file.
|
| I used to have a dedicated folder to manage these environments
| myself, but seeing conda/mamba does a good job managing these;
| and they are also easier to load with command, so I switched to
| the managed approach.
|
| Conda does not only work for Python, but many software necessary
| for our workflow that are written in C/C++/Fortran has been
| precompiled and uploaded to the conda-forge index as well.
___________________________________________________________________
(page generated 2023-01-07 23:01 UTC)