[HN Gopher] Pyenv - lets you easily switch between multiple vers...
___________________________________________________________________
Pyenv - lets you easily switch between multiple versions of Python
Author : punnerud
Score : 225 points
Date : 2024-03-25 08:34 UTC (14 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jphoward wrote:
| Is anyone is the AI/ML area finding success with anything other
| than conda, where installation of CUDA/CUDnn is required?
| Although I often have to pip install a lot of packages, I find
| conda's nvidia/pytorch/conda-forge channels are still by far the
| easiest way to get a deep learning stack up and running, and so I
| just stick with conda environments. I've tried poetry in the past
| but getting the NVidia deep learning stack up and running was
| really tough.
| halflings wrote:
| uv has been really awesome as a replacement for pip:
| https://github.com/astral-sh/uv
|
| So fast it finally made virtual environments usable for me. But
| it's not (yet) a full replacement for conda, e.g. it won't
| install things outside of Python packages
| northzen wrote:
| How about prefix then? https://prefix.dev/blog/uv_in_pixi
| thangngoc89 wrote:
| I use poetry and direnv. Coming from node/npm, it feels natural
| for me to just do this. I have really no troubles of installing
| Pytorch with poetry
| maleldil wrote:
| How are you installing Pytorch with CUDA with Poetry? I
| stopped using Poetry because it wouldn't automatically get
| the CUDA version; instead, it would install the CPU version.
| I migrated to PDM, which does the right thing.
| claytonjy wrote:
| was this before torch 2.0? With the very notable exceptions
| of a few mispackaged versions, torch now includes all the
| relevant Nvidia libs, and I haven't seen it grab the CPU
| version on a GPU box yet, though I'm not sure what it looks
| for.
|
| A notable open issue in poetry is we can't currently
| specify one dependency on torch, and have it grab CPU
| version on some systems and GPU on others. Does PDM solve
| that?
| thangngoc89 wrote:
| Before CUDA 12.0 you have to specify a field in
| pyproject.toml like this
| [tool.poetry.dependencies] python =
| ">=3.10,<3.12.0" torch = {version = "^2.0.1+cu118",
| source = "torch118"} torchvision = {version =
| "^0.15.2+cu118", source = "torch118"}
| [[tool.poetry.source]] name = "torch118"
| url = "https://download.pytorch.org/whl/cu118"
| priority = "explicit"
|
| However, since CUDA 12.0 and Pytorch 2.1.0, just install
| like normal poetry add torch torchvision
| rmnclmnt wrote:
| For anything related to CUDA/CuDNN, use one of NVIDIA base
| Docker images. Then whether you use Conda / Pip / Poetry /
| Pipenv does not matter much. Not at all a Conda fan myself and
| avoid it like the plague
| Y_Y wrote:
| What's surprising to me is that this isn't better known. The
| only reliable solution I've found is to go with the pytorch
| or deepstream images from NGC. Conda is probably a good idea
| for noobs who need Cuda installed for them on windows, but
| otherwise I find it an endless source of finicky issues,
| especially for unsavvy ML scientists who are looking for a
| silver bullet for package management.
|
| This link shows which package versions come in which Docker
| tag and is invaluable:
| https://docs.nvidia.com/deeplearning/frameworks/support-
| matr...
| rmnclmnt wrote:
| 10 years ago, << Data Science >> work past the experimental
| stage was performed by SWE with a knack for applied maths.
| So investing in tooling to do things properly was a given.
|
| Nowadays, most DS people only want to do ML at the
| experimental stage only and get lost when things get on the
| engineering side of things. But for their defense, nowadays
| the bare minimum skills require to do programming,
| containerization, CI/CD, etc. More experienced and swiss
| army knife SWE/MLE have to educate the willing.
|
| It was already the same 10 years ago with MATLAB dudes not
| wanting to get dirty with C/C++/ASM SIMD. The history
| repeats itself, only at a faster pace
| godelski wrote:
| Yes. I simply do python -m pip install torch
| torchvision
|
| and it works. It used to not, but it's been fine for me for
| about a year now.
|
| There's a very good chance I've installed cuda on my system
| before this though. And usually cudnn and some other packages
| because this is part of my standard install. And then I also
| never run into the issue where a package is looking for nvcc.
| michaelmior wrote:
| I use pipenv and I've found it to be much more usable than
| conda. For my use cases, it's generally faster and I've run
| into fewer dependency issues.
| misnome wrote:
| I love poetry but have found it pretty hard once you move off
| of anything that doesn't manage to get wheels on pypi.
|
| We make extensive use of conda/mamba to solve this, and are
| pretty happy with it, especially with conda-forge.
| samus wrote:
| Pyenv just worked for me. I am actually using Fedora Silverblue
| and have GCC and the CUDA SDK available only inside a toolbox
| container. Therefore, I have to enter that toolbox to install
| things like FlashAttention.
| claytonjy wrote:
| I have successfully transitioned an ML/AI team of seasoned
| researchers away from conda and to poetry. Some also use pyenv,
| I suspect a lot don't bother but may get bitten eventually.
|
| It's definitely a learning curve, but it turns out every conda
| user has been bit by the irreproducible tendencies of conda
| quite often. Nobody uses the conda env file, they just start an
| env and pip install things into it. They don't realize the base
| env has stuff, too, and conda envs are hierarchical rather than
| isolated. I know it's possible to use conda in an isolated and
| reproducible way, but have yet to meet someone that does so.
|
| So it hasn't been hard to pitch poetry to these folks, and
| while many complain about the learning curve they appreciate
| the outcomes.
|
| We're a pytorch shop, and torch mostly just works with pip or
| poetry these days, as long as you skip the versions the torch
| maintainers mispackaged. We rarely need anything higher-level
| that only conda could install.
|
| We really like having more than two dependency groups as this
| allows us to keep research and production in the same
| repository. main, dev, research. Then researchers contribute to
| the core library of a project and keep research and production
| using the same code for running and evaluating models.
| abkfenris wrote:
| Have you tried https://pixi.sh/ ? It brings Cargo/NPM/Poetry
| like commands and lock files to the Conda ecosystem, and now
| can manage and lock PyPI dependencies alongside by using uv
| under the hood.
|
| I haven't been using anything CUDA, but the scientific
| geospatial stack is often a similar mess to install, and it's
| been handling it really well.
| blowski wrote:
| OP - I'm curious why you've submitted this today, you seem
| experienced enough to guess this is not the first time you've
| used it. Has there been a major change or something happened in
| the community I need to know about? Or are you just spreading the
| love for a great piece of open source software?
| dallyo wrote:
| Also a bit confused as to why this is currently the #1 post.
| BiteCode_dev wrote:
| Because it's not a solved problem, pyenv is niche and people
| discover it once again and think that they finally solved
| their pain.
|
| They will try it, and most of them will pay the price for it.
|
| It's a cycle.
| airstrike wrote:
| Cue ominous Reaper sounds
| slater wrote:
| username checks out
| punnerud wrote:
| I used Python a lot, but missed pyenv. Used it for some weeks
| now and was thinking this is something I would like to know
| about earlier (thats normally when I post it to HN).
| xeromal wrote:
| There are always new developers coming onboard that might have
| missed this handy tool. It's worth firing up the conversation
| on occasion.
| recursive wrote:
| I've used python lightly for a few decades now. I've never got
| far enough in to figure out how packages work. Once, at least
| 10 years ago, I briefly tried to use some published packages. I
| never figure out how to get it to work at all. Other than that,
| I've limited myself to what's available in the standard
| library. I've seen a lot of conflicting information about how
| python packages work, or should work. If there was a "one true
| way", I'd be interested. I don't want to have to compare a list
| of pros and cons. I want one officially blessed tool. I'll live
| with the constraints.
| BiteCode_dev wrote:
| I have to warn again users that think they have found the silver
| bullet that pyenv comes with a big caveat: it compiles python on
| your machine.
|
| The number of possible modes of failure in this situation is
| huge.
|
| See also: "Why not tell people to "simply" use pyenv, poetry or
| anaconda"
|
| https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
|
| I'm not saying pyenv is not a useful tool, but it is not a tool
| for beginners fighting with python packaging problems. It's a
| specialist tool to normalize your setup.
|
| Very often, I see people tell me they don't have problems with
| pyenv, but later on have other unrelated problems with their
| dependencies. Analysis then prove it was because of pyenv, they
| just didn't know it. The cost is not obvious.
| bmitc wrote:
| > it compiles python on your machine
|
| What is the alternative? Every solution that I know of on Linux
| requires you to build Python on the machine: asdf, official
| Python downloads, etc.
| Leynos wrote:
| Rye downloads pre-built binaries.
|
| I'll leave further discussion of the reliability and
| provenance of said binaries to someone else.
| BiteCode_dev wrote:
| Binaries are quite reliable and from a serious project, but
| rye is young and uses shim.
|
| Still, it's part of those new generation of tooling that
| bringing hope for the next few years.
| bheadmaster wrote:
| > What is the alternative?
|
| Using per-distro official installation channels.
|
| E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
| kzrdude wrote:
| These builds are an alternative:
| https://github.com/indygreg/python-build-standalone
|
| Those are what Rye and hatch use.
|
| Drawbacks: late availability of patch versions, various
| quirks from how they are built (missing readline, missing
| some build info that self-compiled C python modules might
| need.)
| formerly_proven wrote:
| I think PEP711 (https://peps.python.org/pep-0711/) is
| (eventually) a better alternative, because it builds on top
| of the proven manylinux approach to binary compatibility.
| kzrdude wrote:
| Those are proof of concept builds now and will eventually
| hopefully replace indygreg's builds.
|
| Not only does the format need to exist but the service of
| building and publishing them is needed too.
| codethief wrote:
| Very cool, this is the first time I'm hearing of PEP 711!
| I hope this (or some other PEP like it) will get accepted
| eventually!
| BiteCode_dev wrote:
| I hope this will be merged with indygreg builds at some
| point, but the community move slowly on those issues so
| it may take years.
| BiteCode_dev wrote:
| For now most people would do well to stick to python.org
| installers for mac and windows.
|
| For linux, official repos are ideal. If you really, really
| can't (which is different than wanting to), ubuntu deadsnake
| and red hat epl are the best second plan, while already more
| finicky.
|
| If you use something more exotic, you chose hardship, and you
| will have to be up to the task.
|
| Anything else will come with bigger caveats than the people
| promoting them will really admit to.
|
| The story is of course a bit richer because linux packaging
| is fun, so I'll complete with:
|
| https://www.bitecode.dev/p/installing-python-the-bare-
| minimu...
|
| Thete are currently no silver bullet to bootstrap python on
| linux. I tried all of them with hundred of colleagues and
| trainees.
|
| I do have hope for astral to come up with one but don't run
| on rye thinking your life is saved.
| rbanffy wrote:
| Even on Macs, you can use MacPorts. They provide a lot of
| versions pre-compiled (and everything is BSD-solid).
| BiteCode_dev wrote:
| I know you mean well, but such advice is why so many
| beginners have painful Python experience.
|
| Macports and homebrew Pythons are dependencies of other
| packages. They can be used by you, but they are not meant
| for you.
|
| This means at some point, they will contain a surprise,
| and not a good one.
| hot_gril wrote:
| Personally I only use MacPorts or Homebrew to install
| random smaller tools. It's easier to install Python,
| NodeJS, Postgres, etc binaries from the main website.
| rbanffy wrote:
| > Every solution that I know of on Linux requires you to
| build Python on the machine
|
| Unless you need a Python that's not supported by your Linux
| distribution, you can just use what's available.
|
| On macOS, MacPorts provides compiled versions for 3.2 all the
| way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8,
| 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact
| it's not Linux (or x86) might cause some frustration.
| SOLAR_FIELDS wrote:
| If you're lucky enough to be on a Linux system that uses
| apt some thankless soul maintains a repo called deadsnakes
| with all these binaries. Fabulous if you're using any
| somewhat old version of Python in CI for instance. Yum
| based systems are SOL as far as I can tell. Build and host
| your own binary for that. Apk doesn't have this either IIRC
| rbanffy wrote:
| If you are deploying to such an ancient OS, it's perhaps
| easier to have the whole OS packaged as a container or a
| VM and use that. It's not only different Python versions
| that might bite you.
| Hackbraten wrote:
| What makes you think that GP's comment is talking about
| older OSes?
|
| My understanding of their comment is that they're talking
| about getting older Python interpreters to run on more
| _modern_ OSes, modern enough that they don't carry the
| older Python as a system package anymore. Hence,
| deadsnakes.
| hermitdev wrote:
| deadsnakes is useful, but also a bit of a misnomer. One
| can use it to get pretty much anything python version,
| even ones that are too new to be available in your
| default apt repos. That is, it's not just for 'dead'/EOL
| versions.
| BiteCode_dev wrote:
| The thankless soul is the awesome Anthony Sottile, who is
| also the author of pre-commit.ci:
| https://twitter.com/codewithanthony
|
| Drop by and send thanks cause he really needs them.
| bmitc wrote:
| That's exactly why I don't really on that for Ubuntu
| though. It's a single point of failure, when I can just
| use `asdf` to install Python and easily switch between
| it, automatically if setup correctly on projects.
| tgz wrote:
| try pyoxidizer
| rcarmo wrote:
| Actually, it only builds it locally if it can't find a pre-
| packaged version for your system/arch. Admittedly that's most
| of the recent ones on a Mac, but there is a difference (I've
| been using pyenv for nearly ten years[1] now).
|
| The big advantage for me is that I can match whatever runtime
| and standard library a target has (and yes, that's needed more
| times than not, even in this new age of Docker).
|
| Additionally, you can build an _optimized_ Python. I have this
| set for my builds: env
| PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto'
| PYTHON_CFLAGS='-march=native -mtune=native' pyenv install
| 3.12.2
|
| [1]: https://taoofmac.com/space/blog/2015/10/03/1245
| BiteCode_dev wrote:
| That happens way more often than people want to admit, and I
| think it's not genuine to present the tool that way.
|
| E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard
| machine, I'll get: pyenv install 3.9 -v
| /tmp/python-build.20240325124651.73089 ~ Downloading
| Python-3.9.19.tar.xz... ->
| https://www.python.org/ftp/python/3.9.19/Python-3.9.19.tar.xz
| ... LD_LIBRARY_PATH=/tmp/python-
| build.20240325124651.73089/Python-3.9.19 CC='gcc -pthread'
| LDSHARED='gcc -pthread -shared
| -L/home/user/.pyenv/versions/3.9.19/lib
| -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib
| -L/home/user/.pyenv/versions/3.9.19/lib
| -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib '
| OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES=''
| _TCLTK_LIBS='' ./python -E ./setup.py build running
| build running build_ext ... etc
|
| There are obviously binaries for this Python, since I can apt
| install it and I didn't specify the minor version.
|
| But by default it downloads the source, and compiles it.
|
| On top of that it will use a shim which comes with its own
| world of possible pain.
|
| Again, I don't want to bash on pyenv.
|
| But I do want to lower people's expectations.
|
| It's a tool for experts, not beginners.
| orf wrote:
| I've quite literally never seen it download a pre-compiled
| Python - I didn't actually know it had that functionality,
| and have been using it for probably a decade.
| cqqxo4zV46cp wrote:
| Ditto. I had no idea that it ever looked for a pre-
| compiled version.
| ejj28 wrote:
| Actually, I'm pretty sure that binaries for Python 3.9
| aren't available on Ubuntu 20.04 without adding an apt
| repository to get them from, so your statement that you
| "can apt install it" isn't entirely correct in this case -
| by default, binaries aren't available, and therefore pyenv
| has to compile Python 3.9.
| BiteCode_dev wrote:
| Just tried, I get the same result with 3.7, 3.8, 3.10 and
| 3.11.
| notatallshaw wrote:
| > Actually, it only builds it locally if it can't find a pre-
| packaged version for your system/arch. Admittedly that's most
| of the recent ones on a Mac, but there is a difference (I've
| been using pyenv for nearly ten years[1] now).
|
| Really? I regularly use this on Linux exactly because it
| compiles from source rather than using the system provided
| Python which often have patches in them that breaks pip's
| test suite.
|
| I've never encountered it trying to use the system provided
| Python, but maybe there's a weird quirk in the way I am using
| it.
| strangelove026 wrote:
| Currently going through this myself on a new mac I bought
| lol. Still haven't gotten anywhere. Oh well
|
| $ PYTHON_CONFIGURE_OPTS='--disable-ipv6 --enable-
| optimizations --with-lto' PYTHON_CFLAGS='-march=native
| -mtune=native' pyenv install 3.12.2 python-build: use
| openssl@3 from homebrew python-build: use readline from
| homebrew Downloading Python-3.12.2.tar.xz... -> https://www.p
| ython.org/ftp/python/3.12.2/Python-3.12.2.tar.x... Installing
| Python-3.12.2... python-build: use readline from homebrew
| python-build: use ncurses from homebrew python-build: use
| zlib from xcode sdk
|
| BUILD FAILED (OS X 14.4 using python-build 20180424)
|
| Inspect or clean up the working tree at
| /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-
| build.20240325114837.43577 Results logged to
| /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-
| build.20240325114837.43577.log
|
| Last 10 log lines: ./Include/internal/pycore_interp.h:193:24:
| error: field has incomplete type 'struct _dtoa_state' struct
| _dtoa_state dtoa; ^
| ./Include/internal/pycore_interp.h:193:12: note: forward
| declaration of 'struct _dtoa_state' struct _dtoa_state dtoa;
| ^ 1 error generated. make[2]: ** [Objects/boolobject.o] Error
| 1 make[1]: ** [profile-gen-stamp] Error 2 make: ** [profile-
| run-stamp] Error 2
| strangelove026 wrote:
| Actually, I installed rye and got everything working. There
| goes any usage of pyenv going forward lol.
| wodenokoto wrote:
| I generally like bite codes newsletter but this one was a miss
| for me.
|
| I'm sure he is trying to promote some sort of work flow in that
| article but I don't understand which.
| mikemcquaid wrote:
| If you use Homebrew: you can use 'brew pyenv-sync' to use
| Homebrew's pythons with pyenv. Similar commands are available
| for rbenv/nodenv (which always feel like it is missing an 'e'
| to me)
| lb4r wrote:
| > See also: "Why not tell people to "simply" use pyenv, poetry
| or anaconda"
|
| > https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
|
| Just curious: what are the downsides of poetry installed with
| pipx? The article mentions having to install poetry in another
| venv, but that's hardly an issue with pipx (you just add an 'x'
| after 'pip'), and installing pipx is as straight-forward as it
| can be.
| chrisfinazzo wrote:
| I eventually landed on pipx as fighting with Pyenv and
| Anaconda - via Miniconda - was an exercise in frustration.
| There's some mucking about in `$HOME/.local`, but this is
| mostly self-contained and not a huge chore to keep running.
|
| Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid -
| installing a entirely separate package manager just to deal
| with a few projects felt like the wrong thing to do.
|
| Occasionally, I have still needed to compile Python myself in
| order to get things to work, which isn't guaranteed not to
| blow up w/ `brew`, but this has become far less common of
| late.
| bootsmann wrote:
| Agreed pipx solves a lot of packaging issues with no downside
| to speak of. Not just with poetry but also for tools like
| virtualenv, ruff and black and non-dev command line tools.
| mwexler wrote:
| So, what is "the tool for beginners fighting with python
| packaging problems"?
|
| That is, the pattern seems to be that someone mentions a
| solution, then a zillion responses as to why it sucks.
|
| Is there any tool or pair that sucks least for most cases and
| beginners? I get that every case is different, but perhaps
| there are some useful starting points?
| IshKebab wrote:
| There isn't one. Python distribution and packaging is just
| fundamentally horribly broken.
|
| I think his point was that you shouldn't pretend to users
| that just switching to pyenv is the solution.
| notatallshaw wrote:
| > Python distribution and packaging is just fundamentally
| horribly broken
|
| It's clearly not because most people successfully use it
| fine.
|
| The problem of distribution and packaging is often a matter
| of user expectations vs. the actual problem.
|
| The user expectations is that Python is a high level
| language and will run the same across different machines
| regardless of OS and Hardware.
|
| The actual problem is Python is a glue language often
| depending on lots of libraries that are sensitive to how
| they were compiled and what hardware they are targeting. So
| people can end up moving the the glue of the project first
| and then expect everything else to just work.
|
| Things are getting better (e.g.
| https://lukeplant.me.uk/blog/posts/python-packaging-must-
| be-...), a lot of people have put a lot of work into
| standards and improving the ecosystem, but it is a hard
| problem that most other popular languages don't interface
| with nearly as much.
| Capricorn2481 wrote:
| I don't disagree with what you're saying but that article
| seems odd. It's just a story of someone installing
| something once that didn't break immediately. Not only is
| it anecdotal, it doesn't even seem to confirm whether the
| pip story is getting better or if they just got lucky.
| notatallshaw wrote:
| I agree, but there is no large scale study on this.
|
| As someone who has managed Python distributions in a
| large company and who triages issues on the pip github
| issue page that my anecdotal experience is things are
| getting better.
|
| The only hard statistic I can point to is the number of
| top packages on PyPI that offer wheels has substantially
| gone up, and is close to 100% in the top 500.
| hot_gril wrote:
| They use it fine by using Docker. So many major Python
| repos on GitHub come with a Dockerfile compared to, say,
| NodeJS. It's unfortunate, but having dealt with Python
| packages before, I don't blame them.
| notatallshaw wrote:
| Lots of analysts, data scientists, traders, engineers,
| etc., use Python third party packages successfully, and
| have never touched, or maybe even head of, Docker.
|
| And yeah, in general there are significantly less NodeJS
| third party packages interfacing with packages that
| directly depend on OSes and hardware. Python has many
| third party packages that are older than NodeJS that
| depend on foreign function interfaces, win32 com apis,
| directly talking to graphics shaders, etc.
| hot_gril wrote:
| Python packages are painful even if nothing native is
| involved. There are NodeJS packages that rely on native
| code too, difference is the packaging system is a lot
| simpler to use, though it's starting to get worse with
| the `import` vs `require` nonsense and Typescript.
| IshKebab wrote:
| > It's clearly not because most people successfully use
| it fine.
|
| Well, no because it's perfectly possible to successfully
| use a horribly broken system. I use Python
| "successfully", it just meant I have spend probably
| literal weeks of my life fighting pip and virtualenv and
| relative imports and finding I need flags like `--config-
| settings editable_mode=compat`.
|
| > The problem of distribution and packaging is often a
| matter of user expectations vs. the actual problem.
|
| Ha yes, I expect it to work reliably and simply and it
| doesn't!
|
| > The actual problem is Python is a glue language often
| depending on lots of libraries that are sensitive to how
| they were compiled and what hardware they are targeting.
|
| That's completely irrelevant to the kind of problems I
| was talking about. I run into issues with Python failing
| to compile C libraries relatively rarely! Even compiling
| Python itself seems to work quite well (maybe not
| surprising since that's one of the only ways to get a new
| version on Linux).
|
| It's all the packaging infrastructure that's a mess. Pip,
| virtualenv, setuptools, and also the module import system
| is a total disaster. You don't see questions like this
| for Go:
|
| https://stackoverflow.com/questions/14132789/relative-
| import...
| notatallshaw wrote:
| > It's all the packaging infrastructure that's a mess.
| Pip, virtualenv, setuptools, and also the module import
| system is a total disaster. You don't see questions like
| this for Go:
|
| > https://stackoverflow.com/questions/14132789/relative-
| import...
|
| I think you'll find the further you delve into it the
| less the problems are distinct, a lot of the issues with
| the module import system, pip, virtualenv, setuptools,
| etc. is because they are designed with having to support
| a vast range of things, from being depended on to
| interact with system libraries to downloading sdists and
| compiling arbitrary languages, etc.
|
| Though the specific example you linked was largely solved
| with Python 3, there was a lot of confusion during the 2
| to 3 transition because people had to support both
| behaviors, but most people don't have to think about
| Python 2 any more.
| IshKebab wrote:
| > Though the specific example you linked was largely
| solved with Python 3
|
| I can assure you it absolutely was not.
|
| > I think you'll find the further you delve into it the
| less the problems are distinct, a lot of the issues with
| the module import system, pip, virtualenv, setuptools,
| etc. is because they are designed with having to support
| a vast range of things, from being depended on to
| interact with system libraries to downloading sdists and
| compiling arbitrary languages, etc.
|
| Not really. There are plenty of systems that have to
| "support a vast range of things" that aren't this bad.
|
| In my opinion it's because the core Python devs didn't
| particularly care about the issue, never really tried to
| solve it, and as a result we have 10 incompatible half-
| baked third party solutions.
|
| It's similar to the situation with C/C++ - worse in some
| ways, better in others (at least there is a de facto
| package registry in Python). In some ways it's because
| both languages are very old and predate the idea that
| packaging should be easy and reliable. That's fine, but
| please don't pretend that it _is_ easy and reliable now.
| philwelch wrote:
| You could just give up and resort to using Docker.
| fishpen0 wrote:
| Honestly this. I've moved three companies now to docker
| based dev for teams on python. It's better for teams who
| want to customize how they in particular work without
| inadvertently also blocking themselves from quickly
| contributing to another team because that team uses a
| totally different python management scheme. So a team can
| be conda-in-docker and another team can be virtualenv-in-
| docker and another team can be raw-python-in-docker and
| that one group of weirdos can be jython or whatever other
| sad shit they cook up.
|
| The common interface is they all use `docker-compose up`
| and have their editors hooked into the containers.
| Zetobal wrote:
| this is the way.
| BiteCode_dev wrote:
| Using docker is not free, you are effectively trading one
| type of complexity for another.
|
| And of course you'll have to make sure people equally have
| good practices, since incorrectly using sudo pip install in
| docker and not a venv are very common.
|
| So again, one possible solution for a certain context, but
| I wouldn't sell that to most people. Certainly not to
| beginners.
|
| People writing their pythonanywhere website won't pop up a
| container, won't they ?
| hot_gril wrote:
| For a beginner trying to run someone else's Python
| project, it is free. You run one command and it works.
| There's tons more complexity under the hood, but you
| don't have to deal with it. This is why so many Python
| projects do resort to Docker.
|
| And btw, JS doesn't have this problem, they use NPM.
| dragonwriter wrote:
| > For a beginner trying to run someone else's Python
| project, it is free.
|
| Its absolutely, by far, the highest friction alternative
| on Windows for anyone that doesn't use Docker for other
| purposes.
| hot_gril wrote:
| Is it not just install and run? The alternative is
| installing Python, installing pip, etc.
| dragonwriter wrote:
| Installing python with the python.org installer installs
| the entirety of Python and its standard library and
| tools, including pip, venv, etc., and the py launcher
| which supports selecting from multiple python versions.
| Linux distros may split this up into different packages,
| but on Windows (or Mac, afaik, though the py launcher is
| a Windows-only feature) using the official installers is
| one-stop shopping.
|
| And, no, docker isn't just install and run on Windows.
| Before Docker made a heavy push for paid Docker Desktop
| for even personal use, it was close to that.
|
| But now the way to get a free usable docker command line
| is to install WSL and a Linux environment, install docker
| there, and then invoke docker via wsl. (Which, of course,
| you will not find via Docker's own information, which
| will try to sell you a paid subscription.)
| hot_gril wrote:
| I tried it just now on my spare Win10 desktop, and it
| went like this:
|
| 1. Download and install Docker desktop from
| https://docs.docker.com/desktop/install/windows-install/
|
| 2. Restart Windows (guess you don't have to do this with
| just Python)
|
| 3. Run Docker Desktop
|
| 4. Say "no" to signing into a Docker account
|
| 5. Wait for engine to start, which took a few minutes the
| first time, a bit annoying
|
| 6. Pull and run an image (I tried nginx)
|
| It was weird being asked to log in, but the "no" button
| was pretty clear. I didn't feel like I was forced to use
| WSL to avoid paying.
| BiteCode_dev wrote:
| While there are no silver bullet, for beginners, I found that
| "Relieving your Python packaging pain"
| (https://www.bitecode.dev/p/relieving-your-python-
| packaging-p...) is the Pareto solution. That is, the solution
| that has the best ratio of effort, reward, but also the lower
| risk of failure. It's not no risk, but I've been helping
| beginners for 15 years with Python, and have tried everything
| you can think of.
|
| It does imply, on linux, to limit yourself to the choices of
| Python you can install. This constraint is, for most people,
| preferable than the alternative, even if it gets frustrating
| to our geeky soul.
| cqqxo4zV46cp wrote:
| I think that you are painting a bit of an unreasonably bleak
| view of pyenv. I think that one can easily get value out of it
| without being a Python expert. I'm not sure how I can refute
| your "yes, but you'll eventually run into trouble. You just
| haven't clocked enough hours yet". I can't prove a negative.
| But I'll say that I've been writing Python in my day job for a
| decade.
|
| But, to add to the list of problems, these Python versions IIRC
| do not compile with optimisation turned on, so they're by
| default quite a bit slower than they need to be.
| belter wrote:
| I call it the curse of Python.
|
| When the God Of Programming made Python, all other languages
| were jealous of its elegance, simplicity and intuitive beauty.
| When the other programming languages went to complain he
| said..."Wait until you see what package systems I will give
| them...They will never get an environment properly setup..."
| :-)
| throwup238 wrote:
| And then the Devil made CUDA...
|
| _"The CUDA version on your system does not match the CUDA
| version the realm of mortal men was compiled with"_
| b33j0r wrote:
| If you don't care which specific version of python you are
| using, do not use pyenv.
|
| You will know when you care. These days, doing common tasks,
| the constraint solver in poetry will often tell you:
|
| "Hey! I can't find a version of sentencepiece with metadata
| that lets me combine this with python 3.12.2. Sorry! I give
| up!"
|
| Now, if you aren't concerned with using your bare metal for
| CUDA or fancy new MPS or AMD stuff. Just ignore this and use
| containers. I'd use podman compose.
|
| However, I use pyenv on every machine. Because it compiles
| specific versions I actually need, even to create various
| virtual environments. If compiling python automatically sounds
| tough, you probably don't need to anyway.
|
| To describe the problem you'd see. I try to use poetry by
| default, though I think it became popular before it was PEP-
| compliant or useful in devops. It is impossible to control the
| behavior of other package managers, and poetry is/was strict
| about that. Which means you can't force deploy in many cases.
| (Better lately.)
|
| For the problem pyenv helps to solve, I back-up my
| pyproject.toml setuptools backend with pip and
| requirements.txt. These days, requirements-cuda.txt and
| requirements-mps.txt.
|
| The landscape is still a disaster for binary compatibility, but
| it can be done lol. (I've been doing python packaging
| professionally since prom, which was python 2.6 give or take.)
| elevation wrote:
| This kind of error surprises me:
|
| > I can't find a version of sentencepiece with metadata that
| lets me combine this with python 3.12.2
|
| I'm a reasonably advanced python user. I've shipped web apps,
| Desktop GUIs, cli-tools, and even written cpython extension
| modules for custom hardware control. I typically target the
| system `python3` of whatever linux distribution I'm shipping
| to and I use the system 'python3-virtualenv' for venvs.
|
| But I have never encountered a dependency resolution issue or
| been forced to use poetry. What am I doing wrong?
| b33j0r wrote:
| (Edit2 tl:dr: it's that other package maintainers don't
| always keep up with new semver constants in their metadata,
| particularly on anything cutting-edge)
|
| It's always the inclusion of a specific dependency we added
| for a feature, and based on that dev's knowledge and
| experience. It's often me, but not always.
|
| This doesn't happen in ecosystems with a base package
| versioning. This is arguably why anaconda became popular,
| and why we target base docker images of ubuntu.
|
| Doesn't work in complex deployments based on money rather
| than ideals, every time. At least in my career.
|
| Edit: first time I dealt with this, we ended up forking a
| dependency chain instead of using pip. I lost that war and
| they ended up reviving a legacy PHP app instead of funding
| python dev.
| Hackbraten wrote:
| Sometimes, an app or library developer wants to target
| multiple Linux distributions.
|
| For example, I want my own Python apps to work equally well
| on Debian oldstable (which currently provides Python 3.9)
| and Arch Linux (currently on 3.11). That means that I'm
| going to choose the lowest common version (3.9) as the
| language level for my app or library.
|
| And if I program against a 3.9 language level, I absolutely
| *refuse* to use any other interpreter than a 3.9 one at
| development time. (If I used a newer one, then my linter
| and type checker would helpfully give me false alarms all
| the time, unaware of the actual language level.)
|
| Hence, I use pyenv and poetry to get exactly the
| interpreter for the language level I want, and to allow
| pylint and mypy to be perfectly aware of the language level
| I'm using at development time, allowing them to give me
| findings that actually make sense.
| dragonwriter wrote:
| > But I have never encountered a dependency resolution
| issue or been forced to use poetry. What am I doing wrong?
|
| Not using python on a system without the necessary build
| tools while trying to use dependencies with native code
| that don't have binaries built for your combination of
| python version and platform?
|
| This used to be a _huge_ problem with Python on Windows, as
| things that weren 't pure Python would very often not have
| binary packages or have them for a narrow set of python
| versions, often not the same set as other dependencies.
| (Not just a windows problem, but it was definitely big on
| windows.)
|
| Tooling and practices have advanced so that more packages
| are automatically built against a wider set of targets, so
| the problem is a _lot_ smaller than it used to be.
| guappa wrote:
| Python is quite self contained, it's not a chore to build it.
| whalesalad wrote:
| I have not suffered any issues due to pyenv compiling. In our
| prod boxes we compile Python ourselves anyway. It's a really
| trivial build process tbh.
| mrbonner wrote:
| Oh man, you just confirm that I'm not crazy. For every single
| time I use pyenv it just downloads source and build. I got
| tangled into so many issues such as TLS headers in my AL2
| boxes. I thought the whole thing is about getting the official
| binary and install and not compile from source even for a 2
| year old release.
| rewgs wrote:
| What's the problem? The repo clearly states that you need to
| install such and such build prerequisites.
|
| Also, the issues the article you link are things that damn near
| every programmer will eventually need to know. Things like PATH
| are IMO the basics -- if you don't understand this or how to
| `$packageManager install` something, you're gonna have a rough
| time programming in general, regardless of language.
| d0mine wrote:
| Data point: pyenv works just fine for me. It helps me with
| installing/managing many python versions (multiple years).
|
| I guess, I'm in the "expert" category. I'm saying it so that
| people won't be afraid to try it.
| Vaslo wrote:
| Can I ask - why the dislike for virtualenvwrapper? If you are
| saying that it can have occasional problems, I agree with you.
| But it makes the process so much simpler. The occasional
| problems I've had (had an issue once or twice deleting a venv)
| pales in comparison to the advantage I get in remembering a
| couple fast commands. Is there something better or am I
| relegated to "source what/directory/was/it/again/bin/activate?
| godelski wrote:
| Homebrew really bugs me. The devs want you to use the most
| current version of Python. Except the requirement is set at the
| package level and there's no enforcement. So you just keep
| getting python versions installed.
|
| I don't get why you just don't set a minimum version like every
| other manager. Or if you want to do this crazy thing, don't rely
| on package maintainers to enforce it. I just ends up in major
| bloat
| rcarmo wrote:
| This is misleading. The older versions get cleaned out as brew
| does the upgrades (if not, then check your brew settings). My
| take on brew's Python is that it is there to make sure there is
| a uniform runtime for everything with a Python dependency, but
| I am not required to use it.
| mikemcquaid wrote:
| As. Homebrew maintainer: confirmed. You can also adjust your
| settings to pin versions or avoid cleanup, too.
|
| 'brew pyenv-sync' may be useful around this area, too.
| sgarland wrote:
| If I pin a version of Python, isn't that going to wreck any
| tooling that depends on it? Unless you're saying have
| multiple versions of Python installed.
|
| This is practically the only remaining annoyance I have
| with the Python ecosystem (relative imports aside). I use
| some tools, like Glances [0] whose formula relies on a much
| newer version (3.12) than the actual package requires (3.8)
| [1].
|
| So when there's a Python update, all of those update as
| well. I thought I'd fixed this with pipx, but in a way
| that's worse, because the venvs it builds depend on a
| specific version of Python existing, which doesn't work
| well with brew always wanting to upgrade it.
|
| I want a stable, system-level Python that I don't touch,
| don't add packages to, and which only exists as a
| dependency for anything that needs it. If an update would
| break a package I have installed (due to Python library
| deprecation, etc.), it should warn me before updating.
| Otherwise, I don't care, as long as any symlinks are taken
| care of.
|
| Separately, I want a stable, user-level Python that I can
| do whatever I want to. Nothing updates it automatically. I
| can accomplish this by compiling Python and using `make
| altinstall`, but if there's a better way, I'd love to hear
| about it.
|
| [0]: https://github.com/Homebrew/homebrew-
| core/blob/20e744191e74d...
|
| [1]: https://github.com/nicolargo/glances
| mikemcquaid wrote:
| Pinning won't break things but it may block upgrades
| (where they would break something or unpin).
| godelski wrote:
| Isn't it using the Python version in the formula?
| NikkiA wrote:
| I thought the recommended approach these days was to use the venv
| package built in to python3 with `python3 -mvenv ...`
| Hasnep wrote:
| You're right, but venv requires an existing Python install to
| create a venv. Pyenv is for installing multiple different
| versions of Python to create those virtual environments.
| bheadmaster wrote:
| It is, but the problem with `venv` is that it can only use the
| same version of Python that you're using. E.g. if your Python
| installation is 3.8, your `venv` will also use 3.8.
|
| Pyenv allows you to use whatever version of Python you want.
| alwayslikethis wrote:
| Most systems allow you to keep multiple python installations,
| no? I think the best way is to specify the python when you do
| this: python3.11 -m venv env
|
| On most Linux distros you probably don't want to change your
| system python to whatever, because there are a lot of system
| programs that may not be compatible with a version other than
| what is on the system by default.
|
| Then once you activate it you can make sure your python
| _dain_ wrote:
| I used to use this a lot when working on a Windows machine.
| Worked pretty well. But nowadays Nix solves the same problem in a
| fully general way, for all software rather than a single
| language. You can have whatever versions of whatever you want,
| side by side, without interfering with each other.
| pyrophane wrote:
| What is your solution here? Are you using a project-specific
| flake? Do you still install packages with pip?
| _dain_ wrote:
| I was more talking about version management of the Python
| interpreter itself. So far all the packages I need are
| already in nixpkgs so I haven't needed to stray off that path
| yet.
| TomSwirly wrote:
| This is a truly, truly terrible idea. It adds several failure
| modes, some subtle so you can go a long way in a state of error,
| just so beginners can type `python` instead of e.g. `python3.10`.
|
| Many developers, not just me, have a similar setup: we use
| virtual environments everywhere, and if you aren't in one,
| `python` doesn't even resolve to a symbol.
|
| If I want to write a quick script with no dependencies, I
| directly call `python3.xx` on it. Otherwise, I create a
| virtualenv.
|
| Yes, it's a bit harder for beginners, but from a huge amount of
| experience helping people who are starting up in programming,
| people have little issue in following a few more instructions.
| What demolishes beginners is getting into a bad state where
| nothing works and you don't know why.
| fragmede wrote:
| though, with openinterpreter, you can just ask it to fix your
| python and it'll help you out of a weird situation that you as
| an newbie don't understand. Still, node's node_modules
| implementation isn't a bad one, at the expense of disk space,
| you don't run into the same problems as you do python.
| rcarmo wrote:
| This is not a terrible idea. It is a tool that makes it
| possible for people to either retrofit a newer version of
| Python onto an ageing OS or to develop for a specific older
| version of Python on a newer OS. If you don't need the features
| the tool uses, you shouldn't use it--I don't see why you're
| comparing it to virtualenvs, it provides completely different
| runtimes with a different engine and a standard library, and
| the ability to switch between those by just cd'ing to a folder
| with a .python-version file in them.
| happymellon wrote:
| Isn't this a completely different concept?
|
| Virtual environments are for your installed dependencies,
| whilst pyenv is for installing python.
|
| I have a client that uses Python X and another that strictly
| uses Python X+1.
|
| The virtual environments are so that I can have the project
| dependencies installed and the pyenv lets different companies
| have different cadence for their Python upgrades.
|
| I could be completely mistaken and mixing up my Python support
| utils as I've not had a client request Python for a couple of
| years.
| wodenokoto wrote:
| But how do I get a .venv with a new/different version of Python
| in your setup? And how do I add some kind of Python version
| requirement in my project?
| hnfong wrote:
| python3.xx -m venv .
| happymellon wrote:
| And pyenv gives you python 3.xx rather than 3.xy...
| lgxz wrote:
| I think the only problem of pyenv is that you can't upgrade an
| old virtualenv to a newer Python version.
| Hamuko wrote:
| Isn't that more of a limitation with virtualenv? I've been
| using virtualfish as my venv manager in the fish shell and it
| has a `vf upgrade` command for upgrading virtualenvs to newer
| versions, either by updating in-place or rebuilding the entire
| virtualenv. So far has worked nicely with my pyenv Python
| installs.
| rcarmo wrote:
| Yeah. I had a shell snippet someplace that was essentially
| pip freeze | remove virtualenv | make virtualenv | pip
| install. It's not that hard.
| rewgs wrote:
| That has nothing to do with pyenv.
|
| Also, this really isn't a problem -- just blast out of the old
| venv, make a new one with a new version, and you're good.
| danieljanes wrote:
| We use Pyenv successfully for developing the Flower open-source
| project. We use a few simple Bash scripts to manage virtual
| environments with different Python versions via pyenv and the
| pyenv-virtualenv plugin.
|
| The main scripts are `venv-create.sh`, `venv-delete.sh` and
| `bootstrap.sh`. `venv-reset.sh` pulls these three scripts
| together to make reinstalling your venv a single command.
|
| Here's the link if anyone is interested:
| https://github.com/adap/flower/tree/main/dev
| thrownawayjests wrote:
| Obligatory relevant xkcd: https://xkcd.com/1987/
| punnerud wrote:
| Need a new one with pyenv in it? (or to replace it)
| ktosobcy wrote:
| Python (and Ruby) handling of versions and dependencies is truly
| awful... :-(
| bvrmn wrote:
| It's inevitable for non-dead interpreted languages.
| lioeters wrote:
| I've only used Python occasionally, and more than half of that
| time I ran into issues with the installed Python version or
| dependencies. Not only version 2 vs 3, but for example a
| certain project required exactly 3.11, not 3.10 or 3.12. Last
| time, I tried to install Caddy and broke all previously
| installed packages/commands - and after searching for a
| solution and trying different things, the best I could come up
| with was to switch versions (manually) before and after running
| a command to restore the environment.
|
| Now I understand I should have been using Pyenv the whole time.
| But it sure felt unfriendly for a newcomer to the
| language/ecosystem. Granted my learning approach is to jump in
| without much preparation, I wish the language designers or the
| ecosystem provided a better experience when things don't work
| right.
| yxhuvud wrote:
| How is it awful on the Ruby side? In my experience it is a
| solved problem since bundler was invented 15 years ago. It's
| been very free from pain during the last 10 years or so.
| ktosobcy wrote:
| AFAIR it was mostly because Ruby still wanted to use
| not-100%-fixed versions so there was still an issue with
| something updating slightly breaking the build. Just a
| cursory experience when trying to deal with Ruby. Though it
| _felt_ somewhat more stable than Python.
| jscheel wrote:
| In my experience, python has been far, far worse than ruby.
| Now, to be fair, I'm much more comfortable in the ruby
| ecosystem, but still, everything feels much more brittle in
| python package management.
| joshuamcginnis wrote:
| What's wrong with rvm?
| graywh wrote:
| can I interest you in R?
| jpalomaki wrote:
| After some trial and error I have now settled on the following
| "stack": 1)I manage python versions with pyenv 2)For each new
| project I create a new virtualenv with venv "PYENV_VERSION=3.10
| python -m venv .venv" 3)Then I start jump into the venv and
| initiate the project with Poetry ("poetry init -n") and manage
| dependencies with Poetry.
|
| If I'm keeping the projects under Dropbox, then I'll just add the
| .venv folder to the ignore list ("attr -s com.dropbox.ignored -V
| 1 .venv") to reduce the amount of files that need to be synced.
| If I need to get back to old project, I simply recreate the venv
| and install dependecies using Poetry.
|
| A good habit would be to add a .pythonversion file on the project
| folder. Pyenv can pick that up and it is then obvious which
| version was used for the project.
|
| I do development on Windows, but run all the Python stuff on WSL2
| and use VS code for notebooks and everything. One thing I haven't
| looked at is using pipx to manage Poetry. Now I simply have one
| Poetry version installed.
|
| Pyenv has worked fine once I managed to collect all the necessary
| dependencies to Ubuntu. For me it's not just to switch between
| Python 3.x. I like the fact that you can easily have different
| patch versions and match exactly the version I would use in
| production (via containers).
| p33p wrote:
| Odd to me that you don't let poetry create the venv? Why do
| this separately?
|
| Our flow is similar: pyenv (windows and linux), pipx, poetry.
|
| We've also defaulted poetry to utilize the current global
| version of Python and build the venv within the project folder.
| lexicality wrote:
| Personally I've had problems with poetry managing virtualenvs
| in the past so I just don't let it touch them any more. Maybe
| it's better now, but I don't see any reason to risk it, given
| how often Python seems to like to ruin my day. I also don't
| like that by default it wants you to use `poetry run` to run
| things. Sure you can configure it not to, but it still annoys
| me
| Hackbraten wrote:
| One issue I've been encountering is that Poetry isn't aware
| of pyenv or `.python-version`. So what I do to initially
| build my venv is: pyenv exec pip install
| poetry && pyenv exec poetry install
|
| This creates the venv against the correct Python version,
| and I can now do without `pyenv exec` for this repository.
|
| And every time that `.python-version` changes (which I at
| most do a few times per year per project,) I throw away the
| `.venv`, do `pyenv install -s` and start over.
|
| Your `poetry run` point still stands, though.
| jmathai wrote:
| More than Python versions, I use pyenv to keep from polluting the
| system's site-packages.
|
| It also ensures that my requirements.txt is sufficient for the
| code to run.
| nlitened wrote:
| Tools you can use to make sure the Python program you wrote keeps
| working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-
| virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv,
| pyvenv, conda, miniconda, poetry, docker, nix.
|
| Which ones did I miss? Which of them actually ensure your program
| always works the same as when you first wrote it, without
| asterisks?
| riwsky wrote:
| Go
| speedgoose wrote:
| None. The software container image is the best bet but you need
| to keep the image and not only the building scripts.
| nickjj wrote:
| Yeah, containers don't fully solve this problem.
|
| We still need a generated lock file with every top level
| dependency and sub-dependencies locked down to their most
| precise version commit to version control so that when you
| build your image today or in 6 months you end up with the
| same result.
|
| Using pip to freeze your dependencies and writing a tiny
| shell script to generate a lock file at build time is better
| than nothing to solve this problem with nothing more than
| pip. It's what I do in https://github.com/nickjj/docker-
| flask-example and https://github.com/nickjj/docker-django-
| example. It's not perfect but it solves 80% with minimal
| complexity.
| Cu3PO42 wrote:
| This is not true. Nix actually solves the problem. If you
| package with Nix, you'll get the exact same version of Python
| with the exact same version of dependencies, including the
| exact same version of system libraries. Your build will work
| in a year just as it does today. You don't even need to keep
| any binary artifacts.
|
| Of course this doesn't come free: packaging with Nix may
| involve nontrivial effort for some projects.
| speedgoose wrote:
| How does nix solves the problem of Python dependencies not
| specifying their dependencies well or exhaustively? Do you
| have to find them out and fix them manually or can you
| generate a lock file and hope for the best?
| Cu3PO42 wrote:
| It depends. If the dependency is in nixpkgs (Nix' own
| package repository), someone will have done the work of
| figuring out the dependencies properly and you can just
| use that. If it's not, you can either do that yourself
| and pin particular versions, or you can integrate with a
| tool like Poetry: Poetry could generate a lockfile for
| you that you then reference from Nix to get the versions
| of Python packages. You'd still need to specify any
| native dependencies manually, though.
| ris wrote:
| Ultimately the true bellwether of compatibility is the
| package's own test suite, which we try to get working
| (and integrated into the build process) for as many
| packages as possible. For packages which have poor/non-
| existent test suites, often a downstream package's test
| suite will expose compatibility problems (we've found
| bugs in zlib security patches using curl's test suite for
| instance).
|
| nixpkgs maintainers are frequently the first to notify a
| project author of incompatibility with new versions of
| another package.
| ptsneves wrote:
| Pyinstaller, provided you build it in a reasonably old glibc to
| avoid glibc incompatibilities. I know that is a caveat but it
| is a one time build time operation.
| hermitdev wrote:
| Doesn't Pyinstaller also require CPython to have been built
| with '--enable-shared'? I _think_ this is the default for
| most system-installed Pythons that I've seen, but it's not
| the default if you build Python from source or install via
| pyenv.
| bvrmn wrote:
| python -m venv myenv . myenv/bin/activate
| <install reqs> <your program>
|
| Ensures you have the same python environment. The other part is
| OS state.
| _dain_ wrote:
| _> The other part is OS state_
|
| And that's the whole problem. "The same Python environment"
| doesn't mean much because it underspecifies the full runtime
| dependency chain.
| bvrmn wrote:
| It's inherit problem for all languages based on some
| runtime. Java, PHP, Ruby, whatever have the same issue. And
| TBH it's not very major issue to start with.
|
| No other solution besides fixed OS (containers, nix) could
| solve this problem.
| _dain_ wrote:
| _> No other solution besides fixed OS (containers, nix)
| could solve this problem._
|
| You don't have to use NixOS to use Nix. You can install
| it on any Linux distro, or MacOS, and use it as your
| build system.
| jitl wrote:
| The link posted solved this problem without containers
| bvrmn wrote:
| Github issues for pyenv show a slightly opposite picture.
| latexr wrote:
| > Ensures you have the same python environment.
|
| But not the same Python runtime, which does not fulfil the
| request of guaranteeing it works the same later without
| asterisks.
| scaryclam wrote:
| I'm not sure what you might mean by runtime, but a python
| venv does make sure you're using the same runtime. There
| are symlinks in the environments bin directory to the
| specific runtime. If you're plonking a python3.12 binary
| over the top of the python3.11 one, then yeah, you'll be
| using a different one, but that'd be an issue nomatter what
| you're using.
|
| Just don't uninstall your python binaries and you're fine.
|
| If you want to package everything up into am image or a
| zip, you can do that too if you want, but in my 10+ year
| career I've not had much of an issue just using boring
| venvs.
| adolph wrote:
| If you attempt to reproduce a project in a repo that only
| includes source and requirements.txt you will not have
| metadata about the version of Python used.
|
| It seems that there is a way to do this in pip but I
| don't think it is widely used: https://stackoverflow.com/
| questions/19559247/requirements-tx...
| latexr wrote:
| > but a python venv does make sure you're using the same
| runtime.
|
| It does not.
|
| > There are symlinks in the environments bin directory to
| the specific runtime.
|
| Precisely. They symlink to _a path_. Which means that if
| you have a certain version of Python at one location
| (let's say /usr/bin/python3) and later update that
| (let's say by upgrading macOS and the Xcode developer
| tools), the same virtual environment will point to a
| different version of Python.
|
| > Just don't uninstall your python binaries and you're
| fine.
|
| That's not a practical solution. Sometimes you don't have
| a choice, as demonstrated above. By that logic one could
| say "don't change anything about your system and you
| don't even need virtual environments". Which is somewhat
| true, but also profoundly unhelpful.
|
| The point of the question was to reproduce the same thing
| _without asterisks_ and you've introduced a major one and
| called it a day.
| camgunz wrote:
| You can use --copies if you don't want symlinks [0].
|
| [0]: https://docs.python.org/3/library/venv.html
| latexr wrote:
| Not with the macOS Python from the Xcode developer tools:
|
| > Error: This build of python cannot create venvs without
| using symlinks
| camgunz wrote:
| Don't use that one
| latexr wrote:
| -- Doctor, it hurts when I raise may arm.
|
| -- Then don't raise your arm.
|
| Like I said in the original comment you replied to and
| are now ignoring:
|
| > That's not a practical solution. Sometimes you don't
| have a choice, as demonstrated above. By that logic one
| could say "don't change anything about your system and
| you don't even need virtual environments". Which is
| somewhat true, but also profoundly unhelpful.
| scaryclam wrote:
| The binary installs aren't that generic in the machines
| I've used (mac and debian based, so covers a fair bit).
|
| It's python3.11, not python3, and the python3
| "executable" is, itself a symlink to the particular
| binary (in this example python3.11). Upgrading just
| changes the symlink, which wouldn't affect the venv,
| which isn't using python3, it's using python3.11.
|
| I didn't introduce anything, I explained how the links
| are to the versioned binaries, which isn't what you're
| stating is happening.
|
| Edit to add: as someone else also points out, you don't
| even have to use the symlinked versions, you can use
| --copies
| latexr wrote:
| > It's python3.11, not python3, and the python3
| "executable" is, itself a symlink to the particular
| binary (in this example python3.11).
|
| Not for the example I gave. If you're seeing Python 3.11,
| you're definitely not using the /usr/bin/python3 on macOS
| with is made available by the Xcode CLI tools. That's at
| 3.9.6 even on Sonoma.
|
| > as someone else also points out, you don't even have to
| use the symlinked versions, you can use --copies
|
| Which, again, doesn't work for the stated case.
|
| https://news.ycombinator.com/item?id=39818732
| mega_dingus wrote:
| This is not the way you're supposed to be using pyenv
|
| You're supposed to install a specific version of python
| in a specific place, with a specific name. Say,
| /usr/local/python-3.10.6
|
| Use pyenv to use that python. Control that by creating a
| `.python-version` file that says 3.10.6
|
| You now have a project that uses 3.10.6. Unless, of
| course, somebody installs a different version in that
| path - at which point you've got bigger issues
|
| Using pyenv to use `/usr/bin/python3` and hoping for the
| best misses the point
| latexr wrote:
| > This is not the way you're supposed to be using pyenv
|
| Because that's not what the conversation is about. It's
| about virtual environments.
| wodenokoto wrote:
| In this case you don't count the standard library as part of
| your Python environment. Since Python version is not
| specified neither is the standard library.
| _dain_ wrote:
| Only Nix does. The builds are deterministic, transitively all
| the way back to gcc and libc.
| diggan wrote:
| > Which ones did I miss?
|
| Using python and python libraries only from your package
| manager (like APT) for a specific OS version.
|
| Micromamba as well, lightweight version of conda/miniconda.
| anze3db wrote:
| You've missed: pdm, uv, pip-tools, pipx, rye, and probably some
| others.
|
| Only pdm and poetry generate cross-platform lock files by
| default as far as I know, but there are a lot of people trying
| to solve this problem right now.
|
| It's not an easy problem to solve. Python's package management
| predates package managers from most other programming languages
| and Python itself predates Linux. There is a lot of baggage so
| change is very slow.
| rbanffy wrote:
| Shameless plug: don't forget the wonderful pip-chill for
| simplifying gigantic requirements files (and for stripping
| out version numbers to make canaries easier to do).
| kzrdude wrote:
| You missed hatch, uv, rye, pdm.
|
| I think rye hits most of the points pretty well, it ensures
| both python and package versions.
| otabdeveloper4 wrote:
| > Which of them actually ensure your program always works the
| same as when you first wrote it, without asterisks?
|
| Nix does, if you want to actually invest the effort into the
| "without asterisks" part.
| infecto wrote:
| I am always curious how many people, outside of those building
| code for third party clients, actually hit this problem? In the
| 10+ years of using Python I have never had a problem using the
| core tools. The ecosystem is far from perfect but it has never
| cause me a problem.
|
| Edit: Wow y'all are some sour people for voting down this
| question. I truly wonder how often people run into this problem
| compared to just complaining about it.
| renegat0x0 wrote:
| I am not sure about what 'core' means in this context.
|
| In my edperience I had many many problems with OS packages vs
| pip installed ones. There were really strange dependency
| issues.
|
| In some cases I encountered dependency hell.
|
| Even if somebody said to me that the core means also using
| virtualized en I would disagree as places over Internet not
| always explicitly guide you in that route.
| wodenokoto wrote:
| Just getting me and one coworker in sync on our dev machines
| often causes problems. Even more when we try to deploy to a
| cloud service which relies on pip.
| thebytefairy wrote:
| If I were writing applications only for myself it'd be fine,
| but I've definitely run into issues when writing code with
| others - teammates, open source repos, etc.
| crabbone wrote:
| Working in CI in a company that has couple dozens of Python
| packages: I'd say about once a week. There are some weeks
| with no incidents, and there are weeks when everything is
| broken for many days straight.
|
| NB. The latest incident was Friday when I've discovered that
| some CI pipeline ran `setup.py install` that down the lane
| invoked easy_install, which doesn't have a policy of ignoring
| bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground
| when it was trying to install scikit-learn which wanted NumPy
| >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't
| realize that this version should be OK (also, it shouldn't
| have installed non-release versions anyways).
| sumtechguy wrote:
| I just hit it a few days ago. 3rd party script that I did not
| write. The dev has lost interest in it 3 years ago. It is
| written for 2.7.x (and it is a nontrivial amount of code). So
| my choices are some sort of pyenv thing (or one of them), or
| fix up the script myself so it runs on 3.12. I lucked out and
| someone else had already done the second thing. One project I
| worked on was 2.4.x. If they ever have to update that (which
| I hope they have) to something more recent I could see them
| doing it until they port it over. That is not even a lib or
| anything that is just the main prog.
| airstrike wrote:
| download python 2.7 and run it? no pyenv needed
| Hackbraten wrote:
| 2.7 has been EOL for five years. It doesn't get security
| updates nor bug fixes.
| akurtzhs wrote:
| Python Foundation support ended that long ago, but
| vendors maintained it for longer. Red Hat support for it
| through RHEL 7 ends 30 June 2024, however, and my guess
| is they're the last ones.
| wongarsu wrote:
| Just a requirements.txt that you can install with pip in a
| venv works 95% of the time. Unless you need a specific python
| version, which you only figure out halfway through by reading
| the documentation. Or you are using pytorch, because on
| windows and linux the version on pypi lacks GPU support, but
| there's no good way to encode the download url in your
| dependencies. Or when people don't properly maintain the
| requirements.txt, because you are supposed to somehow
| manually keep it in sync.
|
| I run into some kind of packaging or dependency problem for
| basically every non-trivial project. Hence the many
| "solutions", but somehow most end up worse than the problems
| they attempt to solve
| tootie wrote:
| While gross ugly Maven lets you specify the bytecode version
| and every JDK can produce backwards-compatible jars and has
| done for many years
| fbuilesv wrote:
| What package managers are people using in other languages to
| make sure that software "always works the same as when you
| first wrote it, without asterisks"? I'd like to understand how
| they solve the "package no longer exists in a central registry"
| problem.
| crabbone wrote:
| This is not as much about a package manager as it is about
| conventions and necessary dependencies.
|
| Standards ensure that going forward language semantics and
| syntax don't change. Having minimal dependencies ensures
| program longevity. Package manager cannot solve these
| problems, no matter how good it is at its job.
|
| Python doesn't have a standard, it's heavily reliant on
| dependencies which are very plentiful and similarly
| unregulated. A program written in C that uses only
| functionality described in some POSIX standard will endure
| decades unmodified. Even Python helloworld program went stale
| sometime ago, even though it's just one line.
| Shrezzing wrote:
| Dotnet uses Nuget[1]. Packages in the system are immutable, &
| never changing. They can be unlisted, but never deleted
| (except in limited & extreme cases, like malware), which
| means even if a package maintainer stops publishing new
| versions to the repository, existing packages will continue
| to be publicly available for as long as Microsoft continues
| to exist.
|
| [1] https://www.nuget.org/
| mixmastamyk wrote:
| Often a package says it works with OS version X, and not
| X+1. It may be true or false. But what you described does
| not solve either version of that problem.
|
| Or the "says it will work with > X," but doesn't.
| zaptheimpaler wrote:
| I think you're referring to vendoring dependencies? In
| python/pip for example, you can download the source for a
| package and point to the folder directly as a dependency
| instead of the version or a git URL. Most package
| managers/languages support some version of that. I suppose if
| you wanted to vendor all dependencies by default, keep them
| updated etc it would take a little more scripting or extra
| tools.
| codethief wrote:
| asdf can be used as an alternative to pyenv. (In fact, it is
| not only meant for the Python ecosystem, so it can also replace
| nvm and others.)
|
| For me, the combination of asdf and Poetry has worked quite
| well recently: I use asdf to pin the Python & Poetry version
| and then use Poetry to pin everything else.
| leetrout wrote:
| Point of clarification: asdf uses python-build which is from
| pyenv.
|
| https://github.com/asdf-community/asdf-python?tab=readme-
| ov-...
| codethief wrote:
| True. I mentioned asdf mainly because it can be used with
| many languages & tools and is not restricted to the Python
| ecosystem.
| aequitas wrote:
| You're missing Tox[0] & Nox[1]. Tox is actually quite nice when
| dealing with testing code across different versions of Python.
|
| [0] https://tox.wiki [1] https://nox.thea.codes/en/stable/
| spacephysics wrote:
| You didn't miss Poetry, but I have to say up until I started
| using Poetry python in large projects was a pain in the tooling
| department to setup and maintain over longer periods of time.
|
| It's no panacea, but feels more stable and usable (especially
| from onboarding new team members PoV) than other tooling I've
| tried
| john-radio wrote:
| What does poetry deliver beyond what pyenv gets you? I
| haven't used either tool super extensively.
| claytonjy wrote:
| pyenv only manages python versions, while poetry manages
| dependencies and virtual environments. They are
| complimentary, but do not overlap.
| hermitdev wrote:
| To add to this, with poetry, you basically do `poetry env
| use python3.12` to create a python virtualenv on
| python3.12 (you can use whatever python version pyenv
| supports, doesn't even have to be CPython).
|
| The generated virtual env name is a little wonky. I'm not
| sure exactly what the scheme is, but it's basically
| `$(package)-$(some sort of hash?)-$(pyversion)`. I can't
| speak for all tools, but at least VS Code detects the
| poetry env and suggests it (and indicates it as a Poetry
| env) when you go to configure the project interpreter.
| bootsmann wrote:
| You can use the poetry config to enable poetry putting
| the virtualenv into the project folder itself, which
| allows most IDEs to discover it.
| crabbone wrote:
| > Which of them actually ensure your program always works the
| same as when you first wrote it, without asterisks?
|
| There aren't such tools. Python not being a standard and
| heavily reliant on the OS that runs it and on third-party
| components that are also not standard leaves you with no choice
| by to "be at the wheel" all the time. Virtually anything
| written in Python will go stale in a mater of few years. In
| other words, you need to constantly update and test as the
| environment changes just to stand still.
| 2wrist wrote:
| I have been looking at Rye, I like it so far.
|
| The other one is Docker, which some people hate. I don't mind
| it, in some cases I prefer it.
| airstrike wrote:
| You only need pip and virtualenvwrapper
|
| requirements.txt is a text file, not a separate tool...
|
| if you want your program to work exactly as intended in the
| future, there are tools like py2exe and py2app for that.
| mixmastamyk wrote:
| Don't need any wrappers either. venv has been built into
| python for ages.
| airstrike wrote:
| yes but `workon` is too good not to have
| mixmastamyk wrote:
| Looks like what could be a shell alias to cd and
| activate? I always keep a few tabs open to my repos
| folder, so not sure it would help me much. Doesn't seem
| to support fish, though activate does.
| masto wrote:
| I remember when I was a childish Perl fan trading insults with
| the childish Python fans of the time, one of the things "they"
| always chided "us" about was that we had too many different
| ways to accomplish the same thing...
| whalesalad wrote:
| pyenv is all you need. pip is part of python. requirements.txt
| is just a convention used to store the list of stuff to hand to
| pip.
| reddit_clone wrote:
| You missed Pipenv :-)
|
| Edit: Oh I see you didn't. I can't read.
| kissgyorgy wrote:
| I have been using pyenv for years and it's the best tool for
| managing Python versions. You can install any version, switch
| between them, use different versions for different projects, have
| multiple versions installed at the same type without problems. It
| really solved all my Python version managament problems, I never
| looked for any other tool like this since I started using it.
| chaoz_ wrote:
| I've been using pyenv for several years now, but for some reason,
| the basic commands and overall integration don't feel as smooth
| as Node's nvm package. I wonder if that's because Python setup is
| technically harder than Node.
| rsyring wrote:
| I've given up on trying to manage runtime dependencies with
| language specific tooling. Instead, I've moved to mise which
| handles the majority of them (Python, Node, Ruby, Terraform,
| etc.) the same way. It will also activate Python virtual envs:
|
| https://mise.jdx.dev/lang/python.html
|
| It has other nice helpers for development environments (tasks,
| env variables, etc.).
| letmeinhere wrote:
| And for those of you that are already on asdf-vm, it is
| compatible with most of those plugins. I have not yet made the
| jump but I'm tempted.
| rhdunn wrote:
| I'm using pyenv to manage different projects on my machine. I use
| --system-site-packages to share libraries like torch, and some
| others -- especially to ensure those libraries are built with the
| same cuda version, avoid some issues when upgrading drivres, and
| to avoid having to keep multiple copies of several gigabyte
| libraries around.
|
| I'm not using things like anaconda for a similar reason I'm not
| using docker. -- I don't want a separate OS install for each
| service I'm running on my server. It works perfectly fine.
| synergy20 wrote:
| Just use the builtin "python -m venv" and life is good, why the
| others? I tried almost all of them, with various issues, now
| staying with the default venv, it's solid and get the job done.
| jvansc wrote:
| Yeah, my thoughts exactly. I've never found myself in a
| situation where venv did not do exactly what I needed it to do.
| Karunamon wrote:
| The problem there is that by default a virtual environment
| version of Python is just a symbolic link to the actual
| binaries installed elsewhere on the machine. If you do an OS
| upgrade or do anything that increments the system Python
| version, it could cause problems in the environment.
| claytonjy wrote:
| pyenv manages python versions, not virtual environments. venv
| won't help you grab python versions you don't have, and will
| happily install your requirements into a venv with the wrong
| python version.
| kzrdude wrote:
| The premise of controlling python version is that - you're
| sending the project to your friend / colleague and you want
| both of you to use the same python version so that you get the
| same behaviour.
|
| And then the same for your friend the buildserver.
|
| It's all about reproducibility.
| synergy20 wrote:
| which venv does really well, what am I missing.
|
| in the "worst" case you can always do a docker
| dragonwriter wrote:
| venv only works if you already have the required version of
| python installed and have it active as the python for your
| shell (or use a launcher to which you specify the python to
| use in the command line), since venv just creates a virtual
| environment based on the version of python it is run with.
|
| So you actually need something that can read a
| specification of the required python version and use the
| correct one from the available options (maybe even reaching
| out and getting it if it isn't already locally available,
| though I don5 remeber if any of the existing python build
| tools will do this; docker obviously will, but its not
| really a python build tool), across different OS flavors
| (maybe not that last bit, depending on the use case,
| especially for internal development), and with minimal
| overhead for the build tool itself.
| rco8786 wrote:
| Not really a complaint - but I am surprised to see this on the
| front page of HN. This is an enormously popular project that's
| been around for over a decade that basically anyone who writes
| Python regularly is aware of.
|
| Only marginally less surprising than seeing a link directly to
| python.org.
| delduca wrote:
| https://asdf-vm.com/ ASDF is better because it works with many
| more languages, other than only Python, like Rust, Go, Node, etc,
| and other tools, such as AWS/Google/Firebase/Azure CLIs.
| hk1337 wrote:
| I used to be all-in for pyenv and rbenv both but have loved
| using asdf so much more.
| antihero wrote:
| Yeah, ASDF is fantastic, I'm glad stuff like pyenv and rvm
| existed and paved the way for it but ASDF is the way to go
| nowadays.
| llimllib wrote:
| asdf for python is a wrapper around pyenv, it didn't "pave
| the way", it's an important part of asdf.
|
| (Also I like mise better currently:
| https://github.com/jdx/mise)
| secstate wrote:
| Curious what you like about mise! I've been a strong
| advocate for asdf for a few years now. Aside from smooshing
| together tools that are often used together, but not always
| (direnv, make), I can't see what mise adds to the picture.
| And I like still having direnv for what direnv does and
| make for make does, rather than an all-in-one.
| llimllib wrote:
| the asdf CLI has really sharp edges and feels kind of
| aggressively unhelpful.
|
| Even if you just use mise as an asdf alternative, it has
| a nicer CLI and interacts with the same plugin ecosystem
| more smoothly.
|
| (I've found it to be an improvement on direnv, but I
| still use make)
|
| the comparison to asdf page has more:
| https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
| secstate wrote:
| Damn. I'm convinced. The asdf CLI was always a bit of a
| pain ... mostly I don't interact with it that often. But
| mise seems light years better designed. Thanks!
| antihero wrote:
| Well that's excellent!
| jmondi wrote:
| You should check out mise (https://github.com/jdx/mise), it is
| very similar to ASDF (multi langs) but written in Rust instead
| of Bash, and doesnt use the shim technique that ASDF does. My
| shell startup went from a couple hundred ms to a couple dozen
| ms.
| camgunz wrote:
| I'll toss in my (macOS) workflow too:
|
| - Download whatever Python binaries I need from python.org
|
| - virtualenv --python=$PYTHON_VERSION
| ~/.virtualenvs/$PROJECT_NAME
|
| - pyproject.toml
|
| - pip-compile --generate-hashes
|
| There has been _tons_ of churn in the Python project management
| space, and I feel like I 've been blissfully unaware of all of it
| with this workflow. Can't recommend enough.
|
| Why regular Python binaries and not pyenv? I've had enough pyenv
| snafus and meltdowns that I started looking for alternatives, and
| it turned out I could just install whatever Python binary I
| wanted and specify it wherever I wanted. What could be easier?
|
| python -m venv is fine, I've just gotten used to virtualenv.
|
| pyproject.toml is the future and almost everything supports it
| pretty well; it also has fewer implicit weirdnesses than setup.py
| or setup.cfg.
|
| pip-tools are really simple, and they let you have hashed
| dependencies which are super important IMO.
| Capricorn2481 wrote:
| Does it handle OS differences?
| camgunz wrote:
| Yep
| kerkeslager wrote:
| This is the sort of tool that experienced devs all use, but
| nobody teaches or writes it into a tutorial on the larger
| language as a whole, because it feels like a side tool.
|
| For now I'm not seeing a lot of reasons to use PyEnv over the
| `venv` module that ships with Python 3.3+ [1]. I'm sure there are
| some thing PyEnv does that `venv` doesn't, but the fact that
| `venv` ships with Python greatly simplifies things.
|
| Basically my workflow is:
|
| 1. I'll create a Python project with, `mkdir dirname`, `cd
| dirname`, `git init`, `python3 -m venv .env`. This creates a
| hidden folder named `.env/` which contains the virtual
| environment. I'll then usually add `.env/` to my `.gitignore`,
| and also run `pip install --upgrade pip` and `pip install wheel`.
|
| 2. If I'm using an existing project, I'll do `git clone
| `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m
| requirements.txt` (requirements.txt is the idiomatic name for the
| dependencies list in Python projects).
|
| 3. I have the following lines in my `~/.bashrc` (hidden file that
| contains Bash settings): # Gets a directory
| named .env or .venv if it exists in the currend directory or any
| of its parents get_env() { if [ -d "$1/.env" ]
| ; then echo "$1/.env" else if [
| -d "$1/.venv" ] ; then echo "$1/.venv"
| else if [ -d "$1/.." ] ; then
| get_env "$1/.." fi fi fi
| } get_absolute_path() { python3 -c "import
| os; print(os.path.realpath('$1'))" }
| on_prompt() { # Load a virtualenv environment if it
| exists in a file named .env env_folder=$(get_env
| $(pwd)) if [ -d "$env_folder" ] ; then
| if [[ $VIRTUAL_ENV != $(get_absolute_path $env_folder) ]] ; then
| echo "Activating env '$env_folder'" source
| "$env_folder/bin/activate" fi else
| if [ -d "$VIRTUAL_ENV" ] ; then deactivate
| fi fi } # Call on_prompt() every
| time the command prompt executes PROMPT_COMMAND=on_prompt
|
| What this does is when I `cd` or `pushd` into a directory or
| subdirectory of a directory that contains a `.env/` folder, it
| loads the virtual environment, and when I leave said directories,
| it exits the virtual environment.
|
| 4. When I install a dependency, I use `pip freeze` to get the
| dependency string, and I append that line to the file
| `requirements.txt`. For example, if I do `pip install django`, I
| get `Django==5.0.1` in my output from `pip freeze`, so I'll
| append that line to my `requirements.txt`.
|
| 5. To upgrade dependencies, I edit the version number in the
| `requirements.txt` file, and then run `pip install -m
| requirements.txt`. This makes sure that the requirements file
| stays up-to-date with my locally installed dependencies.
|
| 6. To get valid version numbers for a file, you can do `pip
| install <package>==`, which is basically asking pip to install an
| invalid version number. This causes it to list valid version
| numbers. For example, you can do `pip install django==` to view
| all available Django versions.
|
| [1] https://docs.python.org/3/library/venv.html
| crabbone wrote:
| Several years in ops in a large company where I had to hand-hold
| couple hundreds of programmers a lot of whom needed to install
| Python for development on their computers...
|
| One of the troubleshooting steps when "something mysterious"
| happens on developer's computer (eg. package installation fails
| for inexplicable reasons, Python "standard" library components
| missing or present when shouldn't be, incorrect component version
| etc.) was to remove pyenv.
|
| This step was often met with resentment and arguments... but, in
| most cases I was able to win :)
|
| ----
|
| Now, here's a larger point: some programs solve the problem by
| actually solving the problem, while other programs solve the
| problem by adding more code around the problem, which, usually,
| creates new problems while only partially solving the original
| problem.
|
| An example of the former: fsck -- you run it, it looks at your
| filesystem, tries to fix it, if it's broken and then gets out of
| the way entirely. An example of the later: Kubernetes -- you
| start by having a problem of resource allocation / management and
| you end up with a problem of resource allocation / management
| compounded by problems with component version management,
| configuration management etc.
|
| pyenv falls into the second category of programs. The problem
| it's trying to solve is: install and use multiple versions of
| Python. There's really no need for an extra helper program to
| solve this problem. Multiple versions of CPython can be installed
| and used together without the use of any extra tools. I.e. the
| solution to this "problem" is simply to learn how to do that.
|
| Those who advocate for the use of pyenv and the likes usually
| make an argument for "simplicity". I.e. in their mind, not
| needing to know how to install multiple versions of Python is a
| bonus. Something that, potentially, saves them several hours of
| reading the documentation and perhaps saving them a tiny bit of
| typing when setting up Python initially.
|
| I contend that this calculation is off because it doesn't account
| for the problems down the lane. In other words: pyenv helps until
| it doesn't, and then it becomes a liability. Debugging is always
| more difficult if you have more wrappers between you and your
| problem. So, while individual users will not face a lot of
| problems with "wrapper solutions", those who service such users
| will face such problems a lot more frequently. That's why, as an
| ops person, I really dislike "wrapper solutions" -- for me, they
| complicate the task, never help.
| maleldil wrote:
| > Multiple versions of CPython can be installed and used
| together without the use of any extra tools. I.e. the solution
| to this "problem" is simply to learn how to do that.
|
| How do you do that? For example, if you're on Ubuntu LTS but
| want to use the latest Python version, how do you do that? The
| system package manager won't have it. Do you rely on a third-
| party PPA? What about other distros, where you don't have that?
| crabbone wrote:
| I personally? -- Check out CPython Github repository and
| build w/e version I need. CPython's Makefile has altinstall
| command that installs as pythonX.Y rather than just python.
| And if I need to have multiple Pythons of the same version,
| but, say, different in whether they have debug info or
| something like that, I modify the installation prefix.
|
| But, you can also download CPython source tarballs and skip
| the Git part. I haven't used CPython installers in a very
| long time... but, I'd imagine that the MSI would have a way
| to specify at least the location where Python is to be
| installed...
| nxpnsv wrote:
| I use this every day everywhere. Together with pyenv-virtualenv
| and poetry it dramatically improved my python experience...
| claytonjy wrote:
| if you're using poetry, how does pyenv-virtualenv fit in?
| jolux wrote:
| Plug for Mise (https://mise.jdx.dev/) which is like asdf (which
| is like pyenv, but for any language you might want all in one
| tool) but written in Rust and without shims (by default) such
| that it's much faster at running the activated runtime. It
| installs runtimes in parallel and will even download the correct
| plugin if you don't already have it installed. It can also
| replace direnv, which I haven't done because I'm using it as a
| drop-in replacement for asdf with the .tool-versions files my
| team already has in every repo (it supports .python-version as
| well if you've been using pyenv). It works great as a "better
| asdf"/"better pyenv" for my purposes.
| KingOfCoders wrote:
| Migrated some small scripts from Python to Go because of the
| virtual env madness.
| savrajsingh wrote:
| This is my favorite toolset -- pyenv and pyenv-virtualenv! So
| good. Such a great system imo. This is my preferred path through
| the forest of options here. :)
| chasenjohnson wrote:
| This is the way. I moved everyone I work with at over to the
| pyenv, virtualenv, poetry stack. Once you get set up it's pretty
| smooth sailing. It is a shame that so many tools are needed to
| work with python, but I was happy to finally find something that
| works.
| Hackbraten wrote:
| Curiously, post-checkout, how do you go about creating your
| venv and installing your project dependencies?
|
| The shortest ritual I could figure out so far is:
| pyenv install -s && pyenv exec pip install poetry && pyenv exec
| poetry install
|
| and I wonder what might be an easier incantation that still
| works.
| joshuamcginnis wrote:
| I really love miniconda for managing python versions and deps
| myself.
| tryptophan wrote:
| Conda/Miniconda randomly breaks every few months or so when
| updating and I can't fix it. Idk why I stick with it, its been
| like this for years.
| joshuamcginnis wrote:
| Odd. I always do `conda activate base && conda update conda`
| on Mac and never have any issues.
| Decabytes wrote:
| I recently used Pyenv to install an older version of Python on my
| system. The reason for this, is I wanted to create a Docker image
| using 3.8 slim-buster and I wanted to be explicit about the
| Python libraries I needed. By using Pyenv to install the same
| version of python as is used by slim-buster I was able to pip
| freeze my virtual env, and use the same version of libraries for
| the Docker image no problem. Oh the things I have to do in Python
| to avoid dependency hell
| RamiAwar wrote:
| Pls dont.
|
| Just install several pythons with different binary names.
|
| ex. python3.8, python3.9, python3.11
|
| No need to complicate things.
|
| If using Poetry, you can just do `poetry env use python3.8`.
| scriptme wrote:
| Here's all the project boilerplate you need with pyenv when
| working on a new project.
|
| > $ pyenv install 3.8
|
| > $ pyenv local 3.8
|
| > $ python3 -m venv .venv
|
| > $ source .venv/bin/activate
|
| and you're done. Not really sure what you might consider
| difficult about pyenv, but it's just a tool to instigate your
| python venvs which is a built tool for most modern versions of
| python.
| Hackbraten wrote:
| What do you do when (not if) your system package manager
| eventually drops that Python version from their core
| repositories?
|
| The good thing about pyenv is that it makes you rely less on
| package maintainers. You can come back to your Python project
| 10 years later and have a real chance that it still works, even
| if you have changed workstations in between.
| liveoneggs wrote:
| Seeing pyenv inside of docker containers is always a treat
| MrYellowP wrote:
| In windows I just use subst to switch python versions. Python
| always sits on drive P:. Never understood what all the fuzz about
| this was about. Sure, I have the same package on my disk multiple
| times, but if I cared about that there'd be easy ways to get
| around that problem just using OS functionality.
|
| From my perspective, it doesn't come easier than running a batch
| file to switch between python versions.
| cyrialize wrote:
| I've been using Rye[0] lately, which has been pretty good. It's
| really just a wrapper around a bunch of underlying tools - it's
| nice to not have to worry about those and let Rye do it's thing.
|
| All that being said, the creator of Rye is 100% cognizant of that
| XKCD comic, this [1] is a nice read.
|
| I'm not super well versed in Python tooling at all. I've had to
| work a lot in Python in the past 6+ months, and I become super
| confused when I tried making a Python project in my spare time.
|
| I settled on Rye because it just seemed to be the easiest to use.
|
| [0]: https://rye-up.com/ [1]: https://github.com/astral-
| sh/rye/discussions/6
| cgopalan wrote:
| I used pyenv in my previous company. At first it was fine, but
| after a while with multiple versions of Python installed, things
| stopped working. The right virtual env was not activated, etc.
| (It could have also been because our software updates like OS and
| security upgrades were pushed by desktop support). I removed all
| of it, and just resorted to installing multiple versions of
| Python the old way (downloading from python.org) and then use
| `python<version> -m venv /my/virtual/env` to manage my virtual
| environments. Things are more stable, and I don't feel like its
| magic. I am not going back to pyenv.
| andreif wrote:
| Works fine for me on macOS. I am using venv the same way you do
| except some global ones for certain tools.
| andreif wrote:
| Also, when working on packages supporting multiple versions
| it was very helpful to use tox with pyenv.
| cgopalan wrote:
| I thought one of the advantages of pyenv was that you
| switched to also creating virtual envs using pyenv and
| managed virtual environments with it too. Maybe not.
| Joel_Mckay wrote:
| I wonder if it ever occurs to people the whole concept of
| versioned environments for compatibility is fundamentally a
| terrible design kludge.
|
| Someone needs to standardize a single Python environment at least
| annually. As these tools are just turning Python into a perpetual
| Alpha build.
|
| Best of luck, =)
| Karellen wrote:
| I long for the day when python is mature enough for an
| implementation to arise where one binary can run as all previous
| versions of the language and stdlib, so you can do
| allpy --std=py3.4
|
| and have it "just work", in the same sort of way that you can
| currently do gcc -std=c90
|
| (And also therefore `#!/usr/bin/allpy --std=py3.4`)
|
| I mean, imagine if C compilers required that you have a separate
| compiler binary and copy of the standard headers (or even worse,
| all of stdlib!) for each version of the C standard?!? Instead of
| just putting new language features on version flags, and make
| stdlib functions available/marked as deprecated based on version
| checks?
| martinky24 wrote:
| I wonder if "maturity" is the problem, or just different design
| goals. You speak like this is an inherent feature that comes
| with some magical level of "maturity", but if the feature isn't
| a legitimate design goal of the product, it isn't "immature"
| not to have it.
| zaptheimpaler wrote:
| pip+venv mostly work fine for me, but there are some libraries
| like say numpy/scipy that are heavy so I want to share them
| across projects instead of installing multiple copies. I use
| pyenv to create a python version separate from the system version
| and install global dependencies into that. It's been useful for
| that and is just a nice way to manage multiple versions of
| python. So pyenv is a great addition to the traditional pip+venv
| setup.
|
| unrelated rant -
|
| Why is the "requirements.txt" file a stupid flat listing of all
| transitive dependencies with pinned versions? It makes it harder
| to change library versions even if there are no true conflicts.
| I've resorted to making an "actual_requirements.txt" file
| manually listing only direct dependencies and whatever version
| constraints make sense. I wish pip would fix this.
| akurtzhs wrote:
| > I've resorted to making an "actual_requirements.txt" file
| manually listing only direct dependencies and whatever version
| constraints make sense.
|
| Pip-tools has an elegant solution for this. You write an
| requirements.in where you list packages and optionally
| versions, then pip-compile turns that into a requirements.txt
| with every package including dependencies specified by
| versions. You can use it to update specific packages as well.
| airstrike wrote:
| I've been using python on and off since Django 0.96, which was
| released in 2007.
|
| I don't recall ever needing anything other than virtualenvwrapper
| and pip--and even some of the annoyances these tools had early on
| have been solved by now...
|
| https://virtualenvwrapper.readthedocs.io/en/latest/
|
| If you really need different versions of python, you can just
| `mkvirtualenv -p python3 venvname`
|
| I feel like every other tool out there has to explain what
| problem they solve that virtualenvwrapper doesn't
|
| Don't install anything globally, creates lots of envs, and feel
| free to have different versions of python installed side-by-side
| with some "main" version preferably symlinked as `python` and
| `python3`
|
| The end
| packetlost wrote:
| You don't even need virtualenv much less virtualenv wrapper. On
| Debian/Ubuntu distros (and most others, with small
| modifications) you just need python3-venv. Then create your
| venv with `python3 -m venv myenv` or `python3.11 -m venv
| my311venv` or whatever version of Python you need. Less is
| more!
| airstrike wrote:
| Agreed, but `workon` is too convenient not to have
| kstrauser wrote:
| I reimplemented the parts of `workon` I actually used with
| a small shell function.
| cgopalan wrote:
| You should look at the command `python -m venv`. Its built in,
| and is a breeze to create virtual environments. I guess it does
| not provide shortcuts for activation etc, but I am ok with
| that.
| stairlane wrote:
| Glad I'm not the only one. As soon as I started using Pyenv for
| local development I bloated my machine with more versions of
| python than I could manage. Obviously user error, but it never
| solved a problem I couldn't get around with virtualenv.
| divbzero wrote:
| For managing versions of Python packages, I take a similar
| route and keep it simple: python -m venv .venv
| source .venv/bin/activate pip install -r requirements.txt
|
| However, pyenv addresses a different problem: managing versions
| of Python itself.
|
| For example, if you need the latest version of Python and it is
| not available through your OS: pyenv install
| 3.12 pyenv global 3.12
| airstrike wrote:
| on Mac OS I just do brew install python@3.8
| brew install python@3.12
|
| and pick whichever one I want symlinked as `python` and
| `python3` (I think the last one you install is the one that
| gets symlinked, but you could always change those links
| yourself manually)
|
| on Linux it should be just as easy to have multiple versions
| side-by-side
|
| it's been a while since I've done something similar on
| Windows but it's also possible (although I think I would
| prefer to use WSL if I were on Windows these days)
| divbzero wrote:
| Some Linux distros don't have the latest Python versions in
| their package repos, but there are definitely other ways to
| install versions side-by-side.
| rovr138 wrote:
| > feel free to have different versions of python installed
| side-by-side with some "main" version preferably symlinked as
| `python` and `python3`
|
| This is what pyenv does.
| airstrike wrote:
| by compiling from source? god no.
|
| also it's terribly named because it conflates the idea of
| python environments with python versions
|
| just install pre-compiled python from official sources side-
| by-side. no pyenv needed
| j7ake wrote:
| Python package management is so broken. Think of the gymnastics
| that every python package has to go through in order for users to
| install it properly. Absolutely insane that this is still
| happening in such a widely used programming language.
| tapirl wrote:
| A similar tool for Go: https://go101.org/apps-and-libs/gotv.html
| ijijijjij wrote:
| to switch, I use on the command like "python", or "python3"...
| seems a lot easier to me and you dont even need an extra package.
| someotherperson wrote:
| Was a pyenv user for the longest time, but I switched to Docker
| containers instead. I put together bash functions that take in a
| positional argument for the image and port (or randomizes it if
| not specified), and it automatically mounts the current working
| directory to the volume. Now I can just start a container without
| having to think about any of this stuff. And a config file per
| project if I don't want to pass in any variables at all.
| reddit_clone wrote:
| Casual Python user. I found all these things very confusing.
|
| Then I found Pipenv. Sticking with it. (Emacs has support for
| this).
| neon5077 wrote:
| I really find it quite astounding just how much absolute nonsense
| python programmers are willing to put up with.
|
| Needing more than one, MAYBE two versions of your language
| installed is insanity. If I build a Python widget and send it to
| my non-programmer coworker, there is exactly zero chance it will
| work until I walk over and manually set up the correct language.
| Instead I use a language that _natively_ compiles to an exe.
|
| Python is pretty neat, but the concept of a program that just
| works anywhere is so utterly alien that I genuinely cannot find
| any practical use for it.
|
| I can't just install a pip package to use globally, I have to set
| up some goddamn virtual environment because everything is so
| brittle that nothing works without the exact correct language
| version.
|
| It's like NPM all over again. Dependency hell is so bad that your
| package manager is now a dependency so we need a package manager
| manager to manage installing the correct package manager version
| to then install the correct packages.
|
| Every single time I've come up to some Python thing or other, I
| spend about fifteen minutes fucking with it before giving up and
| using a tool built in a sane language.
|
| The fact that I can't just send any random person a program and
| expect it to run _at all_ is just lunacy. And no, compiled Python
| does not count because that 's _even more_ brittle tooling on top
| of all the other bullshit I have to deal with. If it doesn 't
| work out of the box, it's a bad tool and I have _far_ better
| things to do with my very valuable time.
| afiodorov wrote:
| I've been programming in Python for years and I agree: Python
| isn't really for consumers. But hey it is used for LLMs that
| can generate React apps, so there's that.
|
| I am only half-kidding. But it appears the industry as a whole
| is concentrating effort on fewer languages and Python is one of
| them. If I want to distribute my app - I write it in React, but
| I still often use python in the process: prototyping, asking
| GPT to rewrite X from python to typescript, making mock servers
| for tests / preparing data for unit tests.
| globular-toast wrote:
| You're just using the wrong tool for the job.
___________________________________________________________________
(page generated 2024-03-25 23:01 UTC)