[HN Gopher] Why not tell people to "simply" use pyenv, poetry or...
___________________________________________________________________
Why not tell people to "simply" use pyenv, poetry or anaconda
Author : sametmax
Score : 42 points
Date : 2023-03-30 21:25 UTC (1 hours ago)
(HTM) web link (bitecode.substack.com)
(TXT) w3m dump (bitecode.substack.com)
| jooz wrote:
| Ive heard that 'venv' are very problematic, but honestly, Ive
| never had a problem. And I used them daily. I understand that it
| can not be enough on some cases... that don't concern me.
|
| I would recommend to 'python -m venv' and thats all.
| bobx11 wrote:
| This is also my setup. It has the added benefit of being
| already included on every python install already so there is
| nothing extra to use.
| hobs wrote:
| Same no problem, different solution - I use pycharm extensively
| and it manages my venvs 99% of the time with no issues at all,
| the only time it took me a minute of head scratching was
| realizing I needed to install a new system python to make a
| venv with it, but that would be clear if you were doing it via
| the shell approach you are using as well.
| Groxx wrote:
| I've heard a lot ( _a lot_ ) of complaints that are wildly
| misattributed to venv (like "version X of Y broke my project"),
| but I've essentially never heard of issues with venv itself.
|
| Aside from needing to know to use it. Which is certainly a
| problem. But python blessing a single venv-system might be
| worse in the long run...?
| abrichr wrote:
| Agreed, never had a problem with this approach.
|
| The only limitation I've encountered is when moving the
| environment or renaming one of the parent directories. In which
| case it's easy to create a new one: #
| optional: freeze the environment if you don't already have a
| requirements.txt source .venv/bin/activate pip
| freeze > requirements.txt deactivate #
| remove the old environment rm -rf .venv #
| create a new one python3.10 -m venv .venv
| # activate it source .venv/bin/activate #
| install the requirements pip install -r
| requirements.txt
| nicoburns wrote:
| I've found that venv's work. They're just very inconvenient
| (having to constantly manage which venv you're in when you
| change directory). Consider that with NPM, only two of those
| commands are ever needed: # install the
| requirements npm install # remove it
| rm -rf node_modules
|
| The rest is unnecessary because NPM uses the equivalent of a
| venv automatically based on your current working directory
| and the location of your package.json file.
| nagonago wrote:
| I'm curious what these problems are. The only problem I've had
| with venv is that sometimes I forget to activate it.
|
| As the article says, I think these days it's pretty safe to
| just use Python's built-in venv and stay away from everything
| else.
| tetha wrote:
| This is one of the jenga towers that seem to hold up most
| reliably for me as well. (i) Use python installed via the
| package manager. (ii) If python is updated, wipe all venvs,
| because there can be strange interactions between venvs and the
| installed python and rebuilding venvs is cheap, and (iii) Work
| in venvs as much as possible.
| jakewins wrote:
| Are there any efforts akin to deno for python? A "burn all the
| packaging down and start over" path?
|
| It's so thoroughly broken, every day a dev on some team gets
| their poetry env entangled with some system installed python, or
| numpy suddenly decides all the CI builds will now compile it from
| scratch on every build, or.. Today it was segfaults on poetry
| version X on the M1 Mac's, that went away in version Y but of
| course version Y broke pandas for the windows devs..
| blame-troi wrote:
| Every time I get the urge to pick up Python, I find the
| packaging situation so off putting that I quickly find some
| other rabbit hole to dive into. I find emacs and vim add on
| configuration easy and transparent by comparison.
| hodgesrm wrote:
| Exactly the same thing happened to me recently. I found
| refuge in Golang. Not my favorite language but it's tough to
| beat for utilities that you want to distribute quickly.
| nonethewiser wrote:
| For all the shit node gets for being too dependency heavy,
| npm, yarn, pnpm all work exactly how I'd expect.
| nonethewiser wrote:
| Install dependency.
|
| Save in requirements.txt. Includes sub dependencies that are
| system specific.
|
| Install from requirements.txt on different machine and get
| errors.
|
| Uninstall dependency and save to requirements.txt.
|
| Look at requirements.txt and see that sub decencies are still
| there.
|
| What is the right way to avoid these issues on Python?
| cure wrote:
| > A "burn all the packaging down and start over" path?
|
| In Python land one of those seems to come along every couple of
| years. They start over and implement a solution for some subset
| of the problem space. Then they realize the Python packaging
| mess is much, much bigger than anticipated and progress stalls.
| At that point there are 15 partial "standard" packaging
| solutions for Python, where there were 14 before. None of them
| are feature complete, and they don't really coexist well or at
| all.
|
| Cue the obligatory https://xkcd.com/927/ and
| https://xkcd.com/1987/
| rubenfiszel wrote:
| For the need of windmill.dev (OSS retool + FaaS), we support
| deno and python. To provide that same experience that you're
| speaking of which is "python run myscript.py" without having to
| pre-install the dependencies, we wrote our own python parser in
| Rust which infer the PyPi packages from the imports (and then
| our workers download them if they are not cached). Here is the
| code from the parser: https://github.com/windmill-
| labs/windmill/blob/main/backend/... which we could carve out as
| a separate module.
| LanternLight83 wrote:
| Guix (and maybe nix?) insists on being the "one package manager
| to rule them all", so Python packages and Rust crates are all
| re-packaged (sometimes with automated importers). Well I do
| still occasionally need to package something myself (or cheat
| pip/requirements.txt it), it definitely covers this "burn it
| down" philosophy and keeps environments isolated and
| reproducible.
| bayesian_horse wrote:
| I have never used poetry... Mostly I just use conda or plain
| python envs. Never had those problems like you mention. When
| using anaconda in the stable channel you'll get a straight-
| forward "conservative" distribution of all the data
| science/numerical packages (and more). That's why people often
| say "just use anaconda", it really is quite simple as long as
| you don't mess up your stable environments with exotic packages
| you just want to try out. And it works well on Windows. No idea
| about Mac and don't care.
|
| Python doesn't need "something like deno". All programming
| languages need package management.
| nonethewiser wrote:
| How many people work on these projects managed by conda?
|
| > And it works well on Windows. No idea about Mac and don't
| care.
|
| Package management is a cross platform problem. You may not
| need it but that doesn't make the current solution good.
| tetha wrote:
| Dude, you are dragging up bad thoughts of how finicky it can be
| to get ansible to work consistently across a number of systems.
|
| This is where I kinda regret switching from and miss chef. Pick
| up an RPM/APT package from vendor, shove it into your package
| manager, the environment works. It developed a sufficient
| amount of weird behaviors later on, but that's besides the
| point.
|
| Instead we have a really precise documentation of 4-5 steps to
| get everything installed in a way that should overall work, and
| each step is annotated with 4-5 ways that look deceptively good
| but end up being even more quirky. And when you do all of that,
| and review it twice, it still doesn't even work consistently
| across 6 workstations. 3 work the same, 2 have interpreter
| discovery anomalies, 1 has import conflicts with seemingly
| unrelated other installed stuff.
|
| Whenever I have a working python VM / container build or
| installation, I kinda feel a need to shut up about my jenga
| tower because otherwise the universe will find a way
| specifically to ruin my day.
|
| I hate how I have to feel like this about a language I really
| like.
| bb88 wrote:
| I don't think you can, nor would you want to at this point. It
| would just be another standard (ala XKCD-927). Virtualenvs,
| pip, and wheels have been better than eggs and whatever else
| came before wheels.
|
| Once you set the version of python in the virtualenv and
| activate it, the right python will always be there. The
| virtualenv even has an instance of python in it's bin. When
| that python is run, whatever's installed in the virtualenv
| comes with it.
|
| Poetry, however, does suck even though it's using virtualenv
| underneath. It's doing frankly too much and seems to have a lot
| of regressions. In the end it should only be looking at the
| contents of the virtualenv and using pip to install the correct
| packages into it.
| IshKebab wrote:
| I doubt that would work. It isn't just the packaging that's
| broken. The entire import system is broken. I doubt there are
| more than a handful of developers in the entire world that
| really understand how Python's `import` works.
|
| So to fix it you would need to change that, but then you would
| break a load of Python libraries which are most of the reason
| people use Python in the first place.
|
| Deno doesn't really have that issue so badly because JS imports
| are ... not completely sane but they're quite a lot saner.
|
| Anyway I think if you were going to break compatibility with
| Python by fixing the packaging and import system, it's not a
| much bigger step to fix the rest of Python too and then you
| have Nim or Lobster.
| qbasic_forever wrote:
| Develop in a container. It will remove all other issues from
| your machine/environment and let you declaratively specify the
| exact environment you need (i.e. packages to install, version
| of python, etc). Forget all about venv and all the other tools
| in the python ecosystem--in your container there is _one_
| python and you control all of its dependencies.
| monkellipse wrote:
| This was my solution. It feels like overkill at times but I
| _never_ have to worry about packaging messes anymore. That
| being said I can understand the desire to solve it on the
| metal, it's a mess.
| nawgz wrote:
| Pragmatic advice, certainly, but maybe it's missing the point
| - you shouldn't need a clean OS every time you want to use a
| new language version. That's a ridiculously large failure of
| packaging.
| qbasic_forever wrote:
| It's core to the design of python right now. Remember when
| python was designed in the 90s people had big multi-user
| Unix systems that were bespoke pets. It was assumed that
| python would be a system-level tool and all packages, etc.
| would be installed at the system level. Your Python install
| would be a living thing that was upgraded and administrated
| like any other software on the machine.
|
| Fast forward 30 years and we really don't use most systems
| like that anymore, especially in production environments.
| We run things in VMs or containers and build them up from
| scratch with ease--it's all just cattle. Python hasn't
| really adapted to that new reality.
| bb88 wrote:
| You can do that today. You're not forced to run in a
| virtualenv if you don't want. Install the python as a
| system package, and then sudo pip install your way to
| happiness. That's how our production containers are
| built.
| dvdkon wrote:
| The fact that many popular Python packages rely on
| system-level native tools also doesn't help. "Solving
| Python packaging" isn't just about distributing portable
| Python source code, that's easy. It's also about
| packaging and distributing unruly non-Python software for
| a myriad different platforms, all without user
| intervention. Sure, the solution's not great, but it's a
| really hard problem.
| nonethewiser wrote:
| Why aren't virtual environments enough?
| scruple wrote:
| I want to hear the answer to this, too. I'm back around
| to Python professionally (haven't touched it since 2016)
| and I'm working with the basic tooling: pip,
| requirements.txt, and virtualenv. I'd like to know what
| sort of issues I'm going to run into and when I can
| expect them.
| morkalork wrote:
| I work with a bunch of data scientists and they manage to
| go about 1.5 years between "I've totally fucked my Python
| environment and have to burn it down completely and
| reinstall it from scratch" events. So yes, I would agree
| with your statement about Python's packaging system being a
| ridiculous failure.
| doublepg23 wrote:
| I think I mostly wrapped my head around pyenv and used Anaconda
| the other day. It was quite the pain, to setup and then it
| seemingly mangled my fish and bash configs causing a noticeable
| delay every start up. Not something I was hoping for just for
| hacking around on some AI project.
|
| Disclaimer: I was using Fedora which has Python 3.11, using Fish
| which is clearly non-standard and I'm a sysadmin not a Python
| dev.
| nightfly wrote:
| You don't have to activate venvs, you can just refer to the
| paths to pip and python inside of them as needed
| ElectricalUnion wrote:
| Second this, just run python/whatever-binary-you-need inside
| a venv:
|
| `${my-venv-path}/bin/python`
|
| `${my-venv-path}/bin/${whatever-binary-you-need}`
|
| `%my-venv-path%\Scripts\%whatever-binary-you-need%` (because
| Windows...)
| tipsytoad wrote:
| Aside: I usually use direnv to activate the venv (or poetry) when
| entering a dictionary
|
| https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...
| davb wrote:
| Python runtime deployment is a major pain point for us (CS
| department at a university).
|
| On the most tightly managed lab machines, which are all in
| lockstep on a fixed configuration (latest Ubuntu LTS with a
| updated image pushed annually), we can provide a consistent
| Python setup (e.g. Python 3.10 and a fixed set of C-based modules
| like psycopg2). However, our staff and PhD desktops and laptops
| are more diverse - with the OS often only being upgraded when the
| distro is going out of support, they could be running n, n-1 or
| n-2. That, most likely, means three different Python versions.
|
| We could use pyenv to let people install their own preferred
| version. Installing with pyenv requires building from source
| (slow on some of our oldest machines). This also means installing
| the Python build deps, which is fine for our departmental
| machines but not possible on the HPC cluster (operated by a
| different business unit) or the Physics shared servers. It's also
| less than ideal for our servers where students deploy their
| projects (where we want to minimise the packages installed, like
| the build-essentials meta package).
|
| It's also a massive stumbling block for less experienced students
| with their own laptops which could be running any distro, of any
| age. Many CS101 or engineering/business/humanities students
| taking a programming class, who have never programmed before,
| would really struggle.
|
| So, classes might tend towards teaching lowest common denominator
| Python (i.e. the oldest conceivable version a student might have
| installed on their machine).
|
| Sure, we have in-person and remote lab machines students can use
| - but it's not always convenient (especially for the data science
| / ML students running Jupyter notebooks with their own GPU).
|
| There are workarounds, but they all have serious downsides.
|
| Compared with Node.js and Go, where users can just download the
| appropriate package and unzip/untar the runtime or compiler
| version of their choice, deploying the Python runtime has
| enormous friction (especially for less experienced users). This
| has the bonus of simplifying deployments elsewhere in our
| infrastructure (CI/CD, containers, etc).
| rektide wrote:
| Having encountered poetry recently for the first time, it was
| "simply" hell. I just wanted to use a single file python project,
| https://github.com/rumpelsepp/oscclip
|
| I spent about three hours trying to figure out how to setup
| python keyrings to work, to let me just get started using poetry.
| On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I
| spent a while adding random pam rules suggested by archwiki in to
| inject more gnome-daemon stuff in my envs. Random gnome-keyring-
| unlock scripts, which quickly start talking about Clevis and tpm
| and fido 2-factor. Wading through hundreds of responses about
| Seahorse, a gui tool unsuitable for ssh. Many long miserable sad
| stories.
|
| In the end I stumbled upon someone who suggested just nulling out
| & turning off keyring with some config to make it have a null
| provider. After this the poetry project just worked.
|
| The tiny handful of deps this project has were already installed
| on my system, but poetry was also a task runner, instrumental for
| the usage of this single-file script.
|
| There's been so many years of churn in the python world of tools.
| A fractal nesting doll of virtual-env, mkvirtualenv, & various
| offshoots. I hope some day there is a mature reasonable option
| folks generally find agreeable. Poetry eventually worked for me,
| but what a miserable gauntlet I had to walk, and the cries of so
| many who'd walked the path & utterly failed echoed out at me at
| every step.
| whalesalad wrote:
| Yikes. One of my favorite features from pip is how easily you
| can install from a git repo, or even the absolute url to the
| master.zip
| melody_calling wrote:
| If you're just using python as a local scripting language, and
| not pushing production code, the other option is to simply not
| bother with any of this.
|
| When there's a new python version I'm interested in, I install it
| via Homebrew and update my zshrc to clobber everything else via
| $PATH. All my scripts and tools are broken? Just reinstall the
| packages globally. Whatever.
|
| Since the big 3.x transition, it's pretty rare for forwards-
| compatibility to break (IME), and if something does, I can just
| try running prior python3x binaries until I find the last version
| that worked.
|
| It's hideous, but honestly the least stressful way I've found to
| date.
| bayesian_horse wrote:
| In my personal experience, I'll take Python's packaging hell over
| nuget or npm any day.
|
| And it's often less about the package manager and more about the
| ecosystem: Can you find what you need? Is what you need stable
| enough? Does it break every few months with new versions of the
| runtime, either because of actual incompatibility (npm/node) or
| versioning shenanigans (nuget)?
| sacrosancty wrote:
| [dead]
| inferiorhuman wrote:
| Let's back up a moment. npm at least works on *BSD. Anaconda is
| just as toxic as Electron in that regard.
| bayesian_horse wrote:
| Given the target audience of Anaconda, I can see how BSD
| compatibility doesn't matter to them. I guess it would take
| some major lifting because much of what they do is wrangling
| the compiling environment. Supporting Windows is hard enough.
|
| That's not "being toxic", you just can't be everything to all
| people.
| inferiorhuman wrote:
| Nah it's toxic. It's not just that Anaconda won't provide
| binaries, it's that you can't even build a project that
| uses Anaconda on not-Linux/Windows. In terms of why you
| shouldn't tell someone to simply use Anaconda, that's
| pretty high up there.
|
| As tedious as javascript package management can be, python
| has consistently given me more trouble.
| trey-jones wrote:
| > You should really use docker
|
| >> I think you missed the point.
|
| Maybe I did, but I've been using Docker as version management for
| pretty much every technology I employ for five or six years.
| Prior to that I sparsely used things like rbenv and virtualenv
| and I actually thought it was super dangerous and unreliable.
| Maybe it's gotten better in recent years, and certainly people
| who write python and ruby every day are going to know more about
| this than I do.
|
| I don't install _anything_ on my computer if I can just use
| Docker for it. OK, I do have go:latest, but I use docker images
| for various projects that might be on any version of go from 1.8
| to 1.20. Your website still runs on PHP5.3? I can help you (I won
| 't, but I could totally run it locally!).
|
| Reasons I like docker better:
|
| 1. Any scripts or configs can explicitly refer to the version
| number. No guessing or assuming.
|
| 2. Our whole team uses the same version.
|
| 3. Only one dependency: docker.
|
| Granted I'm more of a sysadmin than a developer and I'm sure that
| biases apply.
| whstl wrote:
| Suggesting Docker in the context of Python dependencies is
| still missing the point, because you still need proper
| dependency management in case you have to rebuild your image
| for some reason.
|
| If you have a Docker image that builds with "pip install
| whatever" and this library gets updated with a breaking change,
| you won't be able to rebuild the image without changing the
| dependency or the code itself, for example.
| throwawaaarrgh wrote:
| Newbs add abstractions to avoid complexity. Veterans avoid
| complexity by removing abstractions.
| regularjack wrote:
| My personal experience is the complete opposite of this.
| [deleted]
| dsr_ wrote:
| In general, don't tell people to "simply" or "just" use anything
| unless you're willing to provide the precise config that they
| need or otherwise hand-hold them through the starting phase.
|
| Nothing in computing is "simply".
| inconceivable wrote:
| "trivial" also fits into this category. i've been keyboard
| jockeying for 20 years and every time i see that word i groan a
| little, because it's both a shitty flex and probably untrue.
| tetha wrote:
| My documentation skills have improved when I started to be
| critical about "obvious", or "trivial". And now I routinely
| end up writing "obviously", stop, and end up with 3 pages of
| clarification.
| livelielife wrote:
| except that side of computing for which users are really
| "input" to be turned into "output"
|
| which is the industry for which users are really the product.
| user's data in, profits out.
| 2h wrote:
| 100% agree. I use a programming language to get stuff done. and
| if the day ever comes that someone wants me to show them how I do
| what I do, I dont want to start that conversation with a _sigh_
| "well...", I want to start it with a "OK cool..." and all these
| Python "tools on top of tools" make me sad.
|
| Personally I like Go. If someone wants to build my stuff, then I
| just say go here http://go.dev/dl and download Go, then set
| location to where the code is, and enter "go build". thats it.
| All languages should be that easy.
| paulddraper wrote:
| Wait did you just call Go versioning and dependency management
| "easy"??
| 2h wrote:
| I dont have trouble with it. Link to my code is in my bio,
| wheres yours? Unless you actually write Go code, I dont think
| you have a leg to stand on here.
| jerf wrote:
| Are you running on 5+ year old data now? I'm seriously
| asking, because I'm actually a bit mystified as to the people
| who think Go's dependency management is some sort of disaster
| nowadays.
|
| Just about the only question I've fielded every so often in
| the past several months have been people who found their way
| to an old tutorial based on the old system and them being
| confused about what happens, and a quick link to the current
| docs seems to resolve all their problems.
| cure wrote:
| Compared to the dumpster fire aka Python packaging, Go
| versioning is easy and predictable indeed.
| silverwind wrote:
| Until you hit the obscure rules around v2+ go modules.
| 2h wrote:
| I have been coding in Go for years, even professionally
| sometimes, and I have never had to use "v2" in any of my
| code. granted my stuff is not popular, but its really
| just a way for popular repos to handle big changes. Once
| I got past v1.9.9, I just changed to v1.10.0.
| blondin wrote:
| that was before go modules and now workspaces right?
| bayesian_horse wrote:
| Go gets around all these problems by being not all that popular
| (compared to Python and Node), not being as general purpose,
| and starting from a blank slate with control mostly from a
| corporate overlord. Static linking also simplifies things.
|
| Go doesn't make a habit of interfacing with lots of legacy
| C,C++ and Fortran Code. There's probably some of that. But
| nothing like Numpy, Scipy, Tensorflow...
| nicoburns wrote:
| Rust/Cargo manage to deal with interfacing with lots of
| legacy C and C++ code (not sure about Fortran) while still
| "just working" seamlessly. Usually it's easier to get a C
| library to build via the Rust package (just `cargo add
| package && cargo build`) than it is to integrate it into a C
| build system.
| NuSkooler wrote:
| Why not "simply" admit that Python and it's ecosystem while
| broad, are an absolute clusterfuck?
| whalesalad wrote:
| Because that's not at all the case and your rhetoric is what
| perpetuates the false image.
| stathibus wrote:
| Yeah it totally is, and people who pretend it's not are the
| problem.
|
| Somehow we ended up in a state where step one of doing
| anything new with python is to fire up an empty docker
| container. And I'm awfully tired of folks in the "python
| community" blaming the victims of the mess they made.
| JohnFen wrote:
| As a user, it kinda is.
|
| My recent example: last week, I was installing a new program
| that uses some python scripts during its operation. I could
| not get those scripts to run. I knew it was because of the
| usual python problem of the scripts not matching the version
| of python that was executing them, but figuring out how to
| fix that took me a full day.
|
| Just to make Python work. Not even as a dev.
|
| Wearing my user hat, this is a clusterfuck. There is no other
| language that I am exposed to that presents this sort of
| problem, and this is a very common issue with Python.
|
| This is why I start to get sweaty any time that I'm using
| software that involves Python. It turns into a crapshoot and
| half the time, it's going to cost me a lot of time and
| stress.
|
| > perpetuates the false image.
|
| You can believe it's a false image if you like, but there are
| a whole lot of end user experiences that indicate it's very
| real.
___________________________________________________________________
(page generated 2023-03-30 23:02 UTC)