[HN Gopher] Pex: A tool for generating .pex (Python EXecutable) ...
___________________________________________________________________
Pex: A tool for generating .pex (Python EXecutable) files, lock
files and venvs
Author : eamag
Score : 59 points
Date : 2024-11-15 16:13 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| MikeTheGreat wrote:
| I'm always on the lookout for ways of packaging Python
| programs/script-piles into single-file executables, so thank you
| for posting this!
|
| The GitHub README says it builds on the Python .ZIP App format
| (PEP441), which in turns says you can put your app in a .ZIP file
| and Python will run it as an app.
|
| I think I went this route with PyInstaller's one-file output
| option. But I found it to be too slow because (1) my app was a
| CLI app and I needed it to start, run, and end quickly, (2) I
| imported something from Django because it was useful (which
| ballooned the size), and (3) the single-file .ZIP needed to be
| extracted into a temporary directory, run, and then deleted every
| time (!!!).
|
| Does anyone know how Pex / PEP441 deals with this? Is the
| extract-run-delete thing normal/standard? Is there a way to cache
| the extracted app?
| tylerhou wrote:
| My guess is that without operating system support or
| equivalent, there is not an easy way to avoid extracting the
| zip file. (You could patch syscalls in the Python interpreter
| to read from the zip file for certain paths, but that might be
| hacky.)
|
| I did find https://github.com/google/mount-zip which might be
| useful, but you would likely have to still mount the zip
| manually. However, you don't have to worry as much about
| cleaning up after yourself.
| glass-z13 wrote:
| I used python to write a small utility that i wanted to share
| with a coworker, ended up having to rewrite it in Go because i
| spent 3 hours trying to compile the python version to an exe so
| that it was easier to run on their machine.
|
| I wonder why isn't there an easy way to just package the
| interpreter inside an EXE for people that don't care about binary
| size just so that it would make it seamless to package up
| utilities written in python.
| Esgrove wrote:
| There is: PyInstaller
| glass-z13 wrote:
| Seems like i have it in my browsing history, i remember not
| being able to run the executable it produced, not being able
| to find a library it was supposed to have. This was just
| using the default "pyinstaller your_program.py", and i was
| frustrated enough to not go deeper into why that was. Will
| definitely give it a try again in the future
| randlet wrote:
| Pyinstaller can be a bit fiddly to get right initially if
| you have a lot of package dependencies, but other than that
| it works really well.
| Numerlor wrote:
| Using the spec files for persistent readable
| configuration also goes a long way, if you treat
| pyinstaller as a python module you can automate it whole
| with just python, including the spec files as it executes
| them as python scripts
|
| I've had automated builds running mostly untouched for
| years here https://github.com/Numerlor/Auto_Neutron/blob/
| master/pyinsta..., though it doesn't have any
| particularly troublesome packages other than PySide6
| WhyCause wrote:
| PyInstaller-made executables also used to have a habit of
| getting flagged by security software as malicious (maybe
| that's why you couldn't run it?) -- apparently, so many
| malware writers used it that it ruined the party for
| everyone.
|
| Fortunately, that was only the 32-bit version of Python
| 2.7. Using 64-bit versions or Python 3 was enough to not
| get flagged as malicious. I figured that out when I decided
| I didn't want to teach myself Go just then to deploy
| something that had worked the day before.
| ktm5j wrote:
| Yep I used it at my last job and it worked great! Startup
| times were horrible, but that didn't matter so much to us and
| it solved tons of problems we had with people messing up
| their python environments. Takes some tweaking to get certain
| modules (like scikit-rf) to work, but never found an issue
| that couldn't be solved.
|
| Would recommend!
| dbaupp wrote:
| As of a few months ago, pex supports including an interpreter,
| either directly inline or lazily downloaded (and cached):
| https://docs.pex-tool.org/scie.html
| toprerules wrote:
| I also end up avoiding python not because it's a bad language,
| but because it's so much more convenient to work with compiled
| artifacts that worry about shipping around a bunch of source or
| object files and a full compiler. Not to mention the speed of
| using Go and the ability to write concurrent code without a lot
| of effort.
|
| The only time I really use python is when I need a quick and
| dirty script.
| quietbritishjim wrote:
| It's probably on the list that you've tried, but I've had most
| luck with Nuitka (in standalone mode, not onefile). Unlike
| pyinstaller, it really is the executable it claims to be, not
| just an elaborate zip file containing python.exe and your
| source files.
| rcleveng wrote:
| I've come to realize that putting everything into a container
| is the only viable way to share a python program.
|
| I'll certainly checkout PEX, I think that distribution of a
| binary is likely the largest one for Python right now. Java
| solved it with a JAR file, static-compiled binaries for most
| compiled languages.
|
| At Google PAR files have been quite a useful way to handle this
| for at least 18 years now, hope this can get reasonably solved
| everywhere.
| BiteCode_dev wrote:
| I know it's not very nice to post that here, but given it's hard
| to get good info about python because the ecosystem is so huge,
| I'd rather give it straight:
|
| Don't use pex, use shiv: https://shiv.readthedocs.io/
|
| Pex doesn't work on windows, is slower than shiv and shiv solves
| the resource files problem that you will encounter with
| dependencies like django. To achieve this, shiv unzips the whole
| zipapp in a cache on disk and create a transparent proxy to it.
|
| Still, I wish zipapps had better tooling honestly. Using either
| pex or shiv is more complex than it needs to be.
|
| I hope uv will eventually give you something like "uv zipapp" and
| make it good and simple.
|
| Right now, to bundle all deps for shiv you have to download first
| all wheels, then figure out the proper incantation to include
| them. And if you use WSGI, you probably want some kind of
| dedicated entry point.
|
| The tech is cool, but it deserves maturing to be a true ".war for
| python".
| fsloth wrote:
| "Pex doesn't work on windows"
|
| Oh that's such a weird limitation.
|
| In the early 2000's open source hubris, next year was always
| the year of the Linux desktop. Since then consumer windows
| matured into a totally ok OS and one with the best support for
| graphics and C++ development at that.
|
| Non-windows support nowadays is a fairly strong signal of a
| non-serious software offering if there is no obvious reason for
| it. And that's totally fine, hobby tools developed by
| enthusiasts rock - but they are not _industrial_ in scope as
| such.
| sneed_chucker wrote:
| A lot of serious software offerings are only concerned with
| the server use case, modern servers run Linux unless there's
| a good reason not to, and modern windows has more than one
| acceptable way to run Linux binaries if you absolutely have
| to.
| MikeTheGreat wrote:
| While we're posting not-nice things, I'll throw my current
| favorite out: pipx ( https://github.com/pypa/pipx ). I use
| Python's Poetry to build a wheel and then pipx to install it.
| Super-easy, barely an inconvenience :)
| BiteCode_dev wrote:
| pipx is orthogonal to the problem pex and shiv are solving
| since:
|
| - it doesn't provide a single file for a full project with
| all dependencies.
|
| - it requires internet access on the target machine to
| install things.
|
| - it can't give you access to underlying API for things like
| wsgi.
|
| - it requires pipx on the target machine.
|
| - it requires a wheels made from a build, which many projects
| don't have (such as a lot of web projects).
| packetlost wrote:
| I feel like, in general, you should be able to treat an archive
| as a filesystem. Zipapps could be so much easier if it did that
| transparently.
| BiteCode_dev wrote:
| It can, but:
|
| - It's slow.
|
| - The underlying python code cannot use open() to access its
| own resources, but must use pkg_resources, and most packages
| on pypi don't do that because most dev don't even know it
| exists or what problem is soves.
|
| - _ _ file _ _ is not available, breaking a lot of things.
| zip_safe is a build option, but actual compliance to it is
| hard work and rare.
| packetlost wrote:
| I was thinking more along the lines of using namespaces and
| FUSE. I'm aware that _Python_ kinda treats them like
| folders transparently, but not completely for reasons (and
| more) that you described.
| BiteCode_dev wrote:
| That's a cool idea.
|
| Wonder what the FUSE equivalent for windows is.
| packetlost wrote:
| Probably Windows Projected FileSystem:
| https://learn.microsoft.com/en-
| us/windows/win32/projfs/proje...
| dbaupp wrote:
| FWIW, pex now also has options to unzip the archive to a cache
| directory on startup (I believe this happens by default now,
| but am not at a computer to confirm), to side step the zipapp
| limitations that you reference.
| BiteCode_dev wrote:
| I just checked, and there is indeed a `--pex-root` option
| with and even "-c" to specify a custom entry point.
|
| Thanks for pointing it out.
| emmelaich wrote:
| > .war for python
|
| It shouldn't be that hard to get a general 'jar/war', not just
| for python.
|
| Perhaps jart's cosmolitan libc with a zip file appendage which
| it auto extracts and optionally caches on first use.
|
| A heavyweight solution, but still lighter than Docker.
| BiteCode_dev wrote:
| It would today, because macos requires signing, and windows
| has "the mark of the web".
|
| They don't apply to scripts, so Python is fine, but they do
| apply to executables.
|
| Having to sign your zip every time you produce it would
| nullify the benefit of the concept. And because the checksum
| of the executable file change according to the content of the
| zip, you can't just sign the cosmopolitan part.
| serjester wrote:
| I think the easiest way to do this now is with uv and their
| support for inline metadata. At this point you can just have the
| user install uv with a single command and then have the user run
| the script with `uv run example.py`.
| d0mine wrote:
| --script appears to be missing if you meant
| https://peps.python.org/pep-0723/
| khaledh wrote:
| We've been using pex for a long while to package PySpark jobs
| with dependencies into a single file. Saves a ton of time vs. the
| old way of building/deploying docker images.
|
| https://spark.apache.org/docs/latest/api/python/user_guide/p...
| gavmor wrote:
| I can see how .pex or shiv's zippaps live at the Pareto threshold
| where converting a Python app to an 80% (or less...) executable
| takes only 20% (or less!) of the effort that would be spent re-
| writing the app in eg C, C++, Rust, or Golang. Still, it breaks
| my heart to see all this work being done to keep developers
| employed as "Python developers", instead of leveling them up to a
| broader programming skillset. It seems as if the industry is
| investing rather heavily in a local maximum.
|
| But, of course, the market is has information that I don't.
| Numerlor wrote:
| Eh I wouldn't call PEX proper executables that'd replace the
| languages above as they still need python, but I still prefer
| python for things like simple Qt apps even if it means using
| e.g. PyInstaller
| mappu wrote:
| Pex and Shiv both require the end user to have Python installed,
| so I think PyOxidizer is the best tool in this category:
| https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_compa...
| smallmouth wrote:
| I thought this was going to be a post about plumbing and then I
| read further and was even more excited to say the phrase "python
| executable". Very interesting. Thanks!
___________________________________________________________________
(page generated 2024-11-15 23:00 UTC)