[HN Gopher] A proof-of-concept Python executable built on Cosmop...
___________________________________________________________________
A proof-of-concept Python executable built on Cosmopolitan Libc
(2022)
Author : thunderbong
Score : 140 points
Date : 2024-04-15 11:17 UTC (11 hours ago)
(HTM) web link (ahgamut.github.io)
(TXT) w3m dump (ahgamut.github.io)
| Retr0id wrote:
| Based on the screen recording demo it looks unexpectedly slow.
| What's the bottleneck?
| BlueUmarell wrote:
| The "Python", the "Actually" and the "Portable" parts.
| Retr0id wrote:
| I'm aware of both Python and Cosmopolitan implementation
| details and neither of these things explain a multi-second
| page load.
| bluetomcat wrote:
| The interpreter is linked to a cross-platform libc-like
| library, and compiled to a funky executable format, supporting
| multiple OS targets simultaneously. The initialisation and
| startup are probably computationally-intensive.
| Retr0id wrote:
| We can see the initial startup speed from the time it takes
| the command in the terminal to print something, which isn't
| so bad (looks like under a second).
|
| Maybe Flask is forking on each request, which I could imagine
| being slow, but not slow _er_ than the initial process
| startup.
| jart wrote:
| ahgamut is a college student. He doesn't have $15,000
| workstations like us.
| jart wrote:
| Software linked with Cosmopolitan Libc usually goes 2x faster
| than Musl.
|
| - https://justine.lol/cosmo3/
|
| -
| https://twitter.com/JustineTunney/status/1726141024597324189...
|
| What exactly looked slow to you?
| Retr0id wrote:
| In the screen recorded demo gif in the article, it takes
| several seconds for the "/python" and "/cosmopolitan"
| requests to load (although the index page seemed to load
| pretty quick).
| jart wrote:
| It's probably due to Chrome penalizing the connection for
| not being HTTPS
| ahgamut wrote:
| My 4-core computer's probably building Cosmopolitan Libc in the
| background + the Python binary is 2.7 unoptimized. This blog
| post is originally from 2021.
| llm_nerd wrote:
| For those wondering, the submission is a library / tooling usage
| that generates a single binary that executes across Linux, Mac,
| Windows, and BSDs. The author defines that as Actually Portable
| (which was the confusing part to me as we all know that python
| runs on just about everything)
| Retr0id wrote:
| fwiw the definition doesn't come from the author, it's how
| Cosmopolitan/APE describes itself (APE being Actually Portable
| Executable - as opposed to a mere Portable Executable, aka EXE)
| thesuperbigfrog wrote:
| Justine's actually pdrtable executable website:
| https://justine.lol/ape.html
| moffkalast wrote:
| Couldn't the same result be achieved by just copying an entire
| virtualenv? Though that is rather annoying to source every time
| and in every terminal.
| salviati wrote:
| Any binary dependency you have would need to be present for
| all target architectures in your virtualenv for this to work.
| But when you pip install a package only the binary for your
| current platform gets installed.
| moffkalast wrote:
| I presume you'd have the same problem with the binary,
| there'd have to be separate builds for arm, x64, etc.
| ingenieroariel wrote:
| There are separate static builds for aarch64 and x86_64,
| those are put into a "fat binary" that runs on current
| generation hardware/software combos (osx with aarch64,
| win with x86_64 and linux with both). Just building for
| two things is better than figuring out the whole matrix
| of OS vs architectures which was the other option.
|
| For other architectures like Power/armv7/i686 the
| software can run using the Blink project [0] [0]
| https://github.com/jart/blink
| pfarrell wrote:
| If you are distributing software to an audience who want to
| use your thing without having to build it, then something
| like this is much more appealing than virtualenv.
| idoubtit wrote:
| By "Actually Portable", this article means that one can build a
| portable executable of `python` (compiler and runtime) which can
| run on many systems, including Windows and Linux.
|
| But the reality is a bit different . The article states that
| their result is slow and passes one third of the regression
| tests. So it's a proof of concept, not a portable executable one
| can use for real-world projects.
| ingenieroariel wrote:
| The reality is a bit different, the work on Python 3.6 was
| checked into the Cosmopolitan repo and I have been able to use
| it for production workloads that are in pure python. [0]
|
| As Cosmopolitan Libc has evolved, it has been possible to
| compile more software without modifications, and that includes
| latest Python through a project called superconfigure[1].
|
| Last person who tried to reproduce it from scratch did it last
| week (granted it too them a few days of solid work) but in the
| end they ended with a portable binary with Python 3.11.9,
| brotli, ssl and asyncio for their work related project.[2]
|
| [0]
| https://github.com/jart/cosmopolitan/tree/master/third_party...
| [1] https://github.com/ahgamut/superconfigure/ [2]
| https://github.com/croqaz/cpython/
| pierrebai wrote:
| I find the use case where you have a system in production but
| could not possibly use the official Python for that system...
| unusual?
|
| Double so since Python code (.py) is itself portable, and I
| would assume that the amount of work to make all the
| necessary extension be built into your APE vs just using the
| standard pip/pipenv/poetry/etc to be in favor of the latter.
|
| Not even talking about the maintainability of the resulting
| system by other people who'd have to learn how to handle
| building an APE.
| ingenieroariel wrote:
| The use case was a client creating electrification plans
| based on structures from satellite data. We were able to
| get rid of gdal/pandas/networkx and several other
| dependencies and ended up with a fast python based process
| that could be given to clients for them to reproduce on
| their own machines (windows workstations).
|
| In my use case, the niche is not having WSL/Docker
| available and letting end users repeat studies or re-run
| configuration scripts.
| JamesCoyne wrote:
| This is a good place to mention https://nuitka.net/ which aims to
| compile python programs into standalone binaries.
| ijustlovemath wrote:
| I've had better luck with pyinstaller! Nuitka is buggy when you
| use multiprocessing
| cmehdy wrote:
| Same here.
|
| Wrote a simple program to which I added Fire to parse
| arguments as a CLI, and Gooey/Tkinter for a GUI on top of it.
| To get a working standalone .exe file which does not require
| folders on the side, Pyinstaller did the job. Unfortunately
| it also triggers antivirus scans for some AVs..
| dboreham wrote:
| Not binaries, but I've had success with Shiv[1] which builds a
| Python application into a single-file package that can be run
| on any machine provided it has the Python binary installed (but
| not much else). We use it to ship products that run as-is on
| both Linux (including WSL2) and macos.
|
| [1] https://shiv.readthedocs.io/en/latest/
| ahgamut wrote:
| I've tried nuitka before, and a recent question that occurred
| to me was: does nuitka have an option to output just C files?
| Something like: python -m nuitka example.py
| --no-compile
|
| Might be interesting to see if the above is possible. We could
| get things like a completely-statically-compiled Python stdlib
| within the APE.
| tripflag wrote:
| I built a Win10 binary with Nuitka just the other day, and was
| surprised to find the pyinstaller binary had higher performance
| at runtime. Pyinstaller also had several other advantages, such
| as producing smaller binaries, and building faster, and Win7
| support.
|
| For reference, I kept notes on the exact commands I used:
| https://github.com/9001/copyparty/blob/hovudstraum/docs/nuit...
| FloatArtifact wrote:
| https://gregoryszorc.com/docs/python-build-standalone/main/
| FrustratedMonky wrote:
| The article's ambitious promise of an "Actually Portable" Python
| executable across diverse systems such as Windows and Linux ends
| up revealing a rather melancholic truth. What we encounter
| instead is a slow, underperforming proof of concept that passes
| merely a third of its tests.
|
| Is this not, in some perverse way, reflective of our contemporary
| technological striving, where the ideal of universality often
| crumbles under the weight of practical limitations?
|
| Are we not, in our pursuit of perfect tooling, always chasing a
| mirage? The reality, starkly exposed by these tools, might be
| that our pursuit is as flawed and fragmented as the tools
| themselves.
| kitd wrote:
| The good news is that it's 2024, this article is a few years
| old and many of the problems mentioned have been (or are being)
| ironed out.
| m0shen wrote:
| This article was written when cosmopolitan:
| https://justine.lol/cosmopolitan/index.html was v2. v3 has since
| been released: https://justine.lol/cosmo3/ .
|
| https://cosmo.zip/ Also contains a python build
| https://cosmo.zip/pub/cosmos/bin/python , though I'm not sure if
| the problems mentioned in the article are solved.
| ahgamut wrote:
| most likely before v2 as well. The original post is from July
| 2021, which is a bit after Cosmopolitan Libc's v1 IIRC.
| m0shen wrote:
| Yes, you're correct. I had in mind the 2022 date of the
| update. It also looks like you've updated the article again
| today! Thanks
| ahgamut wrote:
| Hey, this is my post from 2021, when I was testing Python2.7 and
| Python3.6 with Cosmopolitan Libc on an old 4-thread Haswell. It's
| now a lot easier to build Python (and gcc, gnu coreutils, curl
| etc.), and the binaries are faster, multi-threaded, and quite
| convenient to use. There are lots of interesting directions to
| explore when it comes to building software with Cosmopolitan
| Libc.
| ingenieroariel wrote:
| Thanks for updating the blog post too with the datasette screen
| recording, the speed difference is quite noticeable.
|
| Adding it here since a few people were wondering about it in
| the comments, but feel free to check the original article for
| the 2024 update:
|
| https://ahgamut.github.io/images/ape-datasette.gif
| johnea wrote:
| HaHa! 2 weeks too late for April Fools...
___________________________________________________________________
(page generated 2024-04-15 23:01 UTC)