[HN Gopher] I just want to run this one Python script
___________________________________________________________________
I just want to run this one Python script
Author : teddyh
Score : 125 points
Date : 2021-11-08 08:15 UTC (14 hours ago)
(HTM) web link (www.die-welt.net)
(TXT) w3m dump (www.die-welt.net)
| HerrMonnezza wrote:
| Perl has had a similar trick for quite a while:
| eval 'exec /usr/bin/perl -w -S $0 ${1+"$@"}' if 0; # not
| running under some shell
|
| See https://stackoverflow.com/questions/9143726/what-means-
| not-r...
| ktpsns wrote:
| To be fair, this blog article is not much about python but more
| about the bash (shell), the weirdness of quoting and the miracles
| of exec(3).
| teddyh wrote:
| Well, it also wouln't have worked if not for Python ignoring
| raw quoted strings. It's those _two_ behaviours of the two
| languages which makes this limited polyglot script possible.
| callfx wrote:
| I don't get it either. This is pretty basic shell knowledge. Is
| the goal to have at least one Python article on the front page
| every day?
| warvariuc wrote:
| Or #!/bin/sh """:" # bash code
| here; finds a suitable python interpreter and execs this file.
| for pyver in 3.6 3.7 3.8 3.9; do which python$pyver >
| /dev/null 2>&1 && exec python$pyver "$0" "$@" done
| echo "No appropriate python interpreter found." >&2 exit
| 1 ":"""
| cozzyd wrote:
| don't you want to go in descending order?
| dagw wrote:
| That approach will break now that python 3.10 is out
| mattnewton wrote:
| Or, it won't break your code if you haven't tested in 3.10
| yet ;)
| ByThyGrace wrote:
| Is it still bash code if you're running /bin/sh?
| tempodox wrote:
| Not for shellcheck(1). It warns about bashisms if it sees
| `/bin/sh` after the hash-bang.
| mrlemke wrote:
| Presumably not, it depends on your system. Some people
| symlink sh to bash.
| thundergolfer wrote:
| I engaged in this sort of shenanigans recently when I wanted to
| pull a Python interpreter out of a .zip file, save it somewhere
| temporary, and then execute the .zip file with that extracted
| Python interpreter, all when you did ./runme.zip.
|
| It's then a (mostly) hermetic Python executable.
|
| When I got it working, it felt like the first genuine 'hack' of
| my career. Something that's awful, novel*, and awesome at the
| same time.
|
| * novel at least to me. It wasn't something I copied off stack
| overflow or a blog post.
| jfrunyon wrote:
| This sounds like "freezing" a script, which there are many
| solutions to do, f.e. https://pypi.org/project/cx-Freeze/
|
| You might be running with your interpreter, but are you running
| with your standard library? The solutions out there will make
| sure to package up not only the interpreter and your files, but
| also any other files you import.
| syllogism wrote:
| You might be interested in pex:
| https://pex.readthedocs.io/en/latest/
| gompertz wrote:
| You may also want to look into 'EEE'. EEE is a program that
| combines multiple files and actions into a compressed
| Windows, Linux or Mac OS X (Darwin). executable.
| http://www.erikveen.dds.nl/eee/index.html
| brnt wrote:
| Unrelated, but it's been some time since I last heard or
| thought about DDS.nl! This was an early (90ies?) Dutch
| internet community which completely left my field of view
| for decades but apparently is still a thing!
| BiteCode_dev wrote:
| Pex doesn't embed the executable.
|
| Besides, it doesn't work on Windows, if you want a zipapp,
| shiv is a better alternative: http://shiv.readthedocs.io
| bitwize wrote:
| "This is impressive, disturbing, and makes me uncomfortable!"
| --Strong Bad
| kazinator wrote:
| One fix would have been for distros to have a special symlink
| like `/usr/bin/pythonx`. This could be a link to any version of
| Python, guaranteed to exist as long as at least one Python is
| installed, to be used by programs that can work with any version.
|
| Probably too late for that; if you add this, you'are just
| complicating things: scripts then have to worry about _pythonx_
| not existing.
|
| The _env_ program could have a feature to help here. GNU
| Coreutils could adopt an _env_ feature whereby it searches for
| multiple programs. Because of argument limits in the hash bang
| command line this would have to be encoded as one argument.
| #!/usr/bin/env python|python3
|
| So here, env sees there is a pipe in the middle of the program
| name. It splits the program name into two, and first searches the
| PATH for _python_. If that is not found, it searches the path for
| _python3_. Alternately, rather than giving preference to the left
| item, it could prefer the first one that occurs in the PATH. It
| could search the PATH once, stopping whenever any of the
| indicated names is found.
|
| Or, how about supporting regex in env, if the first character is
| circumflex: #!/usr/bin/env ^python3?
|
| The advantage is that the specification is condensed. On some
| systems, there is a strict limit, like 32 characters, into which
| the hash bang line has to fit, so this could be helpful against
| that. The problem with that is that *env now has to read
| directory entries in user space to do the matching; it can't do a
| simple access() test on the name in every directory. This
| wholesale directory reading fails to take advantage of whatever
| name lookup caching mechanisms there are in the underlying
| kernel.
| teddyh wrote:
| > _/ usr/bin/pythonx_
|
| Maybe /usr/bin/sensible-python or /etc/alternatives/python?
| jfrunyon wrote:
| That's what /usr/bin/python is, though? A link to any version
| of Python so long as any version is installed.
| W0lf wrote:
| Firefox has been using these hacks described in the article for
| quite some time now for its build command script mach[1]
|
| [1] https://searchfox.org/mozilla-central/source/mach
|
| Pretty neat, although its support for Windows hosts seems to rely
| on Powershell (or dispatches completely to anything other than
| Python. Not sure about that)
| nunez wrote:
| i usually do this when I want to create docstring-like behavior
| cat <<-DOC line 1 line 2 line 3 DOC
| teddyh wrote:
| I once created a set of shell functions which I wanted to have
| a docstring-like functionality. The solution I came up with was
| to have each shell function start with the : command with a
| string as an argument. Since : is an actual command, not a
| comment, it was preserved as part of the function, and could be
| extracted at runtime using relatively simple parsing to do
| introspection.
|
| Example: foo(){ : "This is a docstring
| for the foo() function" bar --verbose | baz --quiet
| }
|
| For those who don't know: ":" is a shell built-in command which
| does nothing and ignores all arguments. It was the way in which
| shell scripts had comments before the modern syntax of using #
| for comments was introduced.
| sdeer wrote:
| Any reason why he can't use #!/usr/bin/env python
|
| here?
| kevinmgranger wrote:
| I think it's inconsistent whether or not `python` exists if you
| only have python3 installed, depending upon the OS.
|
| This is despite guidance from PEP 0394[1]
|
| 1: https://www.python.org/dev/peps/pep-0394/
| emeraldd wrote:
| Another point here is that python is still commonly a python
| 2.7 on a lot of deployed systems. It's going to be years before
| it's safe to assume that python is anything else...
|
| (Ubuntu LTSs and macOs are an example ...)
| jfrunyon wrote:
| No, that's not another point. It's a point that was made
| hours before you in a sibling comment AND it's a point that
| is completely irrelevant. Read the second sentence of TFA.
| MrStonedOne wrote:
| 'python' might not be a valid command because 'python3' exists
| and 'python' does not.
|
| When designing things for 'just work' scripts you have to take
| esoteric environments into account.
|
| There is no guarantee that '#!/usr/bin/env python' will not
| error in an environment with only python3 installed.
| jfrunyon wrote:
| There is also no guarantee that Python is installed at all,
| and there is no guarantee that it is on the path even if it
| is installed.
| stavros wrote:
| Are you sure? AFAIK python points to the default Python, be
| that 2 or 3.
| nomel wrote:
| It's the other way around, in all systems I've found.
| python3 (or python3.x) is the actual executable, with
| python being a symlink to, usually, python 2.
| stavros wrote:
| Sorry, typo. I meant to say `python` is a symlink to the
| default Python (you can sometimes change the default
| Python), but it'll almost always exist and point to a
| valid interpreter.
| theamk wrote:
| This link may not exist. Notably, if you start with minimal
| bootstrap of ubuntu focal (20.04) and install "python3"
| package, you'll end up with "python3" and not "python"
| maweki wrote:
| If you're living in limbo where "python" still means "python2",
| which isn't installed, as opposed to "python3", then this
| wouldn't work.
|
| Or theoretically also the other way around.
| talideon wrote:
| If you don't care which version of python it runs under, you
| do this: #!/usr/bin/env python
|
| If you do, then `python2` or `python3` will be around, and
| you should specify them explicitly instead of `python`. I'm
| happy to be corrected on this one, but I always slap this at
| the top of my scripts: #!/usr/bin/env
| python3
|
| I'm not aware of anywhere that wouldn't invoke a Python 3
| interpreter if one is installed.
|
| The original post just seems like one big faff.
| maweki wrote:
| PEP 394 (which is now superseeded) clearly states that
| "python" shall invoke python2 and not python3
|
| https://www.python.org/dev/peps/pep-0394/
|
| "The python command should always invoke Python 2 (to
| prevent hard-to-diagnose errors when Python 2 code is run
| on Python 3)."
|
| So there was indeed a time where the mentioned scenario was
| possible.
|
| Edit: love those people who got to downvote reality. At no
| point was python pointing to either 2 or 3 will-nilly, but
| there was always a PEP specifying best practice behaviour.
| eesmith wrote:
| > "python" shall invoke python2 and not python3
|
| The current version of PEP 394 says:
|
| ] If the python command is installed, it is expected to
| invoke either the same version of Python as the python3
| command or as the python2 command.
|
| ] Distributors may choose to set the behavior of the
| python command as follows: python2,
| python3, not provide python command,
| allow python to be configurable by an end user or a
| system administrator.
|
| The part you quoted is from the section "History of this
| PEP". Just before your quoted text is:
|
| ] This PEP originally provided a very simple mechanism
| ...
|
| and just after it is:
|
| ] However, these recommendations implicitly assumed that
| Python 2 would always be available.
|
| This historic advice is no longer recommended.
|
| The change was made on July 5, 2019 - https://github.com/
| python/peps/commit/ae932bd6fd2c493d7d64ce... .
| remram wrote:
| The PEP is not superseded, still active. You got this
| quote from the "history of this PEP" section. This is not
| what the PEP currently mandates and it hasn't for 2.5
| years: https://github.com/python/peps/pull/989
| remram wrote:
| My system doesn't have a `python`. On Ubuntu you need to
| install one of the `python-is-python2` or `python-is-
| python3` packages for the `python` command to be available.
| Command 'python' not found, did you mean:
| command 'python3' from deb python3 command 'python'
| from deb python-is-python3
|
| https://packages.ubuntu.com/focal/python-is-python3
| eikenberry wrote:
| All Debian based distros are this way, not just Ubuntu.
| remram wrote:
| And a lot of non-Debian distros as well. Python 2 is
| officially retired after all.
| II2II wrote:
| The main example I can think of is when you don't know what
| python links to. For example, an Arch system will links to some
| version of Python 3. I believe that Debian based systems link
| to some version of Python 2. You could just swap python with
| python3, but if your script supports Python 2 as well _and_ you
| 're trying to run it on a system without Python 3, it will
| fail.
|
| Of course, if your script will run under both Python 2 and 3, I
| don't know why you would want to use these contrivances to
| simply give preference to Python 3.
| kevinmgranger wrote:
| > I don't know why you would want to use these contrivances
| to simply give preference to Python 3.
|
| Performance gains, preferring future-looking development, and
| better error messages.
| theon144 wrote:
| >Ever had a script that's compatible with both, Python 2 and
| 3...
|
| The article is literally about this scenario, which is why
| I'm also confused. Yes, `/usr/bin/env python` is a bad idea
| IF your script depends on a certain version, but that's
| exactly what the author doesn't need!
| Dunedan wrote:
| > I believe that Debian based systems link to some version of
| Python 2.
|
| Since Debian 11 and Ubuntu 20.04 LTS it's a matter of
| installing the matching package:
|
| python-is-python2 [1] for having /usr/bin/python link to
| Python 2.7, python-is-python3 [2] for having /usr/bin/python
| link to Python 3.x or none of them to avoid having
| /usr/bin/python at all. The latter one will still result in a
| working system, as all Python scripts shipped by
| Debian/Ubuntu nowadays explicitly specify /usr/bin/python2 or
| /usr/bin/python3.
|
| [1]: https://packages.debian.org/bullseye/python-is-python2
|
| [2]: https://packages.debian.org/bullseye/python-is-python3
| jfrunyon wrote:
| Oh god. No, not having a `/usr/bin/python` will _not_
| result in a working system. Does Debian really think they
| 're the only software distributor that their users use?
| chillfox wrote:
| Why make it bash specific instead of sh?
| drewg123 wrote:
| Why make this bash specific by using command -v? How is that
| better than "which" which works on any shell I can think of?
|
| Running FreeBSD, one of my pet peeves is folks who assume
| /bin/bash is going to exist everywhere.
| tyingq wrote:
| Looks like FreeBSD added "command -v" to /bin/sh some time ago.
| Though, yes, putting bash in the interpreter line doesn't help
| that.
|
| https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=77259
| drewg123 wrote:
| What about ash on ubuntu, etc?
| Shish2k wrote:
| ash also supports `command -v`
| SuchAnonMuchWow wrote:
| I always though that "command -v" was more portable than
| "which": "command -v" is specified in posix and should always
| be present but "which" is not.
|
| But all my knowledge about that comes from a small drama some
| times ago when a debian maintainer wanted to remove "which"
| from its current package (see
| https://lwn.net/SubscriberLink/874049/bf89a969ed3dde87/).
| macintux wrote:
| Discussed extensively here recently:
| https://news.ycombinator.com/item?id=29026623
| lambic wrote:
| When I saw the headline I thought the article was going to be
| about the ridiculously confusing tooling landscape around python
| development. I'm never sure what package manager or virtual
| environment thingy I'm supposed to be using.
| runjake wrote:
| > I'm never sure what package manager or virtual environment
| thingy I'm supposed to be using.
|
| pip3 and virtualenv.
|
| Both are pretty easy to learn. Everything else is pretty much
| an abstraction layer on top of one or both of the above.
| nomel wrote:
| pip3 assumes you have python 3, and the correct version of
| it, already installed. virtualenv requires that you
| _manually_ juggle command line contexts, by running some
| nested scripts. For virtualenv, you can set up bash script
| that detect when you cd into a folder, but wow.
| ed_elliott_asc wrote:
| I've settled on using private venv environments for each
| repo/app and that seems to work well enough tbh.
| version_five wrote:
| > I'm never sure what package manager or virtual environment
| thingy I'm supposed to be using
|
| This feels a bit anti-intellectual to me. Like any other thing,
| it takes a bit of reading to figure our virtual environments,
| but its literally one line to create a new one one you know how
| (I almost always use Conda). I don't see how this is different
| than the setup used for any development, and I can't imagine
| it's a major impediment, certainly not a lasting one, to people
| working in python. You could always open a colab notebook if
| you really don't want to be bothered.
| orf wrote:
| Just use poetry[1], there isn't any reason not to. You can
| create and manage virtualenvs by hand if you'd like, but why do
| that to yourself?
|
| 1. https://python-poetry.org/
| fmakunbound wrote:
| I sure want to believe this is the last tooling I'll need to
| use for Python tooling nirvana, but that's what they told me
| about pip, pip3, venv, vitrualenv, conda.. And one coworker
| just suggested I just run the script in Docker. Ugh
| orf wrote:
| I'm not sure anyone said pip or virtualenv where the end of
| the road for Python tooling nirvana. They are just building
| blocks, poetry (or other tools) stick them together.
|
| That being said, I've been using Python for 16 years now
| and for me poetry is pretty close to nirvana. Not perfect,
| but really good. Handles virtualenv management, locking,
| dependency specification, etc etc.
| exdsq wrote:
| Just freeze requirements (pip freeze > requirements.txt) and
| load them into whatever env you're comfortable using. That's
| worked for me
| serial_dev wrote:
| Having three different approaches in the three replies to this
| comment is just perfect.
| burnished wrote:
| It mirrors my own experience. Especially the answer that
| strives to be condescending, as if you are the only one who
| is having this problem.
| orf wrote:
| It's only two, and there are two different approaches you can
| take:
|
| 1. Use something that manages virtualenvs + packages for you.
|
| 2. Don't use something that manages virtualenvs + packages
| for you.
|
| For 1 you have poetry or pipenv. For 2 you have... well
| virtualenv/pip. It's really not that complex and I don't get
| where this sentiment comes from.
| Spivak wrote:
| One problem is that the command `readlink --f` isn't portable, to
| macOS for example, and there's not really a satisfying solution.
|
| https://github.com/whatwg/html-build/issues/90
| jfrunyon wrote:
| Yes, I maintain several packages which are* compatible with
| either Python 2 or 3. Those packages have a defined entrypoint
| which begins with `#!/usr/bin/python`. For packages which are
| Windows-compatible, they end in .py.
|
| If those two things don't cause the script to be executed by
| Python then frankly the user's system is broken and I'm not going
| to worry about it. (Depending on what you're writing,
| `#!/usr/bin/env python` may be a better option, so that you
| respect the PATH - including venv's.)
|
| * Well, they _were_ compatible, but I no longer test against Py2.
| tyingq wrote:
| I had commented on something related to this a couple of weeks
| ago here: https://news.ycombinator.com/item?id=28930278
|
| The top reply had a great suggestion. Putting this at the top of
| a python 3.6+ required script results in a reasonable error for
| those with older /usr/bin/python interpreters, instead of a
| generic syntax error stack trace.
| #!/usr/bin/python 0_0 # Python >= 3.6 is required
| vortico wrote:
| I usually put this at the top, so it raises a syntax error if
| using an old version. f"Python 3.6+ is
| required"
| BiteCode_dev wrote:
| Why use a hack when you can do it cleanly and give a good error
| message to your user ? import sys
| if sys.version_info < (3, 6): print('This script
| requires python 3.6+) else: from
| module import main main()
|
| If you want all of it to be in one file, you can use a zipapp:
| https://docs.python.org/fr/3/library/zipapp.html
|
| If you don't want to make the zipapp manually, or want it to
| automatically extract everything on first run, use shiv:
|
| http://shiv.readthedocs.io
| folkrav wrote:
| Yep. I'd add that making sure your script returns a proper
| error code and to output error logging to `stderr` is good
| practice too. No additional imports required in this case as
| you already have `sys`, by just replacing the `print('...')`
| with `sys.exit('...')`.
| tyingq wrote:
| There's still lots of use cases for sharing a single file
| script where you would like people with Python2 as
| /usr/bin/python to get a reasonable error. Zipapps don't work
| well as gists, for example.
|
| Edit: Zipapps feel hacky to me. I get that not everyone would
| share that opinion.
| BiteCode_dev wrote:
| A zipapp is not a hack, it's an official feature of the
| python stdlib (as shown in the parent comment), and is made
| for the specific use case of grouping several files into
| one module for ease of sharing.
|
| All of this works fine with python 2.7 as long as you put
| the rest of your code in your main() entry point, which you
| should do anyway. So an f-string would cause no problem.
|
| Also, "informative stack trace" is only informative to
| another dev, and an experienced one at that, given how many
| juniors I encounter that can't read a traceback.
|
| Indeed, you script users may not be devs, and for those, a
| traceback would be confusing. For this reason, a script
| should only output a traceback in dev mode, but in
| production, if the dev flag is not set (-X dev), you should
| never see a traceback. It's a good practice to suppress it
| with sys.excepthook, and log it into a file instead.
|
| Using a syntax error like you do for this purpose IS the
| big hack.
|
| Now I'm not against a hack. I can see the value of it for
| your own scripts, of things that you code quick and dirty
| style. After all, I don't always want to have a clean entry
| point and more than one file.
| tyingq wrote:
| Can you trap initial parse errors with sys.excepthook? I
| don't think you can, at least not in a single file
| script.
| BiteCode_dev wrote:
| It is not necessary, as I indicated before, if you
| isolate your entry point, you will not run into syntax
| errors (unless you publish broken code).
| tyingq wrote:
| Well, where broken code is code that assumes
| /usr/bin/python would be python3.
| BiteCode_dev wrote:
| Again, not a problem if you isolate your entry point.
| Python will not parse the incompatible code since it's
| never imported.
|
| You keep repeating the same thing a if you ignored this
| essential part.
| tyingq wrote:
| >You keep repeating the same thing a if you ignored this
| essential part.
|
| Well, yes, and you keep insisting that everything is
| fine, as-is. There was a missed opportunity here.
| [deleted]
| slightwinder wrote:
| While this looks funny and smart, it's not really a good
| solution as it lacks well communicated documentation. Python
| has a proper way to check for version, which is backward-
| compatible till version 2.0, so around 20+ years.
| import sys if sys.version_info.major < 3: raise
| Exception('Need Python 3 or higher')
|
| EDIT: correction, sys.version_info.major is only there since
| Python 2.7. For older versions sys.version_info[0] should be
| used.
| tyingq wrote:
| >so around 20+ years.
|
| Right, but if there's anything below that that would cause a
| syntax error, like an f-string, that Exception you created
| never happens. The end user just sees a syntax error.
|
| It's unfortunate there's not a better way to handle it,
| especially with so many people still having /usr/bin/python
| linked to a 2.x version.
| [deleted]
| maple3142 wrote:
| I think this is worse because once you have a python 3 only
| syntax in your script (such as f-string), python 2 will
| simply throw syntax error because it need to parse the file
| to execute. The original solution looks hacky, but it can
| actually inform user to use python 3.
| julienpalard wrote:
| apt install python-is-python3
| hprotagonist wrote:
| this is reasonably close to what setuptools' entry_points does
| for creating little shell scripts that call a callable inside
| your installed python packages.
| ddtaylor wrote:
| Python 2 should be dead by now. Please, just let it die.
| rsj_hn wrote:
| Friend, we're still running COBOL.
|
| There are still shops with computers running windows XP,
| performing various scripted automation or batch tasks.
|
| The idea that some running system will be taken offline just
| because the language is old is incompatible with a lot of real
| world needs.
| int_19h wrote:
| The problem isn't so much so that Python 2.7 _the language_
| is old, but rather that the implementation is no longer
| supported - including security issues.
|
| Having said that, 2.7 still has commercial third-party
| support, e.g. from ActiveState. Which, presumably, will be
| around for as long as there are enough users willing to pay
| for it.
| at_a_remove wrote:
| Technology sticks around for a lot longer than people think,
| for reasons most cannot imagine until they are faced with these
| very real business scenarios. I should know, I am living with
| Python 2 right now.
| nomel wrote:
| With existing, mostly stable, offline code, I've seen that
| switching to python 3 is not easy to justify, especially if
| you're short staffed. I would say that type hints are the
| most tangible benefit, but most IDEs support the, less than
| ideal, python 2 versions.
| nijave wrote:
| It should be but it's included in LTS software which is still
| supported
|
| CentOS 7, for instance, still has 3 more years of patch support
| mistrial9 wrote:
| could the zealots at Debian-Ubuntu please stop purposefully
| damaging the Python 2.7 interpreter, please? thx
___________________________________________________________________
(page generated 2021-11-08 23:02 UTC)