[HN Gopher] Automate your Python project with Makefile (2021)
___________________________________________________________________
Automate your Python project with Makefile (2021)
Author : asicsp
Score : 44 points
Date : 2023-03-27 15:50 UTC (7 hours ago)
(HTM) web link (antonz.org)
(TXT) w3m dump (antonz.org)
| rcarmo wrote:
| I've been doing this for decades now. Anything I do has "make
| deps", "make serve", etc. I don't care that Make "wasn't made for
| this", it is ubiquitous and reliable at it.
| bragr wrote:
| I would not recommend make for python projects as make does not
| play nicely with virtualenvs and the environment variables that
| power them. You can make it work, but you'll be fighting it
| constantly.
| rcarmo wrote:
| How so? I have a Makefile in all of my Python projects. Never
| had issues.
| kelseyfrog wrote:
| Are you specifically talking about virtualenv or the concept of
| venvs in general?
|
| I've used make with conda, for example, for the last five years
| with no env issues at all, so some clarity or specific problem
| cases would help.
| kortex wrote:
| What exactly are you doing that hits friction? You shouldn't be
| using "make" to "activate" your env - that should occur
| "outside" your makefile. It plays totally nice if you aren't
| trying to use make to set environmental state.
| hprotagonist wrote:
| "never activate venvs in a headless context" is sage advice and
| not just for make.
| cpburns2009 wrote:
| An easy way to use virtual environments with Make is to create
| a wrapper script to activate the virtual environment and run a
| command in its context: # venv.sh
| venv="$1" shift . "$venv/bin/activate"
| "$@"
|
| Then in the Makefile I do: VENV := venv.sh
| ./venv install-reqs: ${VENV} pip -r
| requirements.txt
| bragr wrote:
| That works but the whole point of the article was to avoid
| creating extra scripts.
| jedberg wrote:
| Makefiles are great for local task automation. In college I had a
| friend who started every homework, even the social science and
| English homework, with a makefile to compile his Latex files.
| belthesar wrote:
| I've been seeing a growing use of `just` for cases like this. It
| works similar in function to make, but it's a great deal cleaner
| and easier to use.
| kapilvt wrote:
| as someone who maintains projects both using just and make, I
| have to say just is awesome, and allows things like writing a
| given task directly in a scripting language. sadly its not
| universally available across operating systems and linux
| distros (ubuntu/debian was the last significant dev ecosystem
| where it wasn't native, re apt install). so for projects with
| wider contributor goals, I still default to make, despite its
| warts (which are many imo), cause it tends to work ootb.
| eirki wrote:
| Url: https://github.com/casey/just
| pletnes wrote:
| It also backtracks up the directory tree so you don't have to
| cd to run <<just test>>.
| loloquwowndueo wrote:
| These "run something" targets should be marked as .PHONY so make
| realizes they will not produce a file. Otherwise a file named
| like the target will confuse make and turn your day into a sad
| one.
| pydry wrote:
| I ran into this back when I used make as a command runner.
|
| After a short detour via just Ive been using a shell script
| with a big case statement since.
| kevincox wrote:
| Yup. I've found that in most cases a directory of script files
| ends up better. Or even better a directory of python files
| since shell kinda sucks.
| schemescape wrote:
| Unrelated to Python, but is anyone aware of a tool like Make that
| can handle file path wildcards in prerequisites?
|
| For example, let's say I have a rule that consumes *.json to
| produce some target. I want it to rebuild that target if any
| *.json files are added or removed (or modified).
|
| As far as I'm aware, Make relies on file system timestamps, so
| even if it supported wildcards (I'm not sure if it does or not),
| it wouldn't be able to notice that foo.json has been _deleted_
| and a rebuild of the target is needed.
|
| I thought I'd ask here before I go build such a tool (to support
| incremental rebuilds of a static site).
|
| Edit to add that my particular use case has a non-prescriptive
| set of directories, nested a few levels deep. I'm actually
| realizing that that is probably a bigger hurdle for using
| something like Make (e.g. transform all *.md files, no matter how
| deep; but copy everything else, like images, verbatim; oh, and
| also then aggregate all *.md files into an Atom feed). Yes, I
| know this is asking a lot of something like Make!
| bigmanwalter wrote:
| Did you try searching stack overflow?
| https://stackoverflow.com/a/43402649
|
| If you dedicate a specific directory for all your JSON, then
| you can depend on that directory and it will do what you're
| asking for.
| schemescape wrote:
| I did spend quite a bit of time researching approaches, but I
| didn't come across this particular idea. I'll give it a try.
| Thanks!
|
| Note: in my case, I have a directory structure that's a few
| levels deep, with an non-prescriptive set of directories (one
| subdirectory per category, with no limit on the set of
| categories). Maybe Make handles directories better than I
| realized (I'd always seen it recommended to use a Makefile in
| each directory--something I want to avoid).
| mturmon wrote:
| Here's one way to do this.
|
| I have used this method (directory mod-time triggering,
| let's say) for a simulation-summarizer which analyzes
| whatever pile of simulation-output-files happen to be in a
| given directory. If you run a new simulation, the directory
| changes and the analysis is re-done by running "make".
|
| I used the Gnu make $(wildcard ...) for the template
| expansion, instead of using shell expansion. This is to
| take care of the no-file case, so that jsons/*.json will
| expand to nothing rather than to the literal jsons/*.json
| (which does not exist). $ cat Makefile
| file.out: jsons cat $(wildcard jsons/*.json)
| /dev/null > file.out $ ls -R
| Makefile jsons/ ./jsons: foo.json
| $ make cat jsons/foo.json /dev/null > file.out
| $ make # no file mods => no-op make: `file.out' is
| up to date. $ touch jsons/bar.json
| $ make # new file => re-make cat jsons/bar.json
| jsons/foo.json /dev/null > file.out $ make
| make: `file.out' is up to date. $ rm
| jsons/foo.json $ make # deletion => re-make
| cat jsons/bar.json /dev/null > file.out $ rm
| jsons/bar.json $ make # nothing there
| cat /dev/null > file.out $ make make:
| `file.out' is up to date.
| schemescape wrote:
| Thanks to the folks who replied! Looks like I didn't dig far
| enough into GNU Make. It's got a lot of the functionality I'm
| looking for (and I just didn't realize it). Hopefully I don't
| need to use it, but it even has an "eval" function for
| dynamically generating its own input.
| dwheeler wrote:
| GNU make at least can do that.
| schemescape wrote:
| Interesting! I had originally tried to do this with GNU Make
| (with the most naive approach you can imagine) and it wasn't
| able to notice that a prereq that matched the wildcard had
| been deleted (i.e. it thought the target was up to date, when
| I wanted it to rebuild).
|
| I'll read up on the wildcard function and see if that is what
| I was looking for:
|
| https://www.gnu.org/software/make/manual/html_node/Wildcard-.
| ..
|
| Edit: A sibling comment also pointed out putting the
| wildcard'd files into a directory to help Make notice
| deletions. I'll give that a try first.
| BiteCode_dev wrote:
| Don't. Use pydoit.org.
| digdugdirk wrote:
| As someone not up to date on the Python world, how does your
| suggestion compare with something like Poetry?
| BiteCode_dev wrote:
| poetry is specialized in managing venv and packages, with the
| options to run some commands.
|
| doit is a task runner: you declare a name that groups several
| commands, and when you call the task with the name, it runs
| all the commands in order. It can accept bash commands,
| python functions, declare dependencies on other commands or
| on files, and cascade the tasks to follow those dependencies
| automatically.
|
| It's make, but in Python: nice syntax, works on
| windows/mac/linux, easy things are easy, hard things are
| possible.
| thedumbname wrote:
| Use Makefile if you want to make your cross-platform build
| environment Linux only. You will have all kind of issues on
| Windows, Mac, *BSD, you name it.
| jjgreen wrote:
| Make is in POSIX, so stick to POSIX features and your Makefile
| is portable across all reasonable systems. For Windows, I guess
| you use a .bat file.
| quesera wrote:
| You can resolve the differences between BSD and GNU make.
|
| Not sure about Windows. I'd expect that WSL bundles GNU make,
| no?
| choeger wrote:
| If your make targets don't have dependencies you are probably
| using the wrong tool for the job.
| skratlo wrote:
| It's not true that Python doesn't have anything akin to
| package.json scripts. Bonus points: all is executed within
| project's virtualenv.
|
| https://python-poetry.org/docs/cli/#run
| paiute wrote:
| While poetry is the way to go today, I've got my eyes on pdm.
| https://github.com/pdm-project/pdm Pyproject.toml is a pep
| standard so it will be easy to move around tools.
| w0m wrote:
| i just wish poetry's dependency generation was faster. It's
| painful* today.
| akvadrako wrote:
| pdm is already the better choice than poetry IMHO.
| MuffinFlavored wrote:
| is this separate to pip's "requirements.txt" usually?
| senand wrote:
| Yes, it's assuming poetry is used, whereby normally the
| packages needed by the project are defined in pyproject.toml
| kortex wrote:
| Very different. Requirements.txt only handles dependency
| packages, and only handles on kind of dependency (no
| dev/release distinction). pyproject.toml does dev/release
| deps, it has package metadata, it can configure all your
| lint/test tooling, and it is extensible with plugins.
| MuffinFlavored wrote:
| and they can't be combined, right? like package.json for
| example
| qprofyeh wrote:
| How does this compare to taskipy?
| devrc wrote:
| I have been using Makefile for over 10 years in all of my
| projects, and here are some features I've always found lacking in
| Makefile:
|
| 1. There is no way to display documentation for commands and
| accepted parameters. Yes, you can write a special task that will
| display comments, but you have to copy it from project to
| project.
|
| 2. The need to pass named parameters when calling tasks. I want
| to write `make serve localhost 3000` instead of `make serve
| bind=localhost port=3000`
|
| 3. I've always had the need in different projects to use the same
| commands, so I had to copy tasks from project to project. I need
| a central place with commands that I can apply to any project.
|
| 4. The ability to write tasks in different languages. In some
| cases, it's easier to write in Python or TypeScript/Deno.
|
| 5. And most importantly, it is difficult to write commands in
| Makefile that can be used in different environments. For example,
| I need to run commands on different groups of servers: production
| and staging. This could look like: `make production TASK1 TASK2`
| or `make stage TASK1 TASK2`. In other words, the production/stage
| task sets up the parameters for executing tasks in a specific
| environment. It might be possible to call commands in this way
| with Makefile, but it seems too complicated.
|
| As a result, I decided to write my own utility for task
| automation: https://github.com/devrc-hub/devrc
|
| It solves all of the above problems and has other interesting
| features and also written in Rust .
| aldanor wrote:
| I think Just (as in Justfile) also solves most of your points,
| is reasonably widely used, also written in Rust and has
| integration plugins for it in most editors.
| Messier43 wrote:
| Pipenv has script hooks as well.
|
| https://pipenv.pypa.io/en/latest/scripts/
|
| It also includes many other package.json features.
| hackandtrip wrote:
| Another possibility is using the `scripts` in the
| `pyproject.toml`, as described here: https://python-
| poetry.org/docs/pyproject/#scripts
| leblancfg wrote:
| That's specifically for the Poetry package manager, not Python
| in general.
| nomel wrote:
| Neither is make. You're using an external tool either way.
| The one purpose built for the task, and doesn't have a bunch
| of archaic footguns [1], will probably give a better
| experience.
|
| 1. https://stackoverflow.com/questions/17965806/how-do-i-
| handle....
| paiute wrote:
| pyproject.toml is for python in general.
| https://peps.python.org/pep-0621/ setup.py is legacy.
| hprotagonist wrote:
| that's generally not correct. entry points are how your
| package, once installed, is called from the command line; i
| wouldn't wire up a docker image build step or twine publish
| that way.
| zer0w1re wrote:
| Those two examples are the exact use-case that defining
| scripts in pyproject.toml are meant for. Users of my
| installed package would never need to run `twine publish` or
| build the project's docker image. That's only really needed
| by developers who would be working from the full project
| source including pyproject.toml.
| hprotagonist wrote:
| entry points are installed to the interpreter site when you
| `pip install`: everyone gets them including your users.
| kayson wrote:
| The main reason I still use Make over other alternatives is that
| everyone already has it. No need to install something else to
| bootstrap a project. I put in a `make initialize` that takes care
| of everything: setting up the virtualenv, using our internal pip
| mirror, downloading dependencies, etc. It's just such low
| friction!
|
| Side note: Make is also really popular among self-hosters for
| ansible infrastructure setup.
| arbol wrote:
| Makefile quickly gets complicated when you have to pass in flags
| (--an-example-flag).
|
| e.g.
|
| dev: ./scripts/run-dev.sh $(if $(filter build,$(MAKECMDGOALS)),--
| build,) $(if $(filter restart,$(MAKECMDGOALS)),--restart,) ...etc
|
| Since the project is in python it would be better to write a
| python script and start the tasks as subprocesses. This means you
| can use the python argparser you're already familiar with.
|
| https://docs.python.org/3/library/subprocess.html
| nailer wrote:
| Or in short: using 3 languages (Make, shell interspersed
| between the Make, Python) is more complicated than using one
| language.
| arbol wrote:
| The `.sh` was not implied by the article. That was my
| addition but, yes, the less languages the better!
| delaaxe wrote:
| Invoke is like a Makefile but in Python:
| https://www.pyinvoke.org/index.html
| nathell wrote:
| Make wasn't designed to be a task runner. It can be used as one,
| but it doesn't make (pun not intended) a very good one, and it
| doesn't have the best syntax, either.
|
| Make is an artifact updater. Although it's not its main focus,
| Peter Miller's classical paper 'Recursive Make Considered
| Harmful' does a good job of explaining what it is.
|
| One great benefit of make is that it's present everywhere, so
| there's no additional hassle of installing an extra tool.
| Depending on the project, this hassle-freeness may or may not
| outweigh make's relative incomfort as a task runner.
| taeric wrote:
| What things were designed for is often a side show. We want to
| think that intent of creation matters, for some reason, when
| reality is full of that never really being the case.
|
| So, on the merits, what makes it a bad task runner between
| outputs? I agree it is somewhat obtuse, but I'm still mostly
| crying from trying to get nox and a few other things working.
| Githubs workflow syntax is as painful, all told. (Though, it
| has the very real constraint that it is "per repository" and
| you need something above it.)
| FreakLegion wrote:
| _> ...there is no more important proposition for every sort
| of history than that which we arrive at only with great
| effort but which we really should reach, -- namely that the
| origin of the emergence of a thing and its ultimate
| usefulness, its practical application and incorporation into
| a system of ends, are toto coelo separate; that anything in
| existence, having somehow come about, is continually
| interpreted anew, requisitioned anew, transformed and
| redirected to a new purpose..._
___________________________________________________________________
(page generated 2023-03-27 23:02 UTC)