https://antonz.org/makefile-automation/

[logo]   Anton Zhiyanov
blog contacts ru

Automate your Python project with Makefile

When working on a library or application, certain tasks tend to show
up over and over again:

  * checking the code with linters,
  * running tests with coverage,
  * deploying with Docker,
  * …

JS developers are lucky (ha!): their package.json has a special
scripts section for this stuff:

{
    ...
    "scripts": {
        "format": "prettier --write \"src/**/*.ts\"",
        "lint": "tslint -p tsconfig.json",
        "test": "jest --coverage --config jestconfig.json",
    },
    ...
}

Nothing like this is provided with Python. You can, of course, make a
.sh script for each task. But it litters the project directory, and
it's better to keep all such tasks together. Installing a separate
task runner or using the one built into IDE also seems weird.

Good news: Linux and macOS already have a great task automation tool
for any project - Makefile.

Makefile for task automation

Perhaps you, like me, thought that Makefile is a relict from the 70s,
useful for compiling C programs. True. But it is also perfectly
suitable for automating any tasks in general.

Here's what it might look like in a python project. Create a file
named Makefile:

coverage:  ## Run tests with coverage
        coverage erase
        coverage run --include=podsearch/* -m pytest -ra
        coverage report -m

deps:  ## Install dependencies
        pip install black coverage flake8 mypy pylint pytest tox

lint:  ## Lint and static-check
        flake8 podsearch
        pylint podsearch
        mypy podsearch

push:  ## Push code with tags
        git push && git push --tags

test:  ## Run tests
        pytest -ra

And run linter with tests, for example:

$ make lint coverage

flake8 podsearch
pylint podsearch
...
mypy podsearch
...
coverage erase
coverage run --include=podsearch/* -m pytest -ra
...
coverage report -m
Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
podsearch/__init__.py       2      0   100%
podsearch/http.py          17      0   100%
podsearch/searcher.py      51      0   100%
-----------------------------------------------------
TOTAL                      70      0   100%

Features

Task steps

A task can include multiple steps, like lint in the example above:

lint:
        flake8 podsearch
        pylint podsearch
        mypy podsearch

Each step is executed in a separate subprocess. To run a chain of
actions (for example, cd and git pull) combine them through &&:

push:
        git push && git push --tags

Task dependencies

Consider the test task, which should first perform linting, and then
run the tests. Specify lint as a dependency for test, and you're
done:

test: lint
        pytest -ra

You can specify multiple space-separated dependencies. Or tasks can
explicitly call each other:

lint:
        flake8 podsearch
        pylint podsearch
        mypy podsearch

test:
        pytest -ra

prepare:
        make lint
        make test

Task parameters

Consider the serve task which serves a static site, with IP and port
specified as parameters. No problem:

serve:
        python -m http.server dist --bind $(bind) $(port)

Run task with parameters:

$ make serve bind=localhost port=3000

You can specify default parameter values:

bind ?= localhost
port ?= 3000
serve:
        python -m http.server dist --bind $(bind) $(port)

Now they are optional when running make:

$ make serve bind=192.168.0.1
$ make serve port=8000
$ make serve

And so much more

If basic features are not enough, there are some great in-depth
guides:

  * Learn Makefiles with the tastiest examples
  * Automation and Make

In the wild

Here is a Makefile from one of my projects (podcast search tool):

  * podsearch

Makefiles are great for automating routine tasks, regardless of the
language you prefer. Use them!

Follow @ohmypy on Twitter to keep up with new posts 

16 Mar, 2021
python 

- SQLite is not so easy to compile

How to create a 1M record table with a single query -

  * Subscribe: Twitter RSS
  * Email: m@antonz.org