16.3. Listing And Running Tests

Tests are run with pytest. Pytest features versus the "unittest" standard library module include finer test selection, full parallelism, nicer test execution summaries, better output in case of failures (containing the contents of variables) and the ability to open the PDB debugger on failing tests.

To get started, install pytest 3.0 or newer and pytest-xdist:

# Install required packages on Ubuntu 18.04 or Debian jessie-backports
$ sudo apt install python3-pytest python3-pytest-xdist

# Install required packages on other systems
$ pip install pytest pytest-xdist

Run pytest in the Wireshark build directory, Wireshark binaries are assumed to be present in the run subdirectory (or run\RelWithDebInfo on Windows).

# Run all tests
$ cd /path/to/wireshark/build
$ pytest

# Run all except capture tests
$ pytest --disable-capture

# Run all tests with "decryption" in its name
$ pytest -k decryption

# Run all tests with an explicit path to the Wireshark executables
$ pytest --program-path /path/to/wireshark/build/run

To list tests without actually executing them, use the --collect-only option:

# List all tests
$ pytest --collect-only

# List only tests containing both "dfilter" and "tvb"
$ pytest --collect-only -k "dfilter and tvb"

The test suite will fail tests when programs are missing. When only a subset of programs are built or when some programs are disabled, then the test suite can be instructed to skip instead of fail tests:

# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF)
$ pytest --skip-missing-programs dumpcap,rawshark

# Run tests and ignore all tests with missing program dependencies
$ pytest --skip-missing-programs all

The coverage tests in suite_dissection.py, the classes whose captures all have "_coverage" in their file name, can be limited to those that a change can affect. With --changed-since pytest skips the coverage classes whose dissectors are not among the files changed since the given commit, counting uncommitted changes too:

# Run only the coverage tests affected by the changes on this branch
$ pytest --changed-since origin/master

Every other test still runs, and all of them run when a file outside the dissectors changed. test/dissection_map.json lists the dissector files each coverage class exercises. After adding or changing coverage captures or their tests, regenerate it with tools/gen-dissection-map.py --program-path /path/to/wireshark/build/run; a class missing from the map always runs.

To open a Python debugger (PDB) on failing tests, use the --pdb option and disable parallelism with the -n0 option:

# Run decryption tests sequentially and open a debugger on failing tests
$ pytest -n0 --pdb -k decryption