https://monkey.org/~marius/unix-tools-hints.html
Hints for writing Unix tools
Marius Eriksen (marius@monkey.org)
20 Oct 2014
Note: this article has been translated into Japanese
The workaday world of a modern programmer abounds with Unix tools,
stitched together in myriad ways. While good tools integrate
seamlessly with your own environment, bad ones will constantly
frustrate your efforts. Good tools have a seemingly limitless
application, constrained only by your own imagination. Bad tools, on
the other hand, will often require that you deploy a salvo of brittle
hacks to keep them barely working in your own environment.
"One thing well" misses the point: it should be "One thing well
AND COMPOSES WELL"
-- marius eriksen (@marius) October 10, 2012
I don't want to attempt to explain what makes for good design; this
has been discussed elsewhere. Instead, I want to outline a few
established customs that you should take care to follow when writing
new tools. While making a truly good tool can be an elusive goal, it
isn't difficult to avoid making a truly bad one. Unix demands good
citizenry from its tools: it relies on a set of conventions to make
things work, and importantly, to compose, well. Here follows a few
key customs, often violated. These aren't absolute requirements, but
you should think long and hard before violating them.
Consume input from stdin, produce output to stdout. Put another way,
your program should be a filter. Filters are easily integrated into
shell pipelines, arguably the most important utility for Unix tools
composition.
Output should be free from headers or other decoration. Superflous
output will frustrate users who are trying to parse tool output.
Headers and decoration tend to be less regular and more idiosyncratic
than the structured data you're really trying to get at. Don't do it.
Output should be simple to parse and compose. This usually means
representing each record as a single, plain-text formatted line of
output whose columns are separated by whitespace. (No JSON, please.)
Most venerable Unix tools--grep, sort, and sed among them--assume this.
As a simple example, consider the following output from a benchmark
suite. It is formatted by starting each record with the benchmark
name, followed by a set of key-value pairs associated with the named
benchmark. This is a flexible structure to work with as it allows you
to add or remove keys at will without violating the output format.
$ ./runbenchmarks
Benchmark: fizzbuzz
Time: 10 ns/op
Alloc: 32 bytes/op
Benchmark: fibonnacci
Time: 13 ns/op
Alloc: 40 bytes/op
...
$
While convenient, it is quite clumsy to work with in Unix. Consider a
very common thing we might want to do: look up the timing results for
a single benchmark. Here's how you do it.
$ ./runbenchmarks | awk '/^Benchmark:/ { bench = $2} bench=="fizzbuzz"'
Benchmark: fizzbuzz
Time: 10 ns/op
lloc: 32 bytes/op
$
If instead each line presents exactly one record, where columns are
separated by whitespace, this becomes a much simpler task.
$ ./runbenchmarks
fizzbuzz 10 32
fibonnaci 13 40
...
$ ./runbenchmarks | grep '^fizzbuzz'
fizzbuzz 10 32
$
The advantage becomes even more evident when reordering or
aggregating the input. For example, when the output is
record-per-line, sorting the results by time spent is a simple matter
of invoking sort:
$ ./runbenchmarks | sort -n -r -k2,2
fibonnaci 13 40
fizzbuzz 10 32
...
$
Treat a tool's output as an API. Your tool will be used in contexts
beyond your own imagination. If a tool's output format is changed,
other tools that compose or otherwise build on its output will
invariably break--you have broken the API contract.
Place diagnostics output on stderr. Diagnostics output includes
anything that is not the primary data output of your tool. Among
these are: progress indicators, debugging output, log messages, error
messages, and usage information. When diagnostics output is
intermingled with data, it is very difficult to parse, and thus
compose, the tool's output. What's more, stderr makes diagnostics
output more useful since, even if stdout is being filtered or
redirected, stderr keeps printing to the user's terminal--the ultimate
target of diagnostics output.
Signal failure with an exit status. If your tool fails, exit with a
status other than 0. This allows for simple integration shells, and
also simpler error handling in scripts. Consider the difference
between two tools that build binaries. We'd like to build upon this
tool to execute the built binary only if the build succeeds. Badbuild
prints the word 'FAILED' as the last line when it fails.
$ ./badbuild binary
...
FAILED
$ echo $?
0
$ # Run binary on successful build.
$ test "$(./badbuild binary | tail -1)" != "FAILED" && ./binary
$
Goodbuild sets its exit status appropriately.
$ ./goodbuild
$ echo $?
1
$ # Run binary on successful build.
$ ./goodbuild binary && ./binary
$
Make a tool's output portable. Put another way, a tool's output
should stand on its own, requiring as little context as possible to
parse and interpret. For example, you should use absolute paths to
represent files, and fully qualified hostnames to name internet
hosts. Portable output is directly usable by other tools without
further context. A frequent violator of this is build tools. For
example, both the GCC and Clang compilers try to be clever by
reporting paths that are relative to your working directory. In this
example, the source file paths are presented relative to the current
working directory when the compiler was invoked.
$ cc tmp/bad/x.c
tmp/bad/x.c:1:1: error: unknown type name 'INVALID_C'
INVALID_C
^
tmp/bad/x.c:1:10: error: expected identifier or '('
INVALID_C
^
2 errors generated.
$
This cleverness breaks down quickly. For example if I use make(1)
with the -C flag.
$ cat tmp/bad/Makefile
all:
cc x.c
$ make -C tmp/bad
cc x.c
x.c:1:1: error: unknown type name 'INVALID_C'
INVALID_C
^
x.c:1:10: error: expected identifier or '('
INVALID_C
^
2 errors generated.
make: *** [all] Error 1
$
Now the output is less useful: to which file does "x.c" refer? Other
tools that build on this need additional context, the -C argument, in
order to interpret the compiler's output--the output does not stand on
its own.
Omit needless diagnostics. Resist the temptation to inform the user
of everything that is being done. (But if you must, do it on stderr.)
A good tool is quiet when all is well, but produces useful
diagnostics output when things go wrong. Excessive diagnostics
conditons users to ignore all diagnostics; useful diagnostics output
does not require the user to grub around in endless log files to
discern what went wrong, and where. There's nothing wrong with having
a verbose mode (typically enabled by a '-v' flag) in order to aid
development and debugging, but do not make this the default.
Avoid making interactive programs. Tools should be usable without
user interaction beyond what's provided by the user's shell. Unix
programs are expected to run without user input: it allows programs
to be run in non-interactively by cron, or to be easily distributed
for execution by a remote machine. Even a single interaction forfeits
this very useful capability. Interactivity also makes composition
more difficult. Since Unix's program composition model does not
distinguish the output of the various programs involved, it isn't
always clear which program a user is even interacting with. A common
use of interactive programs is to ask the user to confirm some
dangerous action. This is easily avoided by asking the user instead
to supply a flag on the command line to the appropriate tool.
I wrote this because I find myself continually frustrated by
attempting to use and compose bad tools--bad tools that waste time and
limit their own usefulness. Most of these tools could be made a lot
better by following the above advice.
For a more general discussion of Unix tools design, I encourage you
to read Kernighan and Pike's "The Unix Programming Environment ."
Discussion on Hacker News.
Tweet
(c) 2009-2015 Marius Eriksen [rss]