* * * * *
                                        
                             Some thoughts on make
                                        
From time to time, I find make [1] limiting, think there has to be a better
way and I start playing around with the idea of building a new make. It's not
that uncommon for programmers to think this (SCons [2], CMake [3], ant [4],
rake [5] and a bunch of other programs [6]) but oddly enough, as I try them,
I keep going back to make (specifically, GNU make [7]; note that when I
mention make, I am talking about GNU make).

I think the main reason I go back is that make is mainly a syntax-light
declarative langauge with bits of shell scripting thrown in. The other build
systems are generally based around another language I do not know, they tend
towards being very imperative, and rarely can you parallelize building the
software (and when you get the makefile right, a parallel run of make just
flies).

So when I had that “let's remake make” itch this time, I thought that perhaps
I would see if I could stay within the confines of the existing syntax of
make as much as possible. I wasn't trying to actually program my own make,
but rather, just play around with how I would like make to look and work.

The main problem with make is declaring the dependencies. For instance, I'm
embedding a UUID (Universally Unique Identifier) [8] library into a larger
project, and because recursive make is considered harmful [9], I include in
my makefile:

> lib/libspcuuid.a : build/third_party/uuid/luauuid.o	\
> 		build/third_party/uuid/uuid_ns_dns.o	\
> 		build/third_party/uuid/uuid_ns_null.o	\
> 		build/third_party/uuid/uuid_ns_oid.o	\
> 		build/third_party/uuid/uuid_ns_url.o	\
> 		build/third_party/uuid/uuid_ns_x500.o	\
> 		build/third_party/uuid/uuidlib_cmp.o	\
> 		build/third_party/uuid/uuidlib_parse.o	\
> 		build/third_party/uuid/uuidlib_toa.o	\
> 		build/third_party/uuid/uuidlib_v1.o	\
> 		build/third_party/uuid/uuidlib_v2.o	\
> 		build/third_party/uuid/uuidlib_v3.o	\
> 		build/third_party/uuid/uuidlib_v4.o	\
> 		build/third_party/uuid/uuidlib_v5.o
> 
> build/third_party/uuid/luauuid.o	: third_party/uuid/src/luauuid.c
> build/third_party/uuid/uuid_ns_dns.o	: third_party/uuid/src/uuid_ns_dns.c
> build/third_party/uuid/uuid_ns_null.o	: third_party/uuid/src/uuid_ns_null.c
> build/third_party/uuid/uuid_ns_oid.o	: third_party/uuid/src/uuid_ns_oid.c
> build/third_party/uuid/uuid_ns_url.o	: third_party/uuid/src/uuid_ns_url.c
> build/third_party/uuid/uuid_ns_x500.o	: third_party/uuid/src/uuid_ns_x500.c
> build/third_party/uuid/uuidlib_cmp.o	: third_party/uuid/src/uuidlib_cmp.c
> build/third_party/uuid/uuidlib_parse.o	: third_party/uuid/src/uuidlib_parse.c
> build/third_party/uuid/uuidlib_toa.o	: third_party/uuid/src/uuidlib_toa.c
> build/third_party/uuid/uuidlib_v1.o	: third_party/uuid/src/uuidlib_v1.c
> build/third_party/uuid/uuidlib_v2.o	: third_party/uuid/src/uuidlib_v2.c
> build/third_party/uuid/uuidlib_v3.o	: third_party/uuid/src/uuidlib_v3.c
> build/third_party/uuid/uuidlib_v4.o	: third_party/uuid/src/uuidlib_v4.c
> build/third_party/uuid/uuidlib_v5.o	: third_party/uuid/src/uuidlib_v5.c
> 

Or, if I wanted to save myself some typing:

> define OBJECT_template = 
>   $(1) : $(2)
> endef
> 
> UUIDSRC = $(wildcard third_party/uuid/src/*.c)
> 
> lib/libspcuuid.a : $(subst third_party/uuid/src,build/third_party/uuid,$(UUICSRC))
> 
> $(foreach target,$(UUIDSRC),$(eval $(call OBJECT_template(build/third_party/uuid/$(notdir $(target)),$(target)))))
> 

and have no idea what is going on three months from now (if indeed, I got
that right). Instead, I would like to type:

> lib/libspcuuid.a : build/third_party/uuid/*.o
> build/third_party/uuid/*.o : third_party/uuid/src/*.c
> 

and be done. This should be possible. “lib/libspcuuid.a” doesn't exist, but
it depends upon all the “.o” files in “build/third_party/uuid”, which don't
exist, but we have a rule that says “for ‘.o” files in
‘build/third_party/uuid’, there should be a corresponding ‘.c” file in
‘third_party/uuid/src/’ that is is generated from.” Which also leads to
another problem with make: if “build/third_party/uuid” doesn't exist, make
should make it! Automatically! I mean, make has no problems with
“uuidlib_v5.o” not existing—that's the reason we're using make in the first
place, to make files! If a directory a file lives in doesn't exist, make
should make that too. Right?

By the same token, if I wanted to include some Lua modules [10] but not all
of them, I would like to do:

> SPCMODC = env errno fsys hash math net pollset process strcore sys syslog
> SPCMODL = debug getopt string table unix
> 
> lib/libspcmod.a : build/third_party/lua-conmanorg/*.o
> 
> build/third_party/lua-conmanorg/*.o : third_party/lua-conmanorg/src/{ $(SPCMODC) }.c \
> 	third_party/lua-conmanorg/lua/{ $(SPCMODL) }.lua
> 

(Ah! So that's where yesterday's code snippet [11] came from!)

I'll spare you the expansion.

Another small bit of annoyance is writing implicit rules to build the files.
I learned the hard way that in make, this:

> %.o : %.c
> 	$(CC) $(CFLAGS) -c -o $@ $<
> 

only works if the “.c” and “.o” files are in the same directory! So the above
rules works fine for:

> long/path/to/src/foo.o : long/path/to/src/foo.c
> src/bar.o              : src/bar.c
> snafu.o                : snafu.c
> 

But not for:

> build/foo.o    : long/path/to/src/foo.c
> build/bar.o    : src/bar.c
> build/snafnu.o : snafu.c
> 

Nope, I also have to write:

> build/%.o : long/path/to/src/*.c
> 	$(CC) $(CFLAGS) -c -o $@ $<
> 
> build/%.o : src/%.c
> 	$(CC) $(CFLAGS) -c -o $@ $<
> 
> build/%.o : %.c
> 	$(CC) $(CFLAGS) -c -o $@ $<
> 

if I want to segregate the “.o” files from the “.c” files.

How hard is it to realize that if I have a “.c” file, then the command to
make a “.o” file is the same, regardless of where the “.c” file comes from,
or where the “.o” file is being generated. Hello?

And while we're on that subject, one implicit rule I use is the following:

> %.o : %.lua
> 	$(LUAC) -o $(@D)/$(*F).out $<
> 	$(BIN2C) -9 -o $(@D)/$(*F).c -t $(*F) $(@D)/$(*F).out
> 	$(CC) $(CFLAGS) -c -o $@ $(@D)/$(*F).c
> 	$(RM) $(@D)/$(*F).out $(@D)/$(*F).c
> 

That bit of jibberish first compiles the Lua code using luac, then that
output is converted to a C file which is then compiled into a object file so
I can link it into the final executable [12], them removes the temporary
files it needed to do all that. The noise you see is the cruft necessary to
specify temporary files that won't get overwritten when doing a parallel
build. Much nicer would be something like:

> %.o : %.lua
> 	$(LUAC) -o $$.1 $<
> 	$(BIN2C) -9 -o $$.2 -t $(*F) $$.1
> 	$(CC) $(CFLAGS) -c -o $@ $$.2
> 	$(RM) $$.1 $$.2
> 

(even better—make automatically deletes the intermediate files unless I tell
it not to)

One complaint about make (or rather, makefiles themselves) is that a change
to the makefile does not cause a recompile, mainly because the makefile
itself is never listed as a dependency anywhere. This is never what you want!
If the makefile was listed as a depenency (either explicitly or implicitly),
then adding a new file to be compiled in would cause everything to be
recompiled, when it should be the new file, any files that call code in the
new file, and a final relinking of the code is all that is needed. Also, if
you change a variable, like $(LDFLAGS), then all that should happen is any
rule that uses $(LDFLAGS) should be run, not a complete build.

In other words, you really have a dependency on a portion of the makefile,
not the makefile as a whole. But solving this would require a possible
rethink of how make works (perhaps a cached version of the makefile that mode
checks against, and intelligently applies the changes as dependencies? You
know, so if you change $(CC), any rule using $(CC) is automatically run).

I realize these some of my ideas are not that easy to implement (heck, I
wasted a few hours yesterday writing code to properly build a list of targets
and dependencies based on ideas presented here), but I think they may go a
long way to making make less horrendous to use.

[1] http://en.wikipedia.org/wiki/Make_(software)
[2] http://www.scons.org/
[3] http://www.cmake.org/
[4] http://ant.apache.org/
[5] http://rake.rubyforge.org/
[6] http://en.wikipedia.org/wiki/List_of_build_automation_software
[7] https://www.gnu.org/software/make/
[8] https://github.com/spc476/SPCUUID
[9] http://aegis.sourceforge.net/auug97.pdf
[10] https://github.com/spc476/lua-conmanorg
[11] gopher://gopher.conman.org/0Phlog:2015/04/03.1
[12] gopher://gopher.conman.org/0Phlog:2013/03/23.1

Email Sean Conner at sean@conman.org

.