https://www.humprog.org/~stephen/blog/2024/08/27/#how-to-wrap-cc-really

Rambles around computer science

Diverting trains of thought, wasting precious time

Tue, 27 Aug 2024

How to really wrap a C compiler and preprocessor, really*

* really

Suppose we want to interfere with how a vaguely Unix-style C compiler
does its job, and that we want to try compiling existing software
with this modified compiler. Assuming the build system will let us do
something like:

CC=/path/to/my/wrapper make

or

CC=/path/to/my/wrapper ./configure && make

... then we'd like something that is a drop-in replacement at the
command-line level. Let's also assume that our modifications are
black-box: we want to do "what the stock compiler was doing", modulo
some tweaks at the edges. The tweaks might be running extra tools,
munging the compiler command-line options, doing source-to-source
rewrites on some input files, postprocessing output files, and so on.
Although I say "tweaks", the extent of our interference could be
arbitrary, once we get into rewriting the input or output programs.
But we are still making at least some use of the original compiler.

This requirement is common if you do research with compilers. The
time-honoured way is to write a Python script or Perl script or shell
script that parses a command line, delegates to the real compiler
while also doing the desired extra stuff. Then you build the existing
software using this wrapper script, e.g. by setting CC to point this
wrapper script.

There are some problems with this.

To do any non-trivial interference you probably want to divert the
input and/or output of some stage(s) of compilation. So you have to
be able to identify the input and output files on the command line,
from an understanding of its syntax and semantics. Finding the output
filename is nicely syntactic if -o is given, but gets a bit fiddly
and semantic in other cases: it's a.out if we're linking, otherwise
it's a substitution on the input filename. Input files are hard to
detect too: there might be many of them (hence many output files too)
and we need to know that if we see a pathological argument sequence
like -I foo.c, foo.c in fact names a directory. As well as "obvious"
input and outputs, there may be other outputs generated by options
like -M, -MF and -MD. Sometimes these are in addition to the named
output file, but sometimes they are instead of it and the named
output file will be ignored or left empty.

We know that a single "compiler" command is really a "driver"
command. The driver may run a selection of tools or "subcommands":
the preprocessor, the compiler proper, the assembler and/or the
linker. The selection of these depends on the driver command. To
interfere with specific parts of compilation, our script needs to be
able to deconstruct a driver invocation into the many subcommands. It
will then mess with the relevant ones of these, but leave the others
undisturbed.

Doing the deconstruction into subcommands is non-trivial. It's what
the driver exists for, after all. To duplicate this ourselves,
there's a lot we have to get right. Which arguments to the driver
need to go with which of the subcommands? I have my own Python script
that can usually sort these out, and I'm sure dozens of others are
out there. It works most of the time... but some commands will defeat
it. It also adds Python to the build dependencies of anything we want
to build; not a big deal, but this caused me some annoyance during
the infamous Python-2-to-3 transition. cilly is another such script,
written using Perl and even backing onto both GCC and MSVC. As with
many such scripts, it has annoyingly similar but not-quite-the-same
options to both GCC's and MSVC's. For example, it has --save-temps
whereas GCC has -save-temps, so it is not quite a drop-in
replacement.

These scripts are taking what I call the "new front door" approach.
They give the compiler driver a whole new entry point. This is
doable, but it quickly turns into reimplementing the driver (and
doing so badly, in practice).

I want instead a wrapper script that is simpler, more robust and more
generic. I want to get away with as shallow as possible an awareness
of the command line syntax and semantics. In particular, I don't want
to reimplement the driver's deconstruction to subcommands. I also
want my script to be generic, in the sense that I could
straightforwardly re-use the same wrapper script for many purposes.
This contrasts with, say, cilly that is specialised to running CIL
and tools based on it.

I've been through a few iterations with this now. What I've ended up
with is a "back-room" approach that contrasts with the "front door"
approach. I've made life simpler by only caring about GCC and Clang,
although it could probably work with any compiler that follows
vaguely the same historical command-line conventions (which I suspect
date back at least to pcc, although I haven't verified that). GCC has
some options that appear handy for a "back-room" approach.

       -wrapper
           Invoke all subcommands under a wrapper program.  The name of the wrapper program and
           its parameters are passed as a comma separated list.

                   gcc -c t.c -wrapper gdb,--args

           This invokes all subprograms of gcc under gdb --args, thus the invocation of cc1 is
           gdb --args cc1 ....
...
       -v  Print (on standard error output) the commands executed to run the stages of
           compilation.  Also print the version number of the compiler driver program and of the
           preprocessor and the compiler proper.

       -###
           Like -v except the commands are not executed and arguments are quoted unless they
           contain only alphanumeric characters or "./-_".  This is useful for shell scripts to
           capture the driver-generated command lines.

In short, -wrapper will prefix each of the subcommands with a
user-supplied wrapper program, while -### simply prints out the
subcommands that the driver would run if you ran it without that
option. clang supports the latter of these but not the former.

On my system, the GCC driver run on a hello-world C program generates
the following subcommands.

$ gcc -### -o hello hello.c
(snipped various less interesting lines)
/usr/lib/gcc/x86_64-linux-gnu/10/cc1 -quiet -imultiarch x86_64-linux-gnu hello.c -quiet -dumpbase hello.c "-mtune=generic" "-march=x86-64" -auxbase hello -fasynchronous-unwind-tables -o /tmp/ccXlWTFa.s
as --64 -o /tmp/ccNhuw5L.o /tmp/ccXlWTFa.s
/usr/lib/gcc/x86_64-linux-gnu/10/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/10/liblto_plugin.so "-plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/10/lto-wrapper" "-plugin-opt=-fresolution=/tmp/cc77VLgc.res" "-plugin-opt=-pass-through=-lgcc" "-plugin-opt=-pass-through=-lgcc_s" "-plugin-opt=-pass-through=-lc" "-plugin-opt=-pass-through=-lgcc" "-plugin-opt=-pass-through=-lgcc_s" --build-id --eh-frame-hdr -m elf_x86_64 "--hash-style=gnu" --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o hello /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/10/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/10 -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/

That's already quite a mouthful. But the -wrapper or -### options can
take care of this "deconstructing" thing for us. That's the back-room
approach: our tool needs to work only with the resulting split-out
subcommands. This makes it less error-prone than if we tried to
deduce the subcommands (or equivalents-we-hope) from the driver
command. Despite all this, we still have a fair number of problems to
solve.

De-integrating preprocessing from compilation

In modern versions of both GCC and Clang, the compiler and
preprocessor are integrated by default. You can see that in the
above: there is only a single cc1 command, no separate preprocessing
stage. So, if we want to interfere with the source code after
preprocessing, we appear to be stuck. With GCC we can ask nicely by
adding -no-integrated-cpp and get back a separate preprocessing
subcommand.

$ gcc -### -no-integrated-cpp -o hello hello.c
/usr/lib/gcc/x86_64-linux-gnu/10/cc1 -E -quiet -imultiarch x86_64-linux-gnu hello.c "-mtune=generic" "-march=x86-64" -fasynchronous-unwind-tables -o /tmp/ccdz9uxk.i
/usr/lib/gcc/x86_64-linux-gnu/10/cc1 -fpreprocessed /tmp/ccdz9uxk.i -quiet -dumpbase hello.c "-mtune=generic" "-march=x86-64" -auxbase hello -fasynchronous-unwind-tables -o /tmp/ccisTsYo.s
as --64 -o /tmp/ccALTbUH.o /tmp/ccisTsYo.s
/usr/lib/gcc/x86_64-linux-gnu/10/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/10/liblto_plugin.so "-plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/10/lto-wrapper" "-plugin-opt=-fresolution=/tmp/ccx8FfYe.res" "-plugin-opt=-pass-through=-lgcc" "-plugin-opt=-pass-through=-lgcc_s" "-plugin-opt=-pass-through=-lc" "-plugin-opt=-pass-through=-lgcc" "-plugin-opt=-pass-through=-lgcc_s" --build-id --eh-frame-hdr -m elf_x86_64 "--hash-style=gnu" --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o hello /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/10/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/10 -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/10/../../.. /tmp/ccALTbUH.o -lgcc --push-state --as-needed -lgcc_s --pop-stat
e -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/10/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/crtn.o

With Clang we can't ask directly, but it turns out that there is a
hack that works in the versions I've tried. If we add
-traditional-cpp to the arguments, it will split the two phases
apart, presumably since the integrated preprocessor can't do the
traditional emulations. Compare the following.

$ clang-13 -### -o hello hello.c
"/usr/lib/llvm-13/bin/clang" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "hello.c" "-mrelocation-model" "static" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/tmp" "-resource-dir" "/usr/lib/llvm-13/lib/clang/13.0.1" "-internal-isystem" "/usr/lib/llvm-13/lib/clang/13.0.1/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdebug-compilation-dir=/tmp" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/hello-e8c787.o" "-x" "c" "hello.
c"
"/usr/bin/ld" "--hash-style=both" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "hello" "/usr/lib/x86_64-linux-gnu/crt1.o" "/usr/lib/x86_64-linux-gnu/crti.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/12/crtbegin.o" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/12" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/usr/lib/llvm-13/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/hello-e8c787.o" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/bin/../lib/gcc/x86_64-linux-gnu/12/crtend.o" "/usr/lib/x86_64-linux-gnu/crtn.o"

... versus ...

$ clang-13 -### -traditional-cpp -o hello hello.c
clang: error: the clang compiler does not support '-traditional-cpp'
 "/usr/lib/llvm-13/bin/clang" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-E" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "hello.c" "-mrelocation-model" "static" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/tmp" "-resource-dir" "/usr/lib/llvm-13/lib/clang/13.0.1" "-internal-isystem" "/usr/lib/llvm-13/lib/clang/13.0.1/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdebug-compilation-dir=/tmp" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-traditional-cpp" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/hello-8e16c8.i" "-x" "c" "hello.c"
 "/usr/lib/llvm-13/bin/clang" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-disable-free" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "hello.c" "-mrelocation-model" "static" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/tmp" "-resource-dir" "/usr/lib/llvm-13/lib/clang/13.0.1" "-fdebug-compilation-dir=/tmp" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/hello-c08823.o" "-x" "cpp-output" "/tmp/hello-8e16c8.i"
 "/usr/bin/ld" "--hash-style=both" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "hello" "/usr/lib/x86_64-linux-gnu/crt1.o" "/usr/lib/x86_64-linux-gnu/crti.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/12/crtbegin.o" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/12" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/usr/lib/llvm-13/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/hello-c08823.o" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-nee

So our wrapper can add -traditional-cpp if it's absent, then, if we
added it, strip it out of the preprocessing-only command that
emerges. This is nasty and may stop working, of course. In fact, in a
way, it already has! Notice that although the Clang driver generated
the subcommands, it has also generated an error message saying they
won't work. And they don't! Even though -traditional-cpp is listed in
the --help output of Clang 13, the actual code seems to have been
removed from the preprocessor. This is again illustrating the
distinction between the driver command (which does support the
option) and the subcommands (which don't). Luckily, we only need the
driver part: we make it create these commands in order to then delete
-traditional-cpp from them. That makes them work again! And most
importantly, there's now two of them: preprocessing is now separate
from compilation.

Precise scanning, a.k.a. distinguishing option and non-option
arguments

Even though we're not supplying a whole new front door, there's still
no getting away from it: to be robust, our script needs complete
knowledge of each subcommand's command-line syntax, up to option
arguments. This is a problem I call precise scanning, because
scanning a.k.a. lexing is about chunking things up. Here we're
chunking command-line arguments, so that options (like -o) are
correctly chunked together with their argument (the output filename).
Front-door approaches need this too: although they only need to know
the driver's options, not subcommands', they also need to know the
semantics (i.e. which subcommands are implied), not just the syntax
(how to scan the arguments).

Without precise scanning, we cannot reliably identify input and
output files, nor be sure that we're seeing all options we may be
interested in (and not seeing them spuriously). Consider the slightly
wacky command cc -o -c /tmp/hello.c. If we run it, a fully-linked
output file called "-c" will appear. If I naively scanned the command
line, I'd have seen -c and inferred that the command is not doing
linking, but that would be wrong. In general, while (almost) all
options begin with a -, the converse is not true: arguments beginning
with - need not be options. Note also that the bare - is not an
option but a filename, denoting standard input or output. Now
consider that there are hundreds of options that take arguments: -o
is well-known but there's plenty that aren't. The approach I've taken
is based on scraping the help text of GCC and Clang, since this is
mechanically generated and fairly easy to de-generate. The scrapings
are cached in the script, but the script tells you how to do a
re-scrape using your actual live compiler version, if it sees options
it doesn't understand.

It turns out that this scraping exercise is a good way to find ways
in which the online help is not complete. Some options are also
manually added to the script. For GCC these include our friends
-wrapper and -no-integrated-cpp, but also the -M*, -D*, -L* and -A*
options. Each of these omissions is arguably reportable bug, although
I haven't reported any myself yet. (The help also omits most of the
-O* and -g* ones, although these are covered by the spiel at the
bottom, saying that options "starting with -g, -f, -m, -O, -W, or
--param are automatically passed on to the various sub-processes".
This list is not exhaustive!)

Interpreting compiler-internal subcommands, namely cc1

Preprocessing and/or compiling use an internal and relatively
undocumented subcommand, cc1. GCC actually has a separate cc1
program. In clang's case, it's simply the same clang binary invoked
with the -cc1 option. The effect is the same. Note that clang's cc1
options are different from GCC's.

As part of my "generic" requirement, I want an interface that lets a
client say "wrap preprocessing with the following tool". What options
will this tool receive? I could just say "it will receive cc1
options; suck it up!" But then the tool is coupled to a particular
compiler. Given that our script already needs complete knowledge of
the command-line syntax, I'd rather that only the script need know
this, and that we somehow find a generic way to write the tool. For
now I'll just tackle cc1-related issues, but we'll tackle a strong
version of this "generic" requirement later.

The options to cc1 are mostly the same as ones accepted by the
compiler driver that have to do with compilation proper--but not
always. There are several gotchas. Some driver arguments, like -c,
are not accepted by the compiler proper, as they would make no sense.
Some compiler-proper arguments are not understood by the driver; one
example is that cc1 as invoked by gcc is usually given an extra
argument -auxbase <arg> that the driver does not accept. And finally,
some arguments have different semantics and/or different syntax
between the two. For example, -MD takes no arguments as an option to
the driver but it does take an argument when given to cc1 as invoked
by gcc (it's like -MD -MF). The wrapper script needs to handle all
this nastiness, so that tools don't have to. It contains a scanner
for cc1 options, generated by the same scraping method as before, and
it contains some logic for rewriting specific options such as -MD.
More generally, we can think of the rewriting it does as "lifting"
back to a driver command.

Providing tools with a "lifted" driver-like command interface

We've deciding that giving our own tools a cc1-style command line is
not generic enough. In the case of preprocessing, we might therefore
think instead about lifting to a cpp invocation. cpp is still a
vaguely standard tool with a documented interface. It seems appealing
that our wrapper could translate to this from the cc1 command, and
our tool to emulate cpp. I went a long way down that road, but it was
a mistake. Things start going wrong when we consider: which cpp
should we emulate? The tool will probably run this "real" cpp, with
somehow modified input and/or modifying its output. Which cpp is the
real cpp? Since a modern compiler never actually calls cpp at all,
guesswork is required. There can well be many versions of cpp on the
system. Since older ones will have an earlier default C standard, we
may get subtly wrong preprocessing if we choose wrongly.

There is a real example here, of a rather self-applicative kind.
Older versions of GCC's own source code contain token sequences that
look like U"somestring", where U is supposed to get macro-expanded.

cpphash.h:
#define U (const U_CHAR *)  /* Intended use: U"string" */

In cpplex.c and cpplib.c there are uses of this form. These files are
supposed to be compiled with a C99 or earlier compiler.

In more recent preprocessors, this token sequence is interpreted as a
single Unicode literal token, causing a downstream error when doing
compilation proper (of the compiler).

$ echo 'U"blah"' | cpp-4.9 -DU=X -
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "<stdin>"
X "blah"

$ echo 'U"blah"' | cpp -DU=X -
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "<stdin>"
U"blah"

$ echo 'U"blah"' | cpp -DU=X -
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "<stdin>"
U"blah"

The change was between C99 and C11. Older versions of GNU CPP have a
pre-C11 default standard, but newer ones use C11 and will
mispreprocess the above code. How can we know to select the older cpp
as our underlying cpp, in only these cases? We could at best resort
to substitutions on the driver's executable name, like s/gcc/cpp/ or
s/clang/clang-cpp/, (while making sure we pass along -std= if it's
present). But that is not reliable.

Instead I've abandoned the idea of lifting to cpp. Instead I lift
uniformly to single-function driver invocations, i.e. either a cc -E
command or a cc -S command, or one that just runs the assembler, or
one that just runs the linker. Whereas we might not know which cpp to
run, we do always know which driver to run, because we, the wrapper
script, are called by a specific driver. (In practice, recovering
this still gets a little ugly: one way is to scrape the parent PID's
command line from Linux's /proc or similar. We also accept it
explicitly, via the CC_DRIVER variable, to stop our script becoming
an untestable Heisenmess.)

This approach means our wrapper script has to lift cc1's special
options back into the driver's command-line language. For example, we
have to translate cc1's non-standard -MD option back into the -MD -MF
that is the driver's interface for this purpose. We'll come to this
shortly.

Digression: cc -E is not cpp!

Even make makes this conflation: GNU Make on my system defaults CPP
to cc -E. But in general a system's C preprocessor can and does take
arguments that the compiler driver does not. That is the entire
motivation for -Xpreprocessor and -Wp, after all. However, the real
smoking gun that shows the command lines are not compatible is
infile outfile: the preprocessor lets you name the output file
directly on the command line, without -o. One could easily imagine
writing a %.i: %.c Make rule that tried using CPP with this
infile outfile syntax, and then getting very confused about why it
didn't work.

Not just cc1...

There is the assembler, of course, which is straightforward. But at
least with GCC, linking is less straightforward: it is done via a
non-standard collect2 tool, which exists to support constructors and
destructors (not supported natively by many loaders). It can end up
running the real linker twice. However, it is a very thin wrapper
around the real linker and its command line is very nearly the same.
It also helpfully makes it easy to override its choice of real
linker, so we can easily supply a wrapper ld rather than a wrapper
collect2. I'll skip details of this here.

Expanding the command language, while avoiding in-tool scanning

Suppose our tool is a replacement preprocessor--or, as we've covered,
a replacement for cc -E. Internally, it calls a "real" preprocessor
somehow. So there are now two "preprocessors": the real one and our
tool that wraps it.

We might want there to be options that our tool takes but the real
preprocessor doesn't, and vice-versa. For example, my cilpp is a
preprocessor that supports plugins for source-to-source
transformation after "normal" preprocessing, using a -plugin option
that no ordinary preprocessor supports. But also, there's a ton of
options we might want, orthogonally, to pass to the underlying
preprocessor, such as -dD which will leave the effective #define
directives in the preprocessed output, and which is handy for
debugging.

So, the command-line language that our tool must accept is some kind
of composite. This is the stronger version of the "generic" "script
does the scanning" problem we set for ourselves earlier. We don't
want our tool to have to reimplement scanning of the "standard"
command lines, a tool may want to extend that command-line language.
How can we arrange that the tool itself doesn't have to replicate all
that painful precise scanning logic internally? We know how important
it is to be able to scan command lines precisely, in order to
identify options and input files accurately.

Since we can't bring the standard scanning to the tool, our answer
is: take the tool's scanning to the same place as the standard
scanning, i.e. to our overall wrapper script. We make our tool
declare which options it's interested in, and we have our wrapper
script scan them for it.

This is plumbed through by environment variables: the tool sets
CC_IDENTIFY_ARGS with a vaguely getopt-like spec (but uglier) of the
options it cares about. The wrapper script responds by setting
CC_IDENTIFIED_ARGS, at the time the tool is invoked, to encode the
positions of these arguments on the command line. The tool then
doesn't need to parse its own command line; it can just look at this
environment variable, since the work has already been done. The tool
might additionally make a best-effort attempt at scanning the command
line, since invoking it directly may be convenient. (But it should
give up, with a warning, when it sees an option it doesn't
understand, because precise scanning is impossible from that point
onwards. Clang tools take the approach of requiring their arguments
to be left of a --, with general compiler arguments to the right.
That is similar to what I'm doing here, but results in a narrower
interface. My demo cilpp, that I'll present shortly, also accepts --
with similar meaning, i.e. the tool can conclude "no more options for
me past this point" and stop scanning, much as if it had seen any
other unrecognised argument but without issuing a warning.)

(How does a tool like cilpp set up CC_IDENTIFY_ARGS before the
wrapper is run, as it must do? If the wrapper hasn't run yet, surely
the tool itself hasn't run yet? Well, we have to change that. In fact
we might already have reason to invoke the compiler with help from
some custom tool, maybe as something like cc `my-tool-cflags`, as a
shorthand for cc -wrapper my-wrapper -no-integrated-cpp. That seems
useful, except that obviously, in this scenario a my-tool-cflags
script will be powerless to set the environment variables for the cc
command itself, and hence also cannot influence the environment of
the wrapper script that it invokes. I thought about making our
compiler invocation something like -wrapper "env CC_IDENTIFY_ARGS=
optspec /path/to" but this also doesn't work: -wrapper is not subject
to word-splitting. (ACTUALLY: as the gdb example above shows, it is
subject to comma-splitting! I missed that, so the following is
probably unnecessary.) Instead we have to introduce another level of
indirection: our tool provides its own wrapper tool, that sets
CC_IDENTIFY_ARGS and then runs the generic wrapper (found using
WRAPPER or by pathname guesswork).

Recapping what we've achieved

The net effect of all the above is that using a command like

cc -wrapper /path/to/tool/wrapper ...

we can extend the behaviour of the compiler, as a black-box
intervention around each of its subcommands (preprocess, compile,
assemble, link), while also extending its command-line option syntax.
We do this in way that tool/wrapper defines, but with the argument
processing done by a generic underlying wrapper script that many such
tools can share.

Lifting in the presence of tool-specific tool-directed options

We mentioned that our wrapper script must "lift" from cc1
compiler-private command lines back to cc "driver" command lines,
that are relatively more documented and standardised. There are two
problems. Firstly, how do we translate the options in general? Some
are not understood by the driver (e.g. -auxbase); what should we do
with those? Secondly, how does this interact with options like
-Xpreprocessor or -Wp,..., whose role is to pass options through to
preprocessors, assemblers and the like? Of course there are also
-Xassembler or -Wa,... and -Xlinker or -Wl,... (but, curiously, no
-Xcompiler) but this lifting is only necessary for -Xpreprocessor
because there is no cc1 equivalent for the assembler or linker. (But
see above about collect2.)

This "tool-directed options" feature complements our previous idea of
extending the command-line syntax in a per-tool fashion. Our extended
preprocessor might support a special option, and we'd use
-Xpreprocessor to pass it through.

The problem is that when we see a cc1 command, the -Xpreprocessor
options and the like have already been stripped out by the driver. If
we are to lift the cc1 command back up to a driver command, we need
to reinstate them somehow. Otherwise we will pass the driver cc1- or
tool-specific options in unqualified form, and it will choke. That
creates some guesswork: how do we know which cc1 options need this
treatment? As a pathological example, suppose our driver command is

cc -wrapper mywrapper -Xpreprocessor -blah -o -blah blah.c

... where we are asking the compiler to output a file called -blah.
When we get the cc1 preprocessing command it will say something
including (leaving out the usual cc1 noise)

cc1 -E -blah -o -blah blah.c

... and we know that we want to lift this back to:

cc -E -Xpreprocessor -blah -o -blah blah.c

... but it's not as straightforward as always prefixing our
known-special option -blah with -Xpreprocessor. Again, we're back to
precise scanning. Firstly we assume we can scan the driver command
itself precisely. Secondly we assume that only options, not input
files, make the driver choke, and that we can identify these as
hyphen-prefixed words at the head of a scanned chunk (but not a plain
- by itself, which is a filename). In short, any option we see with
cc1 that appeared at the head of a chunk on the plain driver command,
where that chunk was not qualified by -Xpreprocessor, we know should
pass through fine. Or else if it was so qualified, we add the
qualifier back (perhaps many times--it needs repeating it before any
argument that follows within the chunk).

What if the cc1 option just didn't appear in the driver command? This
is a common case! For example, GCC's cc1 options -quiet and -auxbase
are not understood by the driver. They were generated by the driver,
but we don't know which of the driver's options caused them to be
generated. This is the nastiest semantic issue we face. We have a few
choices. If preprocessing, we could conservatively always prefix
-Xpreprocessor since we know that'll make it come through to cc1
again. Or to be more refined, we could use our scanner for driver's
command-line options: if the option would be recognised according to
those, we let it through unprefixed, otherwise we prefix. (This is
still assuming the option means the same thing to the driver as it
does to cc1! So we must do this only after normalising -MD away. We
also risk introducing duplicates, if the driver re-generates another
copy of the option and that matters semantically. I'm not currently
aware of any issues of this kind.)

This prefixing is all very well, but we don't have an equivalent for
compilation proper. We could alternatively assume that the driver
will always regenerate whatever cc1-specific option we're seeing, so
just drop it from the driver command we build. That is unsafe in
general but it's what I currently do. (For preprocessing, the
"refined" version described above is probably the right fix. Then to
address the lack of -Xcompiler we could write a "sub-wrapper", used
only while doing our own single-function compilation-proper driver
invocations. The sub-wrapper would slurp extra options from an
environment variable. It could add them back in a "no duplicates"
way, so that if the driver created the option of its own accord, it
won't be added twice. The main wrinkle is some guesswork about when
this no-duplication semantics is necessary.)

Tying it all together

The -wrapper and -### interfaces follow a depressingly familiar
pattern: they are extension interfaces that are undoubtedly "useful",
but impossible to use reliably, because using them needs too much
change-prone, implementation-specific knowledge. This built-in
unreliability could largely be solved if GCC and Clang provided two
things: firstly, online help that is complete with respect to their
command line and intentionally machine-readable, and secondly,
instead of ad-hoc options like -### or -no-integrated-cpp, a
systematic interface for boiling down their command-line complexity
by normalisation--possibly as an extension to the machine-readable
interface description, and possibly based on the exact
"single-function driver invocations" idea that I've motivated here.
Currently, thanks to the opaque mapping between driver and cc1
options, and the lack of -Xcompiler, I could only implement it modulo
various hacks and scripted guesswork. (In fairness, GCC does have
spec strings which go some way to declaratively specifying how the
driver generates subcommands. I could possibly read these to
eliminate some of the guesswork in my approach; for now it doesn't
seem worth it, especially as I'm not aware of a Clang equivalent.)

For now, packaging this change-prone nastiness all up in one somewhat
change-prone script is about the best we can do. My own script is in
my toolsub repository's wrapper subdirectory and there exists a
minimal-ish demo of how to use it in the cilpp tool in the same
repository. cilpp is a replacement preprocessor that accepts CIL
source-to-source passes as plugins. Since it has its own command-line
options for identifying these plugins, it demos the "identified
arguments" facility, and has its own "outermost wrapper" that
supplies the necessary CC_IDENTIFY_ARGS configuration. As we saw
above, this allows the wrapper script to do the scanning; the cilpp
tool now does not need to embed a full command-line scanner, an
improvement of earlier versions which did attempt this and were found
to be somewhat bug-prone.

I thought it important that my wrapper be a script, not a compiled
program--given its imperfections, I expected users to have to hack it
from time to time, so I want to make that as immediate as possible.
The downside is that my script is surprisingly slow: on my (old)
laptop it adds something like one second (!) per complete
preprocess-compile-assemble-link invocation. That is pretty bad,
although initial versions were much worse. I gained some speed by
delegating the command-scanning to generated awk code (rather than
generated shell code), more by being careful to avoid quadratic logic
that re-scanned the entire argument list to process one argument
(bash is such that even on a two-dozen-element list you can really
notice these), and some more by allowing some bashisms while being
careful about the patterns I used (associative arrays are handy;
piping into while read loops is mostly to be avoided). That has got
me to the point where more time is spent in awk than in bash, at
least. Still, I would like to get time to investigate why bash is so
amazingly slow at certain things.

A final thought: a compiled wrapper program might not be any less
hackable if it is automatically re-made when new changes are applied.
Currently our awk code, generated from compiler help text, is
effectively cached in the wrapper script file itself, but can be
easily be remade--manually at present, but perhaps automatically
triggerable somehow. If we can trigger this sensibly, then
hackability doesn't depend on being an interpreted script after all.
Why can't the wrapper be a self-remaking C program? One could view
this as a poor person's output-caching JIT--and possibly, on
reflection, with less of the "poor". Is there a principled way to
have self-remaking programs that is not too disruptive to the
established toolchain model? There are some related tricks going on
in Justine Tunney's "actually portable executables" (sorry, not doing
the Greek), in Fabrice Bellard's tcc -run hash-bang lines, and in my
own shell-Make hybrids. Could we imagine a distribution, perhaps
vaguely Nix-like, where all software is self-remaking by default? How
can we hide the latency of rebuilding a large program? Can we tweak
how ABIs work to enable builds to be made more incremental? This and
other kinds of "live programming by self-remaking" are a topic I'd
like to explore further.

[/devel] permanent link contact

---------------------------------------------------------------------
Powered by blosxom

validate this page