[HN Gopher] My Favorite C++ Pattern: X Macros (2023)
___________________________________________________________________
My Favorite C++ Pattern: X Macros (2023)
Author : ibobev
Score : 70 points
Date : 2025-03-25 14:57 UTC (8 hours ago)
(HTM) web link (danilafe.com)
(TXT) w3m dump (danilafe.com)
| Blackthorn wrote:
| I understand the use case of this, but when I see it I always
| wonder if, and think I would prefer, some external code
| generation step instead rather than falling back on macros in the
| preprocessor. Like an external script or something.
| dataflow wrote:
| External codegen introduces a lot of friction in random places.
| Like how your editor can no longer understand the file before
| you start building. Or how it can go out of date with respect
| to the rest of your code until you build. If you can do it with
| a macro it tends to work better than codegen in some ways.
| writebetterc wrote:
| IDEs understand preprocessor macros, so IDE features (jump2def,
| etc) work with this. IDEs also can expand the macro
| invocations. So, I prefer the macros when possible :-).
| Someone wrote:
| > IDEs understand preprocessor macros, so IDE features
| (jump2def, etc) work with this.
|
| Do they? X macros often are used with token pasting
| (https://gcc.gnu.org/onlinedocs/cpp/Concatenation.html), as
| for example in (FTA) #define
| AST_BEGIN_SUBCLASSES(NAME) START_##NAME ,
|
| Are modern IDEs/compiler toolchains smart enough to tell you
| that _START_foo_ was created by an expansion of that macro?
| jcelerier wrote:
| any non-toy IDE can do that. Most IDEs use clang directly
| for parsing nowadays.
| tom_ wrote:
| Yes. I use this with VS2019 for generating enum names, and
| they interact fine with auto complete and go to definition.
| wat10000 wrote:
| Now you have a additional stage in your build, a bunch of new
| code to maintain, and either a bespoke language embedded in
| your standard C++ or a bunch of code emitting C++ separately
| from the code it logically belongs with.
|
| Compare with a solution that's 100% standard C++, integrates
| into your build with zero work, can be immediately understood
| by anyone reasonably skilled in the language, and puts the
| "generated" code right where it belongs.
| MITSardine wrote:
| CMake makes this pretty painless. My codegen targets have
| only two additional instructions to handle the generation
| itself and dependencies: add_custom_command to call the
| codegen exec, and then add_custom_target to wrap my outputs
| in a "virtual" target I can then make the rest of my program
| depend on, but this is just for tidying up.
|
| And I'll dispute the fact that any complex C prepro task "can
| be immediately understood by anyone reasonably skilled in the
| language". Besides, code should ideally be understood by
| "anyone reasonably likely to look at this code to work in
| it", not "reasonably skilled".
| wat10000 wrote:
| This isn't complex. It's a bit unusual, but not hard to
| understand if you understand the basics of how #include and
| #define work.
|
| If you're working on the sort of C++ codebase that would
| benefit from this sort of code generation, and you're not
| reasonably skilled in C++, then god help you.
| MITSardine wrote:
| Are you talking about the X macro itself, or more
| generally?
|
| I may be the obtuse one here, but for a more complex
| example, it took me a few hours to manage to make nested
| loops using Boost PP (for explicit instantiations). Even
| so, I avoid having to write a new one that's not a quick
| copy-paste because it's quite different from usual C++
| programming, so my painfully acquired understanding
| quickly evaporated... as I suspect is the case of anyone
| who doesn't particularly focus on the C prepro.
|
| In the end, it's just simpler to get some Python script
| or C++ program to write a string and dump that to a file
| than to write something illegible with the C
| preprocessor, if doing something at all complicated (in
| my opinion).
| wat10000 wrote:
| I'm talking about X-macros. There's a wide range of
| preprocessor shenanigans, from "everybody needs to know
| this" to "oh my god why." Each construct needs to be
| evaluated on its merits. IMO X-macros are closer to the
| simpler side of that spectrum. Consider writing things
| out by hand if you just have a few, but if you have a lot
| of things repeating like this, they're a fine tool to
| use. Boost PP is a whole different level of
| ridiculousness and I don't see ever using that sort of
| thing for anything serious.
| jasonthorsness wrote:
| The C# "source generator" approach is a good compromise; it
| runs within the build chain so has the ease-of-use of macros in
| that respect, but they don't need to be written in a weird
| macro language (they are C# or can call external tool) and when
| you debug your program, you debug through the generated source
| and can see it, more accessible than macros. Not sure if there
| is something similar in C/C++ integrated with the common
| toolchains.
|
| But when working outside C/C++ I've found myself missing the
| flexibility of macros more times than I can count.
| mauvehaus wrote:
| I've done this in C with the C preprocessor and Java with
| m4[0].
|
| The upside of doing it natively is that it keeps the build
| simpler. And everybody at least knows about the existence of
| the C preprocessor, even if they don't know it well. And it's
| fairly limited, which prevents you from getting too clever.
|
| The big downside of doing it with the C preprocessor is that
| the resulting code looks like vomit if it's more than a line or
| two because of the lack of line breaks in the generated code.
| Debugging it is unenjoyable. I'd recommend against doing
| anything super clever.
|
| The upside of doing it out of band is that your generated
| source files look decent. m4 tends to introduce a little extra
| whitespace, but it's nothing objectionable. Plus you get more
| power if you really need it.
|
| The downside is that almost nobody knows m4[1]. If you choose
| something else, it becomes a question of what, does anyone else
| know it, and is it available everywhere you need to build.
|
| Honestly, integrating m4 into the build in ant really wasn't
| too bad. We were building on one OS on two different
| architectures. For anything truly cross-platform, you'll likely
| run into all the usual issues.
|
| ETA: Getting an IDE to understand the out of band generation
| might be a hassle, as other folks have mentioned. I'm a vim
| kinda guy for most coding, and doing it either way was pretty
| frictionless. The generated java code was read-only and
| trivial, so there wasn't a lot of reason to ever look at it. By
| the time you get to debugging, it would entirely transparent
| because you're just looking at another set of java files.
|
| [0] This was so long ago, I no longer remember why it seemed
| like a good idea. I think there was an interface, a trivial
| implementation, and some other thing? Maybe something JNI-
| related? At least at first, things were changing often enough
| that I didn't want to have to keep three things in sync by
| hand.
|
| [1] Including me. I re-learn just enough to get done with the
| job at hand every time I need it.
| drwu wrote:
| Another issue is cross-compiling.
|
| External code generation requires (cross) execution of the
| (cross) compiled binary program.
| rcxdude wrote:
| Or to build the generating code for the host instead. Most
| build systems that support cross-compilation can do this,
| except CMake.
| MITSardine wrote:
| After trying to wrangle Boost PP and other advertised compile-
| time libraries such as Boost Hana (which still has some runtime
| overhead compared to the same logic with hardcoded values),
| I've finally converged to simply writing C++ files that write
| other C++ files. Could be Python, but I rather keep the build
| simple in my C++ project. Code generation is painless with
| CMake, no idea with other build configuration utilities.
| cbuq wrote:
| This sounds pragmatic, but are you writing C++ executables
| that when run create the generated code? Are there templating
| libraries involved?
| MITSardine wrote:
| Yeah, it's all done automatically when you build, and
| dependencies are properly taken into account: if you modify
| one of the code generating sources, its outputs are
| regenerated, and everything that depends on them is
| correctly recompiled. This doesn't take much CMake logic at
| all to make work.
|
| In my case, no, it's dumb old code writing strings and
| dumping that to files. You could do whatever you want in
| there, it's just a program that writes source files.
|
| I do use some template metaprogramming where it's practical
| versus code generation, and Boost Hana provides some
| algorithmic facilities at compile time but those incur some
| runtime cost. For instance, you can write a while loop with
| bounds evaluated at compile time, that lets you use its
| index as a template parameter or evaluate constexpr
| functions on. But sometimes the best solution has been (for
| me, performance/complexity wise) to just write dumb files
| that hardcode things for different cases.
| Arech wrote:
| > Boost Hana (which still has some runtime overhead compared
| to the same logic with hardcoded values)
|
| Can you elaborate on that? What was your use-case for which
| this was true?
| MITSardine wrote:
| One case I benchmarked was Bernstein/Bezier and Lagrange
| element evaluation. This is: given a degree d triangle or
| tetrahedron, given some barycentric coordinates, get the
| physical coordinate and the Jacobian matrix of the mapping.
|
| Degree 2, Lagrange:
|
| - runtime: 3.6M/s - Hana: 16.2M/s - Hardcoded: 37.7M/s
|
| Degree 3, Lagrange: 2.6M/s, 6.4M/s, 13.4M/s (same order).
|
| "Runtime" here means everything is done using runtime
| loops, "Hana" using Boost Hana to make loops compile-time
| and use some constexpr ordering arrays, "hardcoded" is a
| very Fortran-looking function with all hardcoded indices
| and operations all unrolled.
|
| As you see, using Boost Hana does bring about some
| improvement, but there is still a factor 2x between that
| and hardcoded. This is all compiled with Release
| optimization flags. Technically, the Hana implementation is
| doing the same operations in the same order as the
| hardcoded version, all indices known at compile time, which
| is why I say there must be some runtime overhead to using
| hana::while.
|
| In the case of Bernstein elements, the best solution is to
| use de Casteljau's recursive algorithm using templates (10x
| to 15x speedup to runtime recursive depending on degree).
| But not everything recasts itself nicely as a recursive
| algorithm, or I didn't find the way for Lagrange anyways. I
| did enable flto as, from my understanding (looking at call
| stacks), hana::while creates lambda functions, so perhaps a
| simple function optimization becomes a cross-unit affair if
| it calls hana::while. (speculating)
|
| Similar results to compute Bernstein coefficients of the
| Jacobian matrix determinant of a Q2 tetrahedron, factor 5x
| from "runtime" to "hana" (only difference is for loops
| become hana::whiles), factor 3x from "hana" to "hardcoded"
| (the loops are unrolled). So a factor 15x between naive C++
| and code generated files. In the case of this function in
| particular, we have 4 nested loops, it's branching hell
| where continues are hit very often.
| Arech wrote:
| Hhhmmm, interesting, thanks for reply!
|
| That would be fairly interesting to look at the actual
| code you've used, and have a look at the codegen. By a
| chance, is it viable for you to open-source it? I'd guess
| it should bear lots of interest for Hana author/s.
|
| What compiler/version did you use? For example, MSVC
| isn't (at least wasn't) good at always evaluating
| `constexpr` in compile-time...
|
| > hana::while creates lambda functions, so perhaps a
| simple function optimization becomes a cross-unit affair
| if it calls hana::while. (speculating)
|
| Hmm, I'd say it (LTO) shouldn't influence, as these
| lambdas are already fully visible to a compiler.
| MITSardine wrote:
| I never thought to contact them, but I might do that,
| thanks for the suggestion. This is something I tested
| almost two years ago, I have these benchmarks written
| down but I've since deleted the code I've used, save for
| the optimal implementations (though it wouldn't take too
| long to rewrite it).
|
| I tested with clang on my Mac laptop and gcc on a Linux
| workstation. Version, not sure. If I test this again to
| contact the Hana people, I'll try and give all this
| information. I did test the constexpr ordering arrays by
| making sure I can pass, say, arr[0] as a template
| parameter. This is only possible if the value is known at
| compile time. Though it's also possible the compiler
| could be lazy in other contexts, as in not actually
| evaluating at compile time if it figures out the result
| is not necessary to be known at compile time.
|
| Oh yeah, you're right, I was confusing translation unit
| and function scope.
| rcxdude wrote:
| CMake has a particularly irritating flaw here, though, in
| that it makes no distinction between host and target which
| cross-compiling, which makes it really difficult to do this
| kind of code generation when supporting this use-case (which
| is becoming more and more commoon).
| MITSardine wrote:
| Right, I hadn't thought of that, to be honest. If I
| understand correctly, you're saying the codegen targets
| will be compiled to the target arch, and then can't be run
| on the machine doing the compiling?
|
| I think one solution might be to use
| target_compile_options() which lets you specify flags per
| target (instead of globally), assuming you're passing flags
| to specify the target architecture.
| rcxdude wrote:
| That only works if it's mostly the same compiler,
| unfortunately. They could be completely different
| executables, calling conventions, etc. I don't know why
| CMake still has such a huge hole in its feature set, but
| it's quite unfortunate.
| paulddraper wrote:
| Spoken like a Go programmer :D
|
| That introduces some other tool (with its own syntax), an extra
| build step, possible non-composability with other features.
|
| A preprocessor (for good or bad) IS a code generation step.
| jchw wrote:
| X Macros are a classic C++ pattern. Less known than some of the
| more popular techniques like curiously recurring template
| pattern, but still quite common in large codebases.
|
| (Although... it _is_ a neat trick, but... It is kind of mostly
| useful _because_ C++ macros are not very powerful. If they were
| more powerful, most uses of X Macros could be replaced by just
| having a single macro do all of the magic.)
|
| I recently saw something I hadn't seen before in a similar vein.
| There's a million different bin2c generators, and most of them
| will generate a declaration for you. However, I just saw a setup
| where the bin2c conversion just generates the actual byte array
| text, e.g. "0x00, 0x01, ..." so that you could #include it into
| whatever declaration you want. Of course, this is basically a
| poor man's #embed, but I found it intriguing nonetheless. (It's
| nice that programming languages have been adding file embedding
| as a first-class feature. This really simplifies a lot of stuff
| and removes the need to rely on more complex solutions for
| problems that don't really warrant them.)
| gpderetta wrote:
| > If they were more powerful, most uses of X Macros could be
| replaced by just having a single macro do all of the magic
|
| You can do exactly that, it is just very tedious without a
| support preprocessor library like boost.PP.
| WalterBright wrote:
| > X Macros are a classic C++ pattern
|
| I've seen them in 1970s assembler code.
| jchw wrote:
| Y'know, I've never actually seen this before, but that makes
| a lot of sense. They're also (of course) a common pattern in
| C, and probably basically any language that has a C-style
| preprocessor.
| kragen wrote:
| https://en.wikipedia.org/wiki/X_macro claims they go back to
| the 01960s, but unfortunately the reference is to a DDJ
| article. Is there a chance we could like run a Kickstarter to
| buy DDJ's archives and fix their website?
| loeg wrote:
| I hate X-Macros, but they're very useful in some situations. We
| use them to generate an enum of error values and also a string
| table of those names. (Unlike two separate tables, the X macro
| version is guaranteed to be the exact correct length / values
| align with the corresponding string.)
| IshKebab wrote:
| I agree. Terrible hack, but if you're in terrible hack land
| they're quite useful. LLVM uses them extensively.
| msarnoff wrote:
| That's been my primary usage of them. Another is to create a
| list of config file options (or command line arguments) like
| X(foo, int, "set the number of foos")
| X(filename, std::string, "input filename") X(verbose,
| bool, "verbose logging")
|
| which can then be used to (a) generate the fields of a config
| struct, define the mapping from string to field (using the
| stringifying macro operators), define what functions to use for
| parsing each field, create the help message, etc. Basically
| like `argparse` or `clap` but much hackier.
|
| As gross as they are, the ability to define one table of data
| that's used multiple ways in multiple places is handy.
| gibibit wrote:
| It is a clever trick. Very useful in C also, maybe more than in
| C++.
|
| It can be overused, though.
|
| Kind of works like Rust declarative macros (`macro_rules!`) in
| that it is often used to repeat and expand something common
| across an axis of things that vary.
|
| It's funny that the simple name X Macros has stuck and is a de
| facto standard. E.g. https://en.wikipedia.org/wiki/X_macro
| suggests they date to the 1960s.
| wat10000 wrote:
| Some minor tweaks to what the author shows to make it even
| better.
|
| Give the macro a more descriptive name. For their example, call
| it GLOBAL_STRING instead of X. I think this helps make things
| clearer.
|
| #undef the macro at the end of the header. That removes one line
| of boilerplate from every use of the header.
|
| Use #ifndef at the top of the header and emit a nice error if the
| macro isn't defined. This will make it easier to understand
| what's wrong if you forget the #define or misspell the macro.
| zem wrote:
| despite the extra boilerplate, I feel like it's still better to
| undef the macros in the same scope they were defined in, so
| that they clearly delimit the code in which the macro is
| active.
| wat10000 wrote:
| I think this pattern is obvious enough that it's clear what's
| going on, but I can see where you're coming from.
| randomNumber7 wrote:
| One the one hand it is great and probably usable to solve actual
| problems.
|
| On the other hand it seems fishy that you need all these hacks to
| do it. There must be a way simpler language than C++ where you
| could do the same easier.
| a_t48 wrote:
| X Macros are great, but always happy when I can replace them with
| something else.
| sfpotter wrote:
| No need for any of this if you use D!
| kevin_thibedeau wrote:
| "X" macros are great until you need two of them visible in the
| same translation unit. It is much better to pass a list macro as
| an argument to a uniquely named X macro and avoid the need to
| ever undef anything.
| skribanto wrote:
| I don't think I follow, do you mind giving a concrete example?
| rcxdude wrote:
| Basically, instead of faffing around with undefing values and
| including different files, you define your list like this:
| #define AN_X_LIST(X) \ X(foo, bar) \
| X(bar, baz)
|
| And then you use it like so: #define
| AN_ASSIGNMENT_STATEMENT(a,b) a = STRINGIFY(b);
|
| And so AN_X_LIST(AN_ASSIGNMENT_STATEMENT)
|
| Will expand to foo = "bar"; bar =
| "baz";
|
| The nice thing about this approach is you can define multiple
| lists and macros, and make higher order macros which use
| them. I have a system like this which allows me to define
| reflective structs in C++ easily, i.e. I define a struct
| like: #define STRUCT_FOO_LIST(X) \
| X(int, bar, 0), \ X(float, baz, 4.0),
| DECLARE_REFLECTIVE_STRUCT(Foo, STRUCT_FOO_LIST);
|
| (where DECLARE_REFLECTIVE_STRUCT basically just does the same
| dance as above with passing different per-element structs
| into the list that it is passed for the struct definition,
| and other utility functions associated with it)
|
| which then makes a struct Foo with members bar and baz with
| the right types and default values, but also I can do
| 'foo_instance.fetch_variant("baz")' and other such
| operations.
|
| The biggest pain with this approach is it's basically dealing
| with a bunch of multi-line macros, so it can get messy if
| there's a typo somewhere (and I strongly recommend an auto-
| formatter if you don't like having a ragged line of
| backslashes to the right of all the code that uses it).
| elteto wrote:
| _This_ is the best pattern for X macros, without any of
| that noise of undef'ing anything.
|
| My approach is to wrap the list elements with two macros:
| an inner transformation one and an outer delimiter, like
| so: #define AN_X_LIST(X, DELIM) \
| DELIM(X(int, foo)) \ DELIM(X(int,
| bar)) \ X(std::string, baz)
|
| Then you can compose different pieces of code for different
| contexts by just swapping out the delimiter. A very
| contrived example: #define SEMICOLON(x)
| x; #define COMMA(x) x, #define
| DECLARE(type, var) type var #define INIT(type, var)
| var{} struct s { AN_X_LIST(DECLARE,
| SEMICOLON); s() AN_X_LIST(INIT, COMMA)
| {} };
| PaulHoule wrote:
| Famous in IBM 360 assembly language, in particular these were
| used for building code based on data schemas in CICS [1]
| applications which were remarkably similar to 2000-era cgi-bin
| applications in that you drew forms on 3270 terminals [2] and
| instead of sending a character for each keystroke like the
| typical minicomputer terminal, users would hit a button to submit
| the form, like an HTML form, and the application mostly shuttled
| data between forms and the database.
|
| [1] https://en.wikipedia.org/wiki/CICS
|
| [2] https://en.wikipedia.org/wiki/IBM_3270
| jdnxndnd wrote:
| How is this C++? This is cpp (c preprocessor)
| glouwbug wrote:
| Lucky for us C++ is backwards compatible ;)
| lowbloodsugar wrote:
| We called these "fruit lists".
| KerrAvon wrote:
| I can't believe this actually has a name (is this an attempt to
| make fetch happen?) and it's not considered an anti-pattern. This
| sort of preprocessor/code mixing is impossible to debug and
| maintain; most senior C++ programmers have advised people to
| avoid doing this since the 1990's.
|
| There is a role for the preprocessor to automate simple tables in
| C and such, but anything complex should be avoided like the
| plague if you don't like tech debt. And there's not much excuse
| for using it in C++ to this extent.
| rcxdude wrote:
| It's not particularly difficult to deal with, as far as C++
| goes (I'd take it over most template shenanigans, the rules are
| a little simpler). It would be easier if C++ had ever tried to
| improve the preprocessor instead of just saying "don't use it"
| and trying to get templates to do a subset of the same kind of
| things you might want the preprocessor for.
|
| (Also, there's a much better version of this I decribed above:
| mucking about with re-including the same file multiple times is
| unnecessary and constraints things too much. You can make a
| much nicer interface for users of the macro with only a little
| extra effort)
| gpderetta wrote:
| The vast majority of the use of X macros (and to be clear, they
| are not that common) I have seen in the wild have been very
| simple uses that are quite straightforward to debug.
|
| Should we have a better solution in 2025? Yes. Lacking that,
| are X macros better than other workarounds? Also yes.
| webdevver wrote:
| i hate this "trick" with a passion. i am the one who winds up
| having to debug this shit, and theres nothing i love doing more
| than trawling through hundreds of thousands of preprocessed C++
| that gets defecated by `gcc -E ...`. even better when the macro
| expansion is 10 levels deep.
|
| you need tables? use python (or your scripting language of
| choice) as a pre-compile step that generates the .cpp/.c/.h
| files, and then build _that_.
| malkia wrote:
| I've learned about them, staring endlessly at the luajit source
| code - for example (not the best example I can remember, but
| still) -
| https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_lex.h#L15
|
| there it defines a TOKEN(_,__) macro generator the luajit
| token/keywords and later generates enums with them.
|
| I've used it recently to wrap a "C" api with lots of functions,
| such that I can redirect calls.
___________________________________________________________________
(page generated 2025-03-25 23:01 UTC)