[HN Gopher] Build System Schism: The Curse of Meta Build Systems
___________________________________________________________________
Build System Schism: The Curse of Meta Build Systems
Author : gavinhoward
Score : 58 points
Date : 2024-03-19 11:44 UTC (11 hours ago)
(HTM) web link (yzena.com)
(TXT) w3m dump (yzena.com)
| jgeada wrote:
| Maybe I'm a dinosaur, but if the product you're building is so
| complex that it needs such a complex (and Turing complete!)
| description for how it needs to be built, maybe the problem is
| that product's architecture. Creating a meta build is fixing the
| symptom and not the problem.
|
| Yes, it is unfortunate that "make" syntax is a bit odd. But if
| you can't describe the hierarchical DAG for how your product
| comes together, that ought to be the problem to tackle rather
| than building a Turing complete language to try to describe how
| you think maybe your product might come together.
| bluGill wrote:
| I want everything declarative, and mostly things are. However I
| keep running into something weird that the build system
| designers didn't think of and so I need something to make it
| work. Note that I'm the cmake expert here, so when I say I keep
| running into these situations understand that most of the other
| developers never hit these things, but when it happens they go
| to me and so I run into them often even though they are rare.
| gavinhoward wrote:
| Author here.
|
| My go-to example is multi-language builds. Those get complex,
| and sometimes, multiple languages are necessary.
| jgeada wrote:
| I've designed and shipped more than one multi-million line of
| code product that had parts in C, C++, Fortran, Python, Tcl,
| and even some assembly, plus the usual grab bag of open
| source dependencies. For none of those was anything more
| sophisticated than (admittedly fairly complex) hierarchical
| Makefile description to make it come together.
|
| I'll be the first to admit that I've not seen it at all, but
| color me suspicious that it is an inherent problem of builds
| of a multi-language product, rather than unknown/unnecessary
| complexity in the architecture of the system being built.
| gavinhoward wrote:
| Recursive Make isn't the best.
| https://aegis.sourceforge.net/auug97.pdf
|
| That said, if Rig gives you nothing new, please don't even
| try it!
|
| I'd rather you wait until you _need_ it to try it; then you
| 'll be more forgiving of the bugs. :)
| jgeada wrote:
| There is something very fishy in that article: the order
| of traversal in a hierarchical make should be entirely
| expressed by dependency rules, not mere existence of a
| directory. Failure to describe dependencies is a flaw in
| the creation of the Makefile, not an inherent property of
| make itself.
|
| Specifically, this idiom: all: for
| dir in $(MODULES); do \ (cd $$dir; ${MAKE} all);
| \ done
|
| should be banished from everyone's Makefiles. make isn't
| a just way of cobbling together some collection arbitrary
| collection of scripts. Each of those modules should be a
| target with dependencies. The rule to make that module is
| the recursive make. But the order is determined by the
| explicit dependencies, not whatever random order glob
| gives you. ie, what you should have is:
| module1: (cd $module1_dir; make)
| module2: module1 (cd $module2_dir; make)
|
| etc etc etc
|
| Every single time in my career that I've had someone
| complain about a flaw in make, what I discover is a
| significant flaw in their description of the make rules.
| If it is not behaving correctly, you did not fully
| describe the DAG.
| jcranmer wrote:
| > Every single time in my career that I've had someone
| complain about a flaw in make, what I discover is a
| significant flaw in their description of the make rules.
| If it is not behaving correctly, you did not fully
| describe the DAG.
|
| You can look at the dependency graph in two ways. The
| first is, in order to build X, you first need to build Y
| and Z. These kinds of dependencies tend to be relatively
| easy to figure out. The second way is to ask, if Y
| changes, what needs to be rebuilt--and these kind of
| rebuild dependencies aren't going to be correct unless
| they're autogenerated, and getting autogenerated
| dependencies done correctly is challenging. (This
| basically reaches the point in the article that any
| system that doesn't support dynamic dependencies ends up
| hacking its way to support for dynamic dependencies).
|
| The other key point I will make is that, while it can be
| possible to describe the DAG you want with make, it tends
| to fight you every step in the way. If almost everyone is
| complaining about how hard it is to get the DAG correct
| in make, then maybe the problem isn't that everyone can't
| figure out how to fully describe the DAG, it's that make
| isn't a good option for specifying the DAG.
| rcxdude wrote:
| I suspect that you can describe pretty muich all builds without
| turing completeness, but avoiding it is rarely particularly
| helpful. What you need is a language that has a reasonable
| amount of expressiveness to describe modules and how they fit
| together, including whatever decisions you need to allow for a)
| build configurations and b) dealing with platform variations.
| It's a lot easier to express this in a turing complete language
| than a non-turing complete language, and it's hard to make a
| non-turing-complete expressive enough without accidentally
| making it turing complete.
|
| The reason make is not particularly great for writing a build
| system in is the language is pretty arcane, and it has annoying
| restrictions which make certain things particularly difficult
| to deal with (Spaces in filenames!). The fact that it's not
| turing complete is not really the key reason in my mind to use
| something like cmake instead. But, if you were to try to turn
| make into something that's more flexible and easier to use,
| you'd be making a lot harder on yourself if you still wanted to
| avoid turing completeness.
| TheNewAndy wrote:
| I have made a (internal only unfortunately) meta build system
| for my work. Instead of being Turing complete is uses Horn
| Clauses as the primary mode of logic. This means that the
| problem of determining "is this compiler flag set" or "is
| this file used in this build" is always solvable in
| polynomial time.
|
| It has been in use for over ten years by multiple teams
| around the world for all sorts of whacky toolchaims and
| projects.
|
| I don't believe anyone has missed Turing completeness.
|
| Despite all that, I agree that if you need a metabuild system
| then things are probably too complicated. I guess the main
| reason I would argue for one is when building a cross
| platform thing, it is nice if developers using toolchain X
| don't need to modify build files for all the toolchains that
| aren't X - making Linux developers have to fight Visual
| Studio just to add a new file to a build is not ideal. But in
| principle - I still agree that the resulting question of "how
| do I build this thing from scratch?" Should probably be
| answerable in 2 sentences (e.g. "Compile and link all the C
| files with the top level directory in the include path. Don't
| compile files that have an operating system in their name
| unless you are building for that os")
| jcranmer wrote:
| Ideally, a build system should be just a simple, declarative
| thing: here is the list of the files that need to be compiled,
| go do it. But the simple declarative tends to fail when you
| start getting to complex logic.
|
| Things start out with a simple "oh, we need to do completely
| different things on different OSes. Compile only these files on
| Linux and these on Windows." Then, oops, we also have a feature
| that requires a long build time that half our developers don't
| want to build (also it's unstable and has a nasty dependency or
| something), so only build this files when this feature is
| enabled.
|
| Then something gets complex and you now have part of your
| source code being autogenerated from some tool that needs to be
| built as part of the build. And the autogenerator needs to use
| the support library. And someone complains about the speed of
| the debug build, so the autogenerator needs to be built with
| optimizations enabled even when the rest of the program is
| disabled. And then cross-compilation is now more of a
| headache...
|
| I haven't even touched on so many other possible complexities
| in the build system (autodownloading dependencies is another
| fun one). The point is that the idealized, simple, declarative
| build system only cuts it like 60% of the time (even that
| requires some level of conditional compilation which make can
| do but not nicely). At the upper end, you're going to need some
| complex logic to handle complex cases, and not providing some
| form of general-complexity features to achieve those cases is
| going to result in people trying to emulate them using existing
| features--ironically probably making it _more_ complex as a
| result.
|
| Giving users the ability to have Turing-completeness in their
| build system doesn't necessarily mean the buildsystem has to
| devolve into incomprehensiblity. I've come to like Mozilla's
| meta-build system in this regard, for example, this relatively
| complex build file: https://searchfox.org/mozilla-
| central/source/browser/app/moz...
| pixl97 wrote:
| >so many other possible complexities
|
| This right here. When working with large enterprise clients
| I'm seeing insanely complex build chains.
|
| Artifactories hosting all the built/allowable
| dependencies/binaries allowed in the customers software.
| Systems that check the added dependencies and verify any
| newly added dependencies match the allowed licenses for the
| application being built. Static checkers of different types.
| Open source analysis, dll version security analysis, SBOM,
| and more. Checks to see if enough build system servers are up
| and running at the time or if the job needs delayed.
| Seemingly continuous integration has swept this type of
| customer and the complexity of the build system to allow this
| has exploded.
| trueismywork wrote:
| Build systems do more than just build files. They find and run
| tests, analyze test results, analyze coverage. Conditionally
| generate code based on whether a feature is enabled or disable.
| Run "find dependency" logic. Read pkgconfig files.
| throwway120385 wrote:
| What if my product is not an executable, but a root filesystem?
| taeric wrote:
| Wasn't a large part of the complexity introduced by autoconf from
| how loosely prescribed UNIX systems were in where things were
| installed and how they were implemented? Feels that we are much
| more homogenous now than we have ever been in the past. Such that
| it is not that surprising to find a lot of complexity in the past
| is not as necessary today.
| JohnFen wrote:
| This is my experience. Autoconf used to be really important,
| but I haven't actually needed it for years now.
| Darkstryder wrote:
| > Say you add a new source file. Wouldn't it be great if your
| build system just picked it up?
|
| > Alas, it cannot; the list of stuff to build is passed from the
| meta build system to the build system, usually by fiat.
|
| It can, actually. At least for make. Just use a wildcard in uour
| rules:
| https://www.gnu.org/software/make/manual/make.html#Wildcards
|
| I highly recommend going through the make documentation at least
| once in your career. Per the lindy effect, as it has been around
| for 40 years, it has a decent chance of sticking around for
| another 40.
| gavinhoward wrote:
| Author here.
|
| I mentioned suffix rules and pattern rules.
|
| Unfortunately, meta build systems don't like to generate rules
| like that.
| kjkjadksj wrote:
| If the lindy effect were real we'd all be romans still
| shrimp_emoji wrote:
| It worked on these guys:
| https://en.wikipedia.org/wiki/Sultanate_of_Rum
| layer8 wrote:
| Regarding dynamic dependencies, the author doesn't seem to be
| familiar with the build systems used for Java (Maven and Gradle),
| which in my understanding use static dependencies (static build
| graph).
| gavinhoward wrote:
| Author here.
|
| I have worked with them long enough ago I can't remember.
|
| Do they require the build files to list all dependencies?
| layer8 wrote:
| Yes, that's how they work. To be precise, the direct
| dependencies are listed, and each dependency (after retrieval
| from a package repository, or locally cached) provides a
| static list of its own direct dependencies.
| gavinhoward wrote:
| Okay.
|
| The point of dymanic dependencies is that Rig could run
| some code to figure out what the main function imports,
| transitively, and use that info to build the entire
| project.
|
| All _without_ listing dependencies in the build file.
|
| A good API might look like this:
| build_java_package("com.yzena.rig");
|
| So when you add a new package and use it, you do not have
| to change the build files; Rig would pick it up
| automatically.
| layer8 wrote:
| In Java, there is no 1:1 relation between package imports
| and jars. A jar (dependency) usually contains many
| packages (namespaces), and of course the same packages
| can be contained in multiple (usually mutually
| incompatible) dependencies. (Java imports are often more
| fine-grained than, say, C/C++ header files.) Dependency
| selection therefore requires a conscious decision.
|
| IDEs provide tooling to suggest matching dependencies to
| be added to the build file based on the names of
| unresolved imports. However, the much more common
| direction is to first decide which libraries
| (dependencies) you want to use and add those to your
| build file, which then allows the IDE to suggest imports
| when you reference some unresolved type in your source
| code. So the dependency declarations in the build file
| generally come first, before you start writing your
| corresponding code. (This can be likened to specifying
| coarse-grained includes in your project build file, from
| which fine-grained sub-includes are then selected in each
| source file.)
|
| The conscious selection of dependencies is a good thing,
| IMO. Dependencies also don't change frequently, so there
| isn't much work to be saved by determining them
| automatically (except for transitive dependencies upon
| version bumps, which _are_ determined automatically by
| the build system). Moreover, it's not only the
| dependencies as such that are consciously selected, but
| also their respective version numbers. This prevents
| unexpected breaking changes.
| somewhereoutth wrote:
| A => B
|
| You'd think that would be hard for even engineers to screw up,
| but here we are.
| barryrandall wrote:
| The engineers wanted to write a thin encabulation module in
| K++, but management insisted that the engineering team use the
| industry standard Encabulate.moo library, and the Cows on Crack
| framework doesn't have a stable ABI.
| mdaniel wrote:
| now all we need is a Markov Chain trained to produce this,
| publish YT shorts of them, and retire
| https://www.404media.co/inside-the-world-of-tiktok-
| spammers-...
| barryrandall wrote:
| I'm pretty sure that party is over. "Make Money Online with
| ${x}" is what you pivot to when the return on ${x} drops
| below the going rate for "Make Money Online" courses.
| somewhereoutth wrote:
| Though it would be nice to blame this on Management (who are
| often the root cause of all sorts of screw ups) - managers
| generally have no idea of, let alone interest, in build
| systems. We did this to ourselves.
| KWxIUElW8Xt0tD9 wrote:
| high logical intelligence doesn't imply common sense
| nottorp wrote:
| How about a build system to manage the existing build systems?
|
| Perhap just one layer of meta isn't enough...
| choeger wrote:
| Good post. Good analysis. One problem:
|
| If turing-completeness is necessary, _why_ should there even
| exist a build system independent from a programming language now?
|
| Isn't the obvious solution to script your language's default
| build system in precisely the language you're trying to build
| from?
|
| So when you build Rig, build it in C++ and make C++ your
| scripting ... Ok, that left a bad taste in my mouth.
|
| Seriously, though. Build systems as the author describes them are
| going to compete against the project managers that come with
| languages (Cargo, Go). The only real use case for Rig would be
| legacy languages that don't come with language-central packaging
| repositories, i.e., Fortran, C++, C.
|
| While I applaud the author for the attempt to improve this
| ecosystem, I really hope it will become less relevant some day.
| gavinhoward wrote:
| > If turing-completeness is necessary, why should there even
| exist a build system independent from a programming language
| now?
|
| This is a great question that I will make sure to answer in my
| Turing-completeness post in a week.
|
| But to answer now, it's because of multi-project builds, where
| projects can be in multiple languages.
|
| If you have build systems that are not independent from their
| language, how do you get them to play nice?
|
| I designed Rig to be able to act as a babysitter and make them
| play nice.
|
| If I play my cards right, Rig will be the reason other build
| systems become irrelevant. :)
| codethatwerks wrote:
| What about a single API for builds, implemented in multiple
| general purpose languages.
|
| End users get to pick a familiar language to them.
|
| Similar to how Pulumi is available in multiple languages.
|
| Then code both builds and meta build concerns as you would
| code anything else.
| aappleby wrote:
| Not if Hancho beats you to it. ;)
|
| https://github.com/aappleby/Hancho
|
| In all seriousness, I think both our projects are aiming in
| the right direction, mine is just more focused on minimalism
| at the cost of having extra-pointy corner cases.
| JohnFen wrote:
| > why should there even exist a build system independent from a
| programming language now?
|
| I want my build system to be independent of the programming
| language for two reasons:
|
| 1) What gavinhoward said: I regularly use several different
| languages, sometimes in the same project, sometimes not. But in
| all cases, I want to use the same build system in order to
| minimize my environmental complexity. Much like I want to use
| the same IDE regardless of language.
|
| 2) I don't want to risk version dependencies. I want to be able
| to use the same build system even when I'm compiling something
| using an old version of a compiler, and I don't want to be
| forced into a build system upgrade just because I want to
| upgrade one of my compilers.
|
| For me, the separation of the two roles is very important. If
| they were tied together, it would mean that I'd have to add
| complexity elsewhere to cover my use cases. Languages that
| include their own package managers (like Rust, Go, etc.) give
| me headaches.
| zdragnar wrote:
| I was once on a project that inadvertently used ruby in some
| way for everything. Rails Bundler
| Vagrant Chef Opsworks configuration
| JavaScript bundling Generating PDFs Probably
| other things
|
| The system was basically frozen in time because too many cross-
| dependencies couldn't update in sync to something newer (the
| opsworks cookbook in chef were the worst offenders iirc).
|
| It was so bad when I started that I couldn't even get the
| system up and running locally. I ended up tossing hundreds of
| lines of random ruby code and with a few lines of docker
| commands had it up and running.
|
| Since then I've preferred keeping things nicely not dependent
| on each other whenever possible.
| jeffbee wrote:
| What can be done about libraries on which you want to depend, but
| that have too many opinions about how they should be built? For
| example I use bazel and one large library I use forced -Werror
| into the C compiler invocation, but with my C compiler it has
| warnings, so it can't actually be built unless I patch out that
| line of their bazel configs. Another library only builds
| correctly when using clang if the clang toolchain is named
| literally "clang" and not, for example, "clang-17". Consequently
| I have to adopt their convention or maintain a patch against
| their project. There are many, many unsolved rough edges in bazel
| along these lines.
| gavinhoward wrote:
| Author here.
|
| Rig will have multi-project builds. It also has a concept
| called "context stacks".
|
| The idea would be that you would have a "compiler option"
| context stack, and you would push `-Wno-bad-warning`, then
| inside the context, build that library.
|
| Then the library would add `-Werror`, but it would also have
| `-Wno-bad-warning`, so no problem.
|
| But multi-project builds won't exist in the first release.
| fanf2 wrote:
| Before autoconf there was metaconfig, which was Larry Wall's
| portability support for rn and perl.
| KWxIUElW8Xt0tD9 wrote:
| Bell Labs had a very nice portability framework also -- there
| was a Cygwin-like product called U/WIN that made extensive use
| of it -- might find it associated with ksh93 maybe.
|
| Speaking of Bell Labs, there are some great ideas for a build
| tool in the Bell Labs "nmake" tool - I believe it was
| originally created to speed up ESS5 switch builds, which took 3
| days before make. It has programmable header file scanning for
| example, a shell coprocess to avoid all the fork/exec of
| standard make, build state files, lots of other things I have
| forgotten.
| bluGill wrote:
| This is missing something: what does my ide do? I want a key
| combo or menu select for all the things I do. Build and run tests
| in address sanitizer. Build, deploy to my embedded target then
| run in the debugger with these options.
| gavinhoward wrote:
| Author here.
|
| Every IDE is different, unfortunately.
|
| Rig will have to adapt to each IDE, but it will generate files
| with enough info to call into Rig, which will do the job.
| max-privatevoid wrote:
| Nix with dynamic derivations (RFC92) could potentially beat this
| curse.
|
| https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...
| aappleby wrote:
| I recently wrote Hancho (https://github.com/aappleby/Hancho) that
| is designed to be fully dynamic; I thiiiink it would fit most of
| the desired criteria here (aside from cloud caching) but I'm not
| positive.
|
| I'll have to do a bit more investigating into what this article
| means by "dynamic dependencies".
| tom_ wrote:
| > Say you add a new source file. Wouldn't it be great if your
| build system just picked it up?
|
| I have never enjoyed the experience of working with tools that
| did this. How does it know which files I actually want, and which
| files are there because I (or somebody else) got something wrong?
| The answer, of course: it doesn't.
|
| I moan a lot about having to do the computer's work for it, but I
| think in this case I am happy to put up with some minor drudgery
| (adding a file name to a list, once) in exchange for the
| computer's boundless ability to then patiently follow my
| instructions without trying to second-guess me on every single
| build.
| soulbadguy wrote:
| > So there are five features of build systems that you want, and
| the harsh truth is that meta build systems cannot have some of
| them! Some of these things are only available on end-to-end build
| systems.
|
| Did I miss something or did the authors actually didn't explain
| what prevents meta build systems to implement any of those
| features.
|
| Looking at the current state of cmake it seems to have everything
| mentioned here. Including dependencies and header parsing in c++
___________________________________________________________________
(page generated 2024-03-19 23:01 UTC)