[HN Gopher] Unity builds lurked into the Firefox Build System
___________________________________________________________________
Unity builds lurked into the Firefox Build System
Author : sylvestre
Score : 56 points
Date : 2023-05-05 05:57 UTC (1 days ago)
(HTM) web link (serge-sans-paille.github.io)
(TXT) w3m dump (serge-sans-paille.github.io)
| [deleted]
| Dwedit wrote:
| Note that this is not referring to the Game Engine Unity. It's
| just referring to #including .cpp files.
| jb1991 wrote:
| Indeed, the title almost makes no sense grammatically in its
| current form, a consequence of the word "how" being removed. It
| would be obvious it was not the game engine when the word
| "unity" appeared as the second, lowercases word.
| StellarScience wrote:
| We leverage many third party C++ libraries with complex
| templates, concepts, and constexpr expressions that seem to
| require lots of CPU to compile. We've found unity builds to be
| almost 3X faster, so we make it the default for both developer
| and CI jobs.
|
| But we keep a separate CI job that checks the non-unity build, so
| developers have to add the right #include statements and can't
| accidentally reference file-scoped functions from other files.
| While working on a given library or project, developers often
| disable the unity build for just that project to reduce
| incremental build times. It seems to offer the benefits of both
| approaches.
|
| Precompiled headers don't give nearly the same speedup. We're
| excited for C++ modules of course, but we're trying to temper any
| expectations that modules will improve build speed.
| 10000truths wrote:
| With the advent of LTO, unity builds are mostly a band-aid for
| poor management of header files. The Linux kernel project was
| able to net a ~40% reduction in compilation CPU-time just by
| pruning the contents of some key header files [1].
|
| It really boils down to two rules:
|
| 1. Don't declare anything in header files that is only used in
| one compilation unit. Internal structs and functions should be
| declared and defined in source files, and internal linkage used
| wherever possible. gcc and clang's -fvisibility=hidden is useful
| here.
|
| 2. The more frequently a header file is included (whether
| transitively or directly), the more it should be split up. If a
| "common" or "utility" header file is included in 10000 source
| files, then any struct, function, etc. that you add to that file
| will have to be parsed 10000 times by the compiler every time you
| build from scratch, even if only 10 source files actually use the
| struct/function that you added. gcc and clang's -H flag is useful
| here.
|
| [1] https://lore.kernel.org/lkml/YdIfz+LMewetSaEB@gmail.com/
| pm215 wrote:
| I think "just" is perhaps not the right word for something that
| took a senior dev over a year and more than 2000 commits just
| to get to an RFC patchset that doesn't compile for all
| architectures... Tremendous work, but it clearly wasn't easy or
| a matter of "follow these simple rules".
| thinkling wrote:
| I've been out of C/C++ development for a long time but seem to
| remember that precompiled headers were a thing back in the day.
| That approach didn't have the name space issues pointed out here.
| Why are precompiled headers not used anymore?
| bgmeister wrote:
| They are still used in some places. But they have some
| downsides:
|
| Precompiled headers don't play nicely with distributed
| compilation or shared build caches (which are perhaps the
| fastest way to build large C++ codebases). So while they can
| work well for local builds, they exclude the use of (IMO)
| better build-time optimisations.
|
| They also require maintenance over time- if you precompile a
| bad set of headers it can make your compile times worse.
| pjmlp wrote:
| As far as I am aware, they never work that great on UNIX
| compilers, as no big effort was ever spent improving them.
|
| About 20 years ago, on UNIX workloads we used to speed the
| compilation via ClearMake, a kind of distributed version of
| code cache that would plug into the compilers, however it has
| part of ClearCase SCM product.
|
| On Windows, with Microsoft and Borland (nowadays Embarcadero),
| they work quite alright.
|
| Also, modules will fix that, as per VC++ reports, importing the
| whole standard library (import std, as per C++23) takes a
| fraction of only including iostream.
| maccard wrote:
| They're very much alive and well on MSVC. Our work projects use
| both unity builds _and_ precompiled headers.
| dundarious wrote:
| Why can't static analyzers analyze the main cpp that #include-s
| the actual code? I don't understand that point.
|
| And what were the resulting affects on build times?
| monocasa wrote:
| They can. In addition to that, they get confused by a .cpp that
| isn't the top level file of a compilation unit.
| kccqzy wrote:
| To avoid some of these issues, it can be helpful in a project to
| require that all files including header files must be compilable
| on their own. Doesn't get rid of all the problems (you can still
| depend on transitive includes without explicitly including them)
| but enforces a minimal amount of code hygiene.
| omoikane wrote:
| > This generally leads to faster compilation time in part because
| it aggregates the cost of parsing the same headers over and over.
|
| But this also reduces the opportunity to parallelize compilation
| across multiple files because they have been concatenated into
| fewer build units, and each unit now requires more memory to deal
| with the non-header parts. For some build systems and
| repositories, this actually increases build time.
| simplotek wrote:
| > But this also reduces the opportunity to parallelize
| compilation across multiple files because they have been
| concatenated into fewer build units (...)
|
| Irrelevant. There is always significant overhead in handling
| multiple translation units, and unity builds simply eliminate
| that overhead.
|
| > and each unit now requires more memory to deal with the non-
| header parts.
|
| And that's perfectly ok. You can control how large unity builds
| are at the component level.
|
| > For some build systems and repositories, this actually
| increases build time.
|
| You're creating hypothetical problems where there are none.
|
| In the meantime, you're completely missing the main risk of
| unity builds: increasing the risk of introducing problems
| associated with internal linkage.
| tomjakubowski wrote:
| unity builds do often have worse performance than separate
| compilation for "incremental rebuilds" during development.
| that all depends on how the code is split up and how bad of a
| factor linking is.
|
| as in the article, it's best to support both
| stephc_int13 wrote:
| On very large projects you can always cut them into several
| libraries, and compile them on different cores. Quite easy to
| do in practice.
| cpeterso wrote:
| I believe Firefox builds only unify files within the same
| directory and a maximum of a ~dozen cpp files per unit. So
| there are still plenty of build parallelism across
| directories.
| vinyl7 wrote:
| Compilation units are a relic of a time where computers only had
| a few KB of memory. At this point computers are fast enough and
| have enough memory to compile the whole thing in one go faster
| than whatever gains doing change detection and linking will have.
| smabie wrote:
| Why do clean builds of my code take like 30m then?
| dagmx wrote:
| While everyone else is (rightfully) correcting you, I am
| curious what sort of codebases you're working with?
|
| Are you working on large compiled software? Any game, rendering
| engine or large application benefits from compilation units in
| my experience.
|
| Some of my libraries that I work with take upwards of an hour
| for a fresh compile. Having sane compilation units cuts down
| subsequent iteration to minutes instead.
| regnerba wrote:
| Hahaha tell that to my Unreal Engine build times.
|
| A brand new AMD Epyc, 64 core machine, will take over an hour
| to compile. Good times.
| dagmx wrote:
| I'd really like to see a comparison someday between Epics
| weird C# based build system and something like CMake+Ninja.
|
| I suspect there's compilation optimizations to be made, but I
| don't think it would save more than 30% here and there.
| maccard wrote:
| > I suspect there's compilation optimizations to be made
|
| There definitely are. I've spent a lot of time with UBT,
| and a "reasonable" amount of time with cmake and friends.
| UBT isn't quite the same as CMake + Ninja. UBT does
| "adaptive" unity builds, globbing, and a couple of other
| things.
|
| > but I don't think it would save more than 30% here and
| there.
|
| Agreed. The clean build with UBT is painfully slow compared
| to Cmake + Ninja, but the full builds themselves are
| _pretty_ good, and I 'd bet that there's probably less low
| hanging fruit there.
|
| I did a good chunk of work on improving compile times in
| Unreal, and there is definitely just low hanging fruit in
| the engine for improving compile times. Some changes to how
| UHT works around forward declares would also make a
| significant difference too.
| regnerba wrote:
| I would as well! It's honestly a bit beyond me, the Unreal
| build tools run deep, so I imagine it would take some
| effort.
| vitno wrote:
| This, is just deeply untrue. Do you really think everybody
| working on compilers and linkers are deeply ignorant? I can
| easily saturate my 64 GB RAM home setup during a compile.
| dblohm7 wrote:
| Yeah, no. To this day Firefox developers building Gecko need a
| beefy desktop machine to be able to do it in a reasonable
| amount of time. I could do a clean build in 6 minutes with a
| ThreadRipper whose cores were all pegged, but forget doing the
| same in under an hour on a laptop.
|
| And that was with unified builds enabled.
| Y_Y wrote:
| This used to be mandatory for nvcc/CUDA, if you had multiple
| source files (not just headers) you had to #include all of them
| in your main file. It made me very uncomfortable.
| robalni wrote:
| I always use unity builds for all my projects now. That combined
| with using tcc as compiler (for C code) makes builds really fast.
| Another nice feature of unity builds is that I don't need to
| declare functions twice and keep the declarations synced. It's
| also nice to only have one place to find information about a
| function; people often put comments in header files that you can
| miss if you go to the definition.
|
| All of those things combined make C programming more enjoyable.
| simplotek wrote:
| > Another nice feature of unity builds is that I don't need to
| declare functions twice and keep the declarations synced.
|
| What exactly leads you to have multiple declarations in sync,
| and thus creating the to "keep [multiple] declarations synced"?
| robalni wrote:
| I mean if you use multiple translation units and header
| files, you need to have a copy of the function declaration in
| that header file to be able to call it from other translation
| units.
| andybak wrote:
| "Lurked into"?
|
| You can lurk but surely you can't lurk _into_ something?
| bragr wrote:
| I assume the author is not a native speaker based on some of
| the odd grammar and phrasing in the post. It doesn't really
| detract from the work
|
| Edit: They appear to be french:
| http://serge.liyun.free.fr/serge/
| okeuro49 wrote:
| Probably meant "crept into".
| https://dictionary.cambridge.org/dictionary/english/creep-
| in...
| loufe wrote:
| The word "lurk" doesn't exactly exist in French. "se roder"
| fits in some cases, but another translation "se cacher" (to
| hide) fits others. I'd write "X crept into Y" as "X s'est
| glisse dans Y", but that has a connatation moreso as an
| accidental short-term mistake. I don't know how I'd express
| the idea concisely in French. Also hard to tell exactly how
| he wanted to convey it, something between "crept", "were
| hidden", "were lurking" probably? As I've discovered the hard
| way, there is not always an analogous term for the same
| fundamental idea/concept between two given languages;
| mastering the nuances of these differences is important for
| proper fluency. I probably make errors like this frenquently
| writing/speaking French.
| [deleted]
| tyleo wrote:
| Lots of game studios use Unity builds like this. It saves a
| massive amount of time. Last I heard it also improves
| Incredibuild performance which is another popular tool for
| decreasing build timed.
| zX41ZdbW wrote:
| Tried Unity builds recently for ClickHouse, but without success:
| https://github.com/ClickHouse/ClickHouse/pull/18952#issuecom...
| stephc_int13 wrote:
| I used Unity builds for my projects basically forever, at some
| point I discovered the practice had a name and some debates
| around it.
|
| It is a simple thing to do, and the gains are substantial, faster
| and simpler, less maintenance, especially across different
| platforms.
|
| For big projects I simply cut them into several libraries.
|
| I've seen some incredulous reactions, mostly from young coders,
| and I know that makefiles _should_ be faster, but in practice I
| never found that to be true.
| mastax wrote:
| Interesting. I've been aware of this technique for years because
| of the SQLite Amalgamation, but that was always sold as a way to
| simplify distribution and perhaps improve performance of the
| binary. I hadn't considered it as a build speed optimization,
| though that seems somewhat obvious in hindsight.
| simplotek wrote:
| > I hadn't considered it as a build speed optimization, though
| that seems somewhat obvious in hindsight.
|
| Some build systems like cmake already support unity builds, as
| this is a popular strategy to speed up builds.
|
| Nevertheless, if speed is the main concern them it's preferable
| to just use a build cache like ccache, and modularize a project
| appropriately.
| maccard wrote:
| Why not both?
|
| Also, does ccache work with MSVC?
| cpeterso wrote:
| Another benefit not mentioned is optimization. The compiler may
| be able to inline more function calls when function definitions
| and callers are in the same unified compilation unit.
| nine_k wrote:
| It's another reminder how <expletives omitted> C++ is, but for a
| long time nothing better existed.
| firstlink wrote:
| The compilation-unit-per-file model (and in fact the whole
| concept of linking) are a legacy incremental build solution for C
| which somehow metastasized into fundamental requirements of
| building software on current OSes. It is an atrocity and should
| be disavowed by all developers.
| Scubabear68 wrote:
| Headers and C style macros are probably the most unfortunate
| aspects of C (and by extension, C++).
|
| So many hacks in compilers to try to work around this. A shame
| there is no language level fix for this nonsense.
|
| Really wish there could be a C++--- that would improve on C in
| areas like this, and avoid all the incredible nonsense of C++.
| And no, not Rust or Go.
| pphysch wrote:
| Headers (in new code) will hopefully become optional due to
| modules. That would be such a big boost to the language.
|
| C Macros are pretty much considered code smell in C++, right?
| zyedidia wrote:
| I think Hare (https://harelang.org/) might fit the bill: it
| retains the minimalism and simplicity of C, but fixes issues
| like this (and others). Unfortunately I don't think it's ready
| for real use yet, but I am keeping an eye on it.
| pxeger1 wrote:
| Have you tried Zig? I think it fits those criteria, and is
| known for its good build system, although AIUI it is quite a
| large language compared to C
| leni536 wrote:
| Unity builds mean that you can no longer use internal linkage
| safely anymore, and that's not something I like to give up. It
| forces the codebase to follow a certain style that I don't like.
| Hopefully modules will give the advantage of unity builds without
| this downside.
___________________________________________________________________
(page generated 2023-05-06 23:01 UTC)