[HN Gopher] Cutting down Rust compile times from 30 to 2 minutes...
___________________________________________________________________
Cutting down Rust compile times from 30 to 2 minutes with one
thousand crates
Author : Qadriq
Score : 123 points
Date : 2025-04-17 11:22 UTC (11 hours ago)
(HTM) web link (www.feldera.com)
(TXT) w3m dump (www.feldera.com)
| HippoBaro wrote:
| Eminently pragmatic solution -- I like it. In Rust, a crate is a
| compilation unit, and the compiler has limited parallelism
| opportunities, especially since rustc offloads much of the work
| to LLVM, which is largely single-threaded.
|
| It's not surprising they didn't see a linear speedup from
| splitting into so many crates. The compiler now produces a large
| number of intermediate object files that must be read back and
| linked into the final binary. On top of that, rustc caches a
| significant amount of semantic information -- lifetimes, trait
| resolutions, type inference -- much of which now has to be
| recomputed for each crate, including dependencies. That
| introduces a lot of redundant work.
|
| I also would expect this to hurt runtime performance as it likely
| reduces inlining opportunities (unless LTO is really good now?)
| JJJollyjim wrote:
| They mention that compiling one crate at a time (-j1) doesnt
| give the 7x slowdown, which rules out the object file/caching-
| in-rustc theories... I think the only explanation is the rustcs
| are sharing limited L3 cache.
| lsuresh wrote:
| The L3 cache angle is one of our hypotheses too. But it
| doesn't seem like we can do much about it.
| Crisco wrote:
| Are there any performance implications for the final binary
| because you're splitting it up into thousands of crates?
| epage wrote:
| Loss of inlining
| dgacmu wrote:
| Loss of automatic inlining of non-generics without LTO, to be
| a little pedantic.
|
| Functions marked #[inline] can still be handled across
| crates.
|
| LTO can inline across crates but, of course, at a substantial
| compile time cost.
| hu3 wrote:
| > We're using rustc v1.83, and despite having a 64-core machine
| with 128 threads, Rust barely puts any of them to work.
|
| > That's right -- 1,106 crates! Sounds excessive? Maybe. But in
| the end this is what makes rustc much more effective.
|
| > What used to take 30-45 minutes now compiles in under 3
| minutes.
|
| I wonder if this kind of trick can be implemented in rustc itself
| in a more automated fashion to benefit more projects.
| zwnow wrote:
| 1106 crates? Are they sure this is not a Javascript project?
| faitswulff wrote:
| They compile their customers' SQL to Rust code. Hence the
| preponderance of crates. It's a somewhat unique scenario.
| lsuresh wrote:
| That's correct. These aren't external dependencies but a
| dataflow graph being split into crates.
| pcwalton wrote:
| > I wonder if this kind of trick can be implemented in rustc
| itself in a more automated fashion to benefit more projects.
|
| It partially is, with codegen units. The problem is that you
| can't generally do that until codegen time, because of circular
| dependencies.
| kzrdude wrote:
| I wonder if it would make a difference, at the starting point, to
| use fully optimized compile for dependencies but only opt-level=1
| for the main crate?
| xrisk wrote:
| Not sure about the nature of the generated code, but wouldn't
| doing the equivalent of dynamic linking mitigate this problem?
| HKH2 wrote:
| How can you get around Rust's lack of a stable ABI?
| amelius wrote:
| > despite having a 64-core machine with 128 threads, Rust barely
| puts any of them to work.
|
| Rust is fast in theory, but if in practice they can't even get
| their compiler to squeeze any juice from the CPU, then what's the
| value of that language from a software engineering viewpoint?
| swiftcoder wrote:
| Compilation is inherently pretty hard to parallelise, and
| various design decisions around how Rust modules/creates work
| make it even harder (as demonstrated by achieving greater
| parallelism here with many smaller crates). There's no
| particular reflection on the performance of rust code here,
| it's really a design/algorithms problem.
| amelius wrote:
| > Compilation is inherently pretty hard to parallelise
|
| I don't agree. In a large project there is going to be a lot
| of stuff that can be compiled in parallel without problems.
| swiftcoder wrote:
| And Rust's module design makes this significantly more
| complicated than in superficially-similar languages. That's
| why splitting it into modules brings drastic improvements -
| you are effectively giving the compiler clear boundaries
| across which it doesn't need to propagate as much
| information.
| steveklabnik wrote:
| Splitting it into _crates_ , not modules.
| Ygg2 wrote:
| C++ is fast in theory, but if they can't get C++ compiler
| (LLVM) to squeeze any juice from CPU, then what's the value of
| that language from a software engineering viewpoint.
|
| Hopefully, you can see why this reasoning is a problem. The
| main stumbling point being compilation speed != runtime speed.
| amelius wrote:
| I have been using make's -j flag to compile C++ projects with
| great success.
|
| The main point being, Rust focuses on the wrong type of
| optimizations.
| deciduously wrote:
| Which is exactly what this project is now able to do. Your
| parallel make jobs don't help if you have one gigantic
| compilation unit, as they originally did.
| wahern wrote:
| LTO can be parallelized, both implicitly from the
| Makefile, and also within the compiler. GCC's -flto
| itself takes an optional argument to control the number
| of parallel threads/jobs. See also the -flto-partition
| option for selecting symbol partitioning strategies.
|
| Rust/Cargo does this automagically, except the only
| control you have are the crate and module boundaries. The
| analogous approach for C is to (optionally) manually
| group compilation units into a smaller set of object
| files in your Makefile, LTO'ing each object file in
| parallel (make -j), and then (optionally) telling the
| compiler to partition and parallelize a second time on
| the backend. Which is what Rust does, basically, IIUC--a
| crate is nominally the LTO codegen unit, except to speed
| up compilation Rust has heuristics for partitioning
| crates internally for parallel LTO.
| dathinab wrote:
| codegen units is quite the same as the `-j` flag
|
| cargo has the `-j` flag and defaults it to #cpus (logical
| cpus), so it's by default using the most times most optimal
| choice there
|
| And this will parallelize the compilation of all "jobs",
| roughly like with make. Where a job is normally
| (oversimplified) compiling one code unit into one object
| file (.o).
|
| And cargo does that too.
|
| The problem is that where rust and C/C++ (and I think D)
| etc. set code unit boundaries differ.
|
| In rust it's per crate. In C/C++ it's (oversimplified!!)
| per .h+.c file pair.
|
| This has drawbacks and benefits. But one drawback is that
| it parallelizes less good. Hence why rust internally split
| one "semantic code unit" into multiple internal code units
| passed to LLM. So this is an additional level of
| parallelism to the -j flag.
|
| In general this works fine and if people speak about rust
| builds being slow it is very rarely related to this aspect.
| But it puts a limit onto how much code you want in a single
| crate which people sometimes overlook.
|
| But in the OP article they did run into it due to placing
| like idk 100k lines of code (with proc macors maybe _way_
| more then that) into a single crate. And then also running
| into a bug where this internal parallelization somehow
| failed.
|
| Basically imagine 100k+ line of code in a single .cpp file
| passing `-j` to the build to it will not help ;)
|
| I think one important takeaway is that it could make sense
| to crate awareness about this by emitting a warning if your
| crate becomes way to big with a link to a in-depth
| explanation. Through practically most projects either
| aren't affected or split it into crates way earlier for
| various reasons (which sometimes include build time, but
| related to caching and incremental rebuilds, not fully
| clean debug builds).
| wahern wrote:
| > Hence why rust internally split one "semantic code
| unit" into multiple internal code units passed to LLM.
|
| And the same has happened in C and C++ land, albeit in
| the opposite direction, where multiple compilation units
| can be optimized together, i.e. LTO. See, e.g., GCC's
| -flto-partition option for selecting strategies for
| partitioning symbols for LTO.
|
| Also note that you can manually partition LTO in your
| Makefile by grouping compilation units into object files
| to be individually LTO'd.
| sfink wrote:
| I also use make -j for C++ with great success. And so I am
| also stuck with having to declare my functions in a
| separate file from where they're defined, and thus when
| stepping through code with a debugger or just a good source
| viewer that can jump to callers/callees, I never see the
| comments because they're in the header not the source. Not
| to mention the problems I run into when linking with a
| library that was compiled with different options or a
| different version of the source. And bending over backwards
| to implement concurrent algorithms to get decent runtime
| performance, and debugging the inevitable bugs that follow.
|
| Rust focuses on a different set of optimizations. I'm still
| working out when I prefer the Rust set or the C++ set or
| the Python set. I want to love Rust, but when doing
| exploratory work with unfamiliar APIs, the slow recompile
| loop and need to get everything correct for each
| incremental experimental build I do to try to figure out
| how something works is pretty painful. I don't know how
| much better it gets with familiarity. Rust is very nice
| when I fully understand the problem I'm solving and the
| environment I'm working in, and I vastly prefer it to C++
| for that. But I frequently dive into unfamiliar codebases
| to make modifications.
| pjmlp wrote:
| Except the Rust ecosystem lacks the solutions we have in C++
| land to compile fast and have easy paralelisation of builds.
|
| Because C and C++ communities for historical reasons embrace
| binary libraries, and binary component frameworks like COM,
| so while in theory a full build from scratch takes similar
| time as Rust, in practice that isn't the case.
|
| Also note that D, a language as complex as C++, with three
| compilers, one of them being based on LLVM, is largely faster
| to compile than Rust while using the LLVM backend, because
| Walter Bright did the right decisions on what to focus for
| development workflow.
| Ygg2 wrote:
| > Also note that D, a language as complex as C++
|
| While D is complex, it's a different beast than Rust. I
| suspect think various checks, from lifetime to trait
| resolution, might make it more complex to parallelize than
| C++.
| steveklabnik wrote:
| Rust is already able to do more fine-grained parallel
| compilation than C or C++, at least in the codegen step.
| The "codegen units" concept doesn't work for those
| languages.
| wahern wrote:
| > The "codegen units" concept doesn't work for those
| languages.
|
| Sure it does. Or at least could, depending on what you
| mean by that term. See, e.g., GCC's -flto-partition
| option, which supports various partitioning strategies
| for parallel LTO codegen.
| steveklabnik wrote:
| LTO is distinct, and happens after what codegen units
| does.
|
| That said, it is a more fine grained parallelism, for
| sure. Rust does LTO as well as codegen-units.
|
| Really, as you gesture towards, on some level, this is
| all semantics: our linkers are also basically compilers
| too, at this point.
| dathinab wrote:
| > have easy paralelisation of builds.
|
| except the company had a straight forward solution to the
| problem
|
| and it's not always possible in C++ land either
|
| and COM isn't part of C/C++ but a microsoft specific
| extension which solves very different issues as it's for
| cross application communication while we here have compile
| time perf issues inside of a single library
|
| > binary libraries
|
| I'm not sure if you mean dynamic linking or binary objects
| but there is the thing:
|
| - dynamic linking isn't the issue here as the issue has
| nothing to do with re-compilation and similar (where
| dynamic linking can help),
|
| - binary object files on the other hand are also something
| rust has and uses, it just sets the boundaries in different
| places (crate instead of file) which makes development
| easier, can lead to better runtime performance etc. It just
| has the drawback that you sometimes have to split things
| into multiple crates.
|
| > a language as complex as
|
| how complex a language is to write has not too much to do
| with how complex it is to split a single code unit into
| multiple parts for parallel compilation. C, C++ and D
| mainly side steps this by making each file a compilation
| unit, while in rust it's each crate. But that isn't
| fundamentally better or worse, it's trade offs. It's trade
| offs.
|
| > because Walter Bright did the right decisions on what to
| focus for development workflow.
|
| and so did rust, just with different priorities and trade
| offs
|
| and given that D is mostly irrelevant and rust increasingly
| more successful maybe it was pursing the more important
| priorities
|
| Also the OP case is about release builds i.e. not the
| normal dev loop
|
| neither does the splitting affect dev experience as it's
| all auto generated code
|
| and if projects which aren't auto generated it's very
| normal to split out libraries etc. anyway, and weather you
| split them into their own module, file or crate doesn't
| matter too much as long as you keep code clean (as in not
| having supper entangled Spagetti code)
|
| and I wouldn't even be sure if D does perform in any
| relevant way better in compiler time if you compare their
| performance with the end result after they split the crate
| dathinab wrote:
| you are missing a lot of things
|
| - Compilation is an inherently hard to parallelize thing, and
| not just hard to parallelize but there are a lot of trade offs.
| This trade offs aren't even rust specific (i.e. C,C++, etc. are
| affected as much) and can lead to less performant generated
| binaries. And much more memory pressure which could make the
| compilation in total slower. Luckily for most code this doesn't
| matter as long as you don't goo too parallel. But's it's the
| reason why max codegen units is 16 not #num_cpus (for release
| builds, 256 for debug builds).
|
| - Codegen means producing machine code, i.e. we are speaking
| about of LLVM so the bug might not be rust specific and might
| affect other languages, too.
|
| - This still should mean 16 threads at high load not 1, so they
| seem to have hit a perforamnce-bug, i.e. not how things
| normally work. It's very unclear if the bug is in rust of LLVM
| if it's the later C/C++ might be affected, too.
|
| - While rust does strongly prefers you splitting you code into
| multiple crates, it still shouldn't be stuck at a singe threat,
| i.e. from everything we can tell we are hitting some
| performance bug.
|
| - Algorithm most times dominate performance much more then if
| you language is slightly faster or slower this is clearly an
| algorithmic error i.e. failing to parallelize while it normally
| does parallelize and the issue might be in C/C++ code, so "rust
| is fast" has pretty much nothing to do with this.
|
| - Through it still should be mentioned that for certain design
| reasons rust doesn't want you to make a single crate too large.
| In most normal situations you often end up splitting a crate
| for various reasons before it becomes too large, but if you
| have a very huge blog of auto generated code that is easy to
| miss. Funnily what can lead to compiler time issues if your
| crate is way to big also does "in general" lead to better
| performance which brings us back to a lot of decisions in
| compilers having trade offs.
|
| > then what's the value of that language from a software
| engineering viewpoint?
|
| You mean besides producing fast code in a way which is much
| better to maintain then C++ (or many other languages) but has
| many of the benefits C++ has over C when it comes to being able
| to reuse code and algorithm which practically makes it much
| easier to use better algorithms which often is a much higher
| performance gain then any normal language code optimizations.
| Something which has shown repeatedly in Praxis. Not even
| speaking about the fact that tends to have less bugs, makes it
| much easier to communicate interface constraints in a reliable
| maintainable way etc.
|
| The argument of a single case of running into a compile time
| performance bug while already doing something which isn't
| exactly a normal use case and doesn't follow the common advice
| to split crates if they become large somehow implying that rust
| has no value for software engineers is just kind dump. I mean
| you also wouldn't go around saying cats have no value from a
| family POV in general because one specific case where a cat did
| repeatedly scratch a teenager twice.
| imtringued wrote:
| It's a single Rust file with 100k lines of code spit out by a
| code generator.
| dathinab wrote:
| yesn't
|
| codegen in the OP article is machine gode generation (i.e.
| running LLVM)
|
| It's semantically kinda like having a single 100k file, but
| because rust knows it often generates huge "files" there is a
| splitting step, somewhere between parsing AST and generating
| machine code (I think after generating MIR but not fully
| sure). And the codgen-unit setting is in how many parts rust
| is allowed to split a thing which semantically is just one
| code unit. By default for release builds 16 (and as it can
| affect perf. of generated code it's not based on #cpus). But
| in there case there seems to be a bug which makes it
| effective more like 1! Which is much worse then it should be.
| (But also the statistic they show aren't sufficient to draw
| too much conclusions).
| dathinab wrote:
| The main issue here is:
|
| - in rust one semantic compilation unit is one crate
|
| - in C one semantic compilation unit is one file
|
| There are quite a bunch of benefits in the rust approach, but
| also drawbacks, like huge projects have to be split into multiple
| workspaces to maximize parallel building.
|
| Oversimplified the codegen-units setting tells the compiler into
| how many parts the compiler is allowed to split the a single
| semantic code gen unit.
|
| Now it still seems strange (as in it looks like a performance
| bug) that most times rust was stuck in just one threat (instead
| of e.g. 8).
| simfoo wrote:
| Agreed, this is the underlying main issue. I've faced it before
| with generated C++ code too, and after long and painful
| refactorings what ultimately helped the most was to just split
| the generated code into multiple compilation units to allow for
| parallel compilation. It comes with the drawback of potentially
| instatiating (and later throwing away) a lot more templates
| though
| bryanlarsen wrote:
| Rust has a great compromise between crate and file: module. I
| wonder why that's not the compilation unit?
| lmkg wrote:
| Per a reddit comment, modules are allowed to have circular
| dependencies while crates are not.
| colonial wrote:
| I think it's due to the fact that (unlike crates) cyclic
| dependencies are allowed between modules without any extra
| ceremony (e.x. forward declarations in C.)
| lsuresh wrote:
| > Now it still seems strange (as in it looks like a performance
| bug) that most times rust was stuck in just one threat (instead
| of e.g. 8).
|
| Agreed, seems like there are some rustc performance bugs at
| play here.
| steveklabnik wrote:
| I haven't dug into the details, but it may not even be a
| performance bug, depending on how you define 'bug': the Rust
| compiler is not fully parallel itself yet. That's a bug in
| the sense of something that needs to be improved and fixed,
| but isn't one in the sense of "unexpected bad behavior".
| lsuresh wrote:
| Makes sense. We'd appreciate some more eyeballs here for
| sure. Between HN and a Reddit thread, there are a few
| hypotheses floating around. I've shared a repro here for
| anyone interested:
| https://github.com/feldera/feldera/issues/3882
| steveklabnik wrote:
| You may want to post on the Zulip, I think that's the way
| to get in touch with the team these days.
| dathinab wrote:
| the thing is:
|
| codegen-units defaults to 16 in release builds, and by far
| the most time in the "passes" list is spend in LLVM passed
| (which is was codegen-units parallelizes),so most times it
| shouldn't be stuck with 1 high load core (even if it's not
| 16 all the time).
|
| so it looks a lot like something is prevented the intended
| codegen parallelization of the crate
|
| Through it indeed might not have been a bug, e.g. before
| the change in generation to split it across crates source
| code might have been in a way where it can't split the
| crate into multiple units. Or maybe something made rust
| believe splitting it is a bad idea, e.g. related to memory
| usage or similar.
| steveklabnik wrote:
| Ah yeah, that does sound like a bug to me; it's the
| earlier stages that I'm thinking of that aren't parallel
| yet.
| dietr1ch wrote:
| > back of the envelope calculation for how long it should take:
| 25 min / 128 = 12 sec (or maybe 24 sec since hyper-threads aren't
| real cores). Yet it takes 170s to compile everything.
|
| I'd aim for this linear speedup for compiling (sans overhead to
| compile a small crate), but the linking part won't be faster,
| maybe even slower. Maybe a slightly bigger envelope can tell you
| how much performance is there to extract and the cost of using
| "too many" crates (which I'm not even sure it's too many, maybe
| your original crate was too big to ease incremental compilation?)
| lsuresh wrote:
| Towards the end, the article says it takes 7s for linking using
| mold.
| dietr1ch wrote:
| Yeah, I read that part too late. In that case it seems that
| there's indeed a lot of overhead when building many crates
| from a cold-start, but it pays off in wall time and can
| probably save resources in incremental builds.
| lsuresh wrote:
| Yes. We found both cold and incremental builds sped up. The
| incremental builds were the main win -- small changes to
| the SQL can sometimes complete in seconds for what used to
| be a full recompilation.
| rc00 wrote:
| Rewrite it in Zig? You might even be able to sidestep the
| LLVM bottleneck entirely.
| https://news.ycombinator.com/item?id=43016944
| UncleEntity wrote:
| Yeah, I was working on this project to generate a python C-API
| module from the SVG schema and found when I generated one huge
| file (as opposed to one file per class) the compilation times
| were significantly faster. Or maybe it was generating the C++
| SVG library, don't quite remember which one was super slow as I
| did both at around the same time since the code changes between
| the two were minimal.
|
| Looks like I settled on the slower 'one class per file'
| compilation method for whatever reason, probably because
| generating a 200k+ file didn't seem like such a good idea.
| wiseowise wrote:
| > Given that we now fully utilize 128 threads or 64 cores for
| pretty much the entire compile time, we can do a back of the
| envelope calculation for how long it should take: 25 min / 128 =
| 12 sec (or maybe 24 sec since hyper-threads aren't real cores).
| Yet it takes 170s to compile everything.
|
| Amdahl's Law would like to have a word.
| stouset wrote:
| This is an observed change of going from 1 core at 100% to 64
| cores at 100%. This is establishing a _lower_ bound, assuming
| there is no wasted work or contention for shared resources.
| gz09 wrote:
| FWIW the article carefully worded it as a "back of the
| envelope" calculation, says we can't expect a linear speedup in
| practice and also gives the time it takes for linking (7 secs).
|
| (Disclaimer I am the author of the article and I am quite
| familiar with the law)
| throwawaymaths wrote:
| Amdahls law is about coordination costs, so either you would
| expect cores to be starved or lots pf extra coordination-
| related compute to be happening, which, i guess is not totally
| crazy since there are that many crates, but as a first guess
| OP's back of the envelope is fine
| jmull wrote:
| That's a cool project.
|
| But I wonder if generating rust is the best approach. On the plus
| side, you can take advantage of the rich type and type checking
| system the compiler has. On the other hand, you're stuck with
| that compiler.
|
| I wonder if the _dynamic_ constraints can be expressed and
| checked through some more directly implemented mechanism. It
| should be both simpler to express exactly the constraints you
| want (no need to translate to a rust construct that rustc will
| check as desired), and, of course, should be a lot more
| efficient. Feldera may have no feasible way to get away from
| generated rust, but a potential competitor might avoid the issue.
| (That 's not to say the runtime shouldn't/couldn't be implemented
| in rust. I'm just talking about the large amounts of generated
| rust.)
| pjmlp wrote:
| The right way done by the likes of Oracle and SQL Server is to
| JIT compile their queries and stored procedures, with PGO data
| from query analyser.
| lsuresh wrote:
| We think a JIT compiler is the right approach too. Will be a
| substantial effort though, so we're waiting to get a bit of
| bandwidth on that front.
| slashdev wrote:
| I would probably have gone with generating C here. You don't
| need all the safety of the Rust compiler, you're the one
| generating the code. As you point out, you can check all the
| constraints you want before you compile it.
| lesuorac wrote:
| > Of course, we tried debug builds too. Those cut the time down
| to ~5 minutes -- but they're not usable in practice.
|
| I wonder how true this is.
|
| Haven't use feldera but other rust stuff I have if I run as debug
| it has serious performance problems. However, for testing I have
| it compile a few crates that do work like `image` to be optimized
| (and the vast majority as debug) and that is enough to make the
| performance issues not noticeable. So if the multi-crate hadn't
| worked, possibly just only compile some of the stuff as
| optimized.
| lsuresh wrote:
| For any Rust compiler experts interested in taking a look, I've
| put together a short repro here:
| https://github.com/feldera/feldera/issues/3882
|
| It will give you a workspace with a bunch of crates that seems to
| exercise some of the same bottlenecks the blog post described.
| sitzkrieg wrote:
| the rust compiler is so impressively slow
| ameixaseca wrote:
| It actually isn't.
|
| It becomes more evident when you consider the amount of work it
| is doing as well.
| ay wrote:
| I have just went through this with a project of mine, though
| unfortunately the code wasn't autogenerated, so I needed to do a
| lot of mind-numbingly boring search-and-replace commands. I have
| cobbled together a little utility that allowed to automate the
| process somewhat.
|
| Mostly a throwaway code with a heavy input from Claude, so the
| docs are in the code itself :-)
|
| But in case anyone can find it useful:
|
| https://github.com/ayourtch/tweak-code
| nixpulvis wrote:
| I love Rust, but it seems like a really bad intermediate language
| if you have a compiler/transpiler which is sound. You don't need
| the type system from Rust telling you something's wrong.
|
| That said, I could see how it would make writing the transpiler
| easier, so that's a win.
| skeptrune wrote:
| Rust documentation needs some work to emphasize that splitting
| the codebase into many smaller crates is the "correct" way to do
| things if you care about build time.
| 1vuio0pswjnm7 wrote:
| The dependency costs seem to be self-evident but perhaps it would
| be enlightening to see a comparison of the energy costs, as well
| as the time costs and storage costs, of compiling Rust programs
| versus their C equivalents. For example, comparison might reveal
| that there is no difference and no trade-off or that any
| differences and trade-offs are small enough to be worth making in
| the interest of some higher purpose.
___________________________________________________________________
(page generated 2025-04-17 23:02 UTC)