[HN Gopher] I made a real-time C/C++/Rust build visualizer
___________________________________________________________________
I made a real-time C/C++/Rust build visualizer
Author : dhooper
Score : 158 points
Date : 2025-08-14 16:06 UTC (6 hours ago)
(HTM) web link (danielchasehooper.com)
(TXT) w3m dump (danielchasehooper.com)
| bgirard wrote:
| That's really cool. Fascinating to think about all the problems
| that get missed due to poor or missing visualizations like this.
|
| I did a lot of work to improve the Mozilla build system a decade
| ago where I would have loved this tool. Wish they would have said
| what problem they found.
| dhooper wrote:
| (OP here) Thanks!
|
| My call with the Mozilla engineer was cut short, so we didn't
| have time to go into detail about what he found, I want to look
| into it myself.
| bvisness wrote:
| Hello, I am the engineer in question. I am not actually super
| familiar with the details of the build system, but from when
| I saw, the main issues were:
|
| - Lots of constant-time slowness at the beginning and end of
| the build
|
| - Dubious parallelism, especially with unified builds
|
| - Cargo being Cargo
|
| Overall it mostly looks like a soup of `make` calls with no
| particular rhyme or reason. It's a far cry from the ninja
| example the OP showed in his post.
| xuhu wrote:
| Is there a tool that records the timestamp of each executed
| command during a build, and when you rebuild, it tells you how
| much time is left instead of "building obj 35 out of 1023" ?
|
| Or (for cmake or ninja) use a CSV that says how long each object
| takes to build and use it to estimate how much is left ?
| dhooper wrote:
| OP Here. Thats an interesting idea. What The Fork knows all the
| commands run, and every path they read/write, so I should be
| able to make it estimate build time just by looking at what
| files were touched.
| Surac wrote:
| but why? I have to admit it's a fun project
| rvrb wrote:
| here, I'll copy the first paragraph of TFA for you:
|
| > Many software projects take a long time to compile. Sometimes
| that's just due to the sheer amount of code, like in the LLVM
| project. But often a build is slower than it should be for
| dumb, fixable reasons.
| corysama wrote:
| Looks like a general `fork()` visualizer to me. Which is great!
| supportengineer wrote:
| Amazing! Great job!
|
| What limits your tool to compiler/build tools, can it be used for
| any arbitrary process?
| dhooper wrote:
| Thank you! Yeah it can be used for any type of program, but I
| haven't been able to think of anything besides compilation that
| creates enough processes to be interesting. I'm open to ideas!
| DiddlyWinks wrote:
| Video encoding and 3-D rendering are a couple that come to
| mind; I'd think they'd launch quite a few.
|
| This looks like a really cool tool!
| Night_Thastus wrote:
| I am extremely interested in this.
|
| I am stuck in an environment with CMake, GCC and Unix Make (no
| clang, no ninja) and getting detailed information about WHY the
| build is taking so long is nearly impossible.
|
| It's also a bit of a messy build with steps like copying a bunch
| of files from the source into the build folder. Multiple
| languages (C, C++, Fortran, Python), custom cmake steps, etc.
|
| If this tool can handle that kind of mess, I'll be very
| interested to see what I can learn.
| unddoch wrote:
| I wrote a little GCC plugin for compile time tracing/profiling,
| if that's something you're interested in:
| https://github.com/royjacobson/externis
| phaedrus wrote:
| When I was trying to improve compile time for my game engine, I
| ended up using compiled size as a proxy measure. Although it is
| an imperfect correlation, the fact that compiled size is
| deterministic across build runs and even across builds on
| different machines makes it easier to work with than wall clock
| time.
| mlsu wrote:
| Wait, this is not intuitive at all for me.
|
| If the compiler is working harder wouldn't that result in a
| more compact binary? Maybe I'm thinking too much from an
| embedded software POV.
|
| I suppose the compiler does eventually do IO but IO isn't
| really the constraint most of the time right?
| staticfloat wrote:
| While you can cause the compiler to run longer to squeeze
| the binary size down, the compiler has a baseline number of
| compiler passes that it runs over the IR of the program
| being compiled. These compiler passes generally take time
| proportional to the input IR length, so a larger program
| takes longer to compile. Most compiler passes aren't
| throwing away huge amounts of instructions (dead code
| elimination being a notable exception, but the analysis to
| figure out which pieces of dead code can be eliminated
| still is operating on the input IR). So it's not a perfect
| proxy, but in general, if the output of your compiler is
| 2MB of code, it probably took longer to process all the
| input and spit out that 2MB than if the output of your
| compiler was 200KB.
| johannes1234321 wrote:
| Of course there are the cases where a huge template
| structure with complex instantiation and constexpr code
| compiles down to a single constant, but for most parts of
| the code I would assume there is a proportion from code
| size, via compile time to binary size.
| pklausler wrote:
| strace might help, if you have it.
| aanet wrote:
| This is fabulous!!
|
| Is there a version available for MacOS today?? I'd love to give
| it a whirl... For Rust, C++ / Swift and other stuff.
|
| Thanks!
| dhooper wrote:
| I'll be sending out the a macOS version to another wave of beta
| users after I fix an outstanding issue, if you sign up (at
| bottom of article) and mention this comment I can make sure
| you're in that wave.
| aanet wrote:
| Thanks. Signed up
| Night_Thastus wrote:
| It looks like it doesn't have a public release for any OS yet,
| but has a way to enter for early access.
| mgaunard wrote:
| The real solution is to eliminate build systems where you have to
| define your own targets.
|
| Developers always get it wrong and do it badly.
| tiddles wrote:
| Nice, I've been looking for something like this for a while.
|
| I've noticed on my huge catkin cmake project that cmake is
| checking the existence of the same files hundreds of times too.
| Is there anything that can hook into fork() and provide a cached
| value after the first invocation?
| lights0123 wrote:
| My tips for speeding up builds (from making this same project
| but with ebpf):
|
| - switch to ninja to avoid that exact issue since CMake + Make
| spawns a subprocess for every directory (use the binary from
| PyPi for jobserver integration)
|
| - catkin as in ROS? rm
| /opt/ros/noetic/etc/catkin/profile.d/99.roslisp.sh to remove 2
| python spawns per package
| brcmthrowaway wrote:
| What about OSes that dont use fork()?
| dhooper wrote:
| I use whatever the equivalent is on that OS.
| klik99 wrote:
| Nice! Leaving a comment to easily find this later, dont have
| anything to add except this looks cool
| boris wrote:
| > It also has 6 seconds of inactivity before starting any useful
| work. For comparison, ninja takes 0.4 seconds to start compiling
| the 2,468,083 line llvm project. Ninja is not a 100% fair
| comparison to other tools, because it benefits from some "baked
| in" build logic by the tool that created the ninja file, but I
| think it's a reasonable "speed of light" performance benchmark
| for build systems.
|
| This is an important observation that is often overlooked. What's
| more, the changes to the information on which this "baked in"
| build logic is based is not tracked very precisely.
|
| How close can we get to this "speed of light" without such
| "baking in"? I ran a little benchmark (not 100% accurate for
| various reasons but good enough as a general indication) which
| builds the same project (Xerces-C++) both with ninja as
| configured by CMake and with build2, which doesn't require a
| separate step and does configuration management as part of the
| build (and with precise change tracking). Ninja builds this
| project from scratch in 3.23s while build2 builds it in 3.54s. If
| we omit some of the steps done by CMake (like generating
| config.h) by not cleaning the corresponding files, then the time
| goes down to 3.28s. For reference, the CMake step takes 4.83s. So
| a fully from-scratch CMake+ninja build actually takes 8s, which
| is what you would normally pay if you were using this project as
| a dependency.
| remexre wrote:
| > What's more, the changes to the information on which this
| "baked in" build logic is based is not tracked very precisely.
|
| kbuild handles this on top of Make by having each target depend
| on a dummy file that gets updated when e.g. the CFLAGS change.
| It also treats Make a lot more like Ninja (e.g. avoiding
| putting the entire build graph into every Make process) -- I'd
| be interested to see how it compares.
| forrestthewoods wrote:
| This is great. I was skeptical from the title but the
| implementation is very clever. This could be a super super useful
| tool for the industry.
| kirito1337 wrote:
| That's interesting.
| jeffbee wrote:
| This seems like a good place to integrate a Bazel Build Event
| Protocol stream consumer.
| MathMonkeyMan wrote:
| I was going to comment that "what the fork" might not work for
| a client/server build system like Bazel, but now I have
| something to google instead.
| CyberDildonics wrote:
| I love the visualization, I think it's great information and will
| be very helpful to whoever uses it.
|
| I would think about a different name. Often names are either
| meant to be funny or just unique nonsense but something short and
| elegantly descriptive (like BuildViz etc.) can go a long way to
| making it seem more legitimate and being more widely used.
| dhooper wrote:
| Thanks CyberDildoNics!
| hiccuphippo wrote:
| Name checks out.
| entelechy wrote:
| Love it! We did something similar using strace/dtruss back in
| 2018 with https://buildinfer.loopperfect.com/ and were generating
| graphs (using eg. graphviz and perfetto.dev) and BUCK files on
| the back of that
|
| Whilst we regrettably never came around to package it as a
| propper product, we found it immensly valuable in our consulting
| work, to pinpoint issues and aid the conversion to BUCK/Bazel. We
| used graphviz, https://perfetto.dev/ and couple other tools to
| visualise things
|
| Recently we cicled back to this too but with a broader usecase in
| mind.
|
| There are some inherent technical challanges with this approach &
| domain:
|
| - syscall logs can get huge - especially when saved to disk. Our
| strace logs would get over 100GB for some projects (llvm was
| around ~50GB)
|
| - some projects also use https and inter process communications
| and that needs ot be properly handled too. (We even had a
| customer that was retriving code from a firebird database via
| perl as part of the compilation step!)
|
| - It's runtime analysis - you might need to repeat the analysis
| for each configuration.
| proctorg76 wrote:
| the parallels between tech and manufacturing never cease to
| amaze, this looks so much like the machine monitoring / execution
| system we use in the car parts plant I want to ask if you've
| calculated the TEEP and OEE of your build farm
| Cloudef wrote:
| LLVM is taking its sweet time, brew coffee
| mrlonglong wrote:
| Does it work with cmake?
| tom_ wrote:
| If you use the Visual C++ compiler on Windows, vcperf is worth a
| look: https://github.com/microsoft/vcperf - comes with VS2022, or
| you can build from github.
|
| I've used it with projects generated by UBT and CMake. I can't
| remember if it provides any info that'd let you assess the
| quality of build parallelism, but it does have some compiler
| front end info which is pretty straightforward to read.
| Particularly expensive headers (whether inherently so, or just
| because they're included a lot) are easy to find.
| metalliqaz wrote:
| Isn't `wtf` already a fairly common command?
| audiofish wrote:
| Really cool tool, but perhaps not for the original use-case. I
| often find myself trying to figure out what call tree a large
| Bash script creates, and this looks like it visualises it well.
|
| This would have been really useful 6 months ago, when I was
| trying to figure out what on earth some Jetson tools actually did
| to build and flash an OS image.
___________________________________________________________________
(page generated 2025-08-14 23:00 UTC)