https://mill-build.org/blog/12-direct-style-build-tool.html

[logo-white] The Mill Build Tool
[                    ]
GitHub Blog API Issues Discuss

  * 
      + Mill as a Direct Style Build Tool
      + Strategies for Efficiently Parallelizing JVM Test Suites
      + Invalidating build caches using JVM bytecode callgraph
        analysis
      + Fast Incremental JVM Assembly Jar Creation with Mill
      + What does a Build Tool do?
      + How to Compile Java into Native Binaries with Mill and Graal
      + Understanding JVM Garbage Collector Performance
      + How JVM Executable Assembly Jars Work
      + How To Manage Flaky Tests in your CI Workflows
      + Faster CI with Selective Testing
      + Why Use a Monorepo Build Tool?
      + How Fast Does Java Compile?

The Mill Build Engineering Blog

  * Mill Documentation
      + main-branch
      + 0.12.11
      + 0.11.13
      + 0.10.15
      + 0.9.12
  * The Mill Build Engineering Blog

Edit this Page

Mill as a Direct Style Build Tool

Mill is a JVM build tool that targets Java/Scala/Kotlin and has
potential to serve the large-monorepo codebases that Bazel currently
serves. Mill has good traction among its users, benchmarks that
demonstrate 3-6x faster builds than its competitors, and a unique
"direct-style" design that make it easy to use and extend. This page
discusses some of the most interesting design decisions in Mill, and
how it sets Mill apart from other build tools on the market.

 What is a Build Tool?

A build tool is a program that coordinates the various tasks
necessary to compile, package, test, and run a codebase: maybe you
need to run a compiler, download some dependencies, package an
executable or container. While a small codebase can get by with a
shell script that runs every task every time one at a time, such a
naive approach gets slower and slower as a codebase grows and the
build tasks necessarily get more numerous and complex.

In order to prevent development from grinding to a halt, you need to
begin skipping the build tasks you do not need at any point in time,
and caching and parallelizing those that you do. This often starts
off as some ad-hoc if-else statements in a shell script, but manually
maintaining skipping/caching/parallelization logic is tedious and
error-prone. At some point it becomes worthwhile to use an purpose
built tool to do it for you, and that is when you turn to build tools
like Maven, Make, Mill, or Bazel. For this article, we will mostly
discuss Mill.

 What is Mill?

The Mill build tool was started in 2017, an exploration of the ideas
I found when learning to use Google's Bazel build tool. At a glance,
Mill looks similar to other build tools you may be familiar with,
with a build.mill file in the root of a project defining the
dependencies and testing setup for a module:

package build
import mill._, javalib._

object foo extends JavaModule {
  def mvnDeps = Seq(
    mvn"net.sourceforge.argparse4j:argparse4j:0.9.0",
    mvn"org.thymeleaf:thymeleaf:3.1.1.RELEASE"
  )

  object test extends JavaTests with TestModule.Junit4
}

The syntax may be a bit unfamiliar, but anyone familiar with
programming can probably guess what this build means: a JavaModule
with two ivy dependencies argparse4j and thymeleaf, and a test
submodule supporting Junit4. This build can then be compiled, tested,
run, or packaged into an assembly from the command line:

> /mill foo.compile
compiling 1 Java source...

> /mill foo.run --text hello
<h1>hello</h1>

> ./mill foo.test
Test foo.FooTest.testEscaping finished, ...
Test foo.FooTest.testSimple finished, ...
0 failed, 0 ignored, 2 total, ...

> ./mill show foo.assembly
".../out/foo/assembly.dest/out.jar"

> ./out/foo/assembly.dest/out.jar --text hello
<h1>hello</h1>

Mill was originally a Scala build tool competing with sbt, and by
2023 it had reached around 5-10% market share in the Scala community
(Jetbrains Survey, VirtusLabs Survey). It recently grew first-class
Java support, demonstrating 3-6x speedups over existing Java build
tools like Maven or Gradle. Mill also has gained experimental support
for Java-adjacent platforms like Kotlin and Android, and has
demonstrated the ability to branch out into supporting more distant
toolchains like Typescript and Python.

Mill also works well with large builds: its build logic can be split
into multiple folders, is incrementally compiled, lazily initialized,
and automatically cached and parallelized. That means that even large
codebases can remain fast and responsive: Mill's own build easily
manages over 400 modules, and the tool can likely handle thousands of
modules without issue.

 The React.js of Build Tools

We've briefly covered what Mill is above, but one question remains:
why Mill? Why not one of the other 100 build tools out there?

Mill is unique in that it shares many of its core design decisions
with React.js, the popular Javascript UI framework. I was among the
first external users of React when I introduced it to Dropbox in
2014, and while people gripe about it today, React was really a
revolution in how Javascript UIs were implemented. UI flows that used
to take weeks suddenly took days, requiring a fraction of the code
and complexity that they previously took to implement

React's two most important innovations are:

 1. Letting users write "direct style" code to define their UI -
    Javascript functions that directly returned the HTML structure
    you wanted - rather than a "code behind" approach of registering
    callbacks to mutate the UI in response to events

 2. Using a single "general purpose" programming language for your
    UI, rather than splitting your logic into multiple
    special-purpose domain-specific languages

While React does a huge number of clever things - virtual dom diffing
, JSX, de/re-hydration, etc. - all of those are only in service of
the two fundamental ideas. e.g. At Dropbox we used React for years
without JSX, and many of the later frameworks inspired by React
provide a similar experience but use other techniques to replace
virtual dom diffing. Furthermore, React isn't limited to the HTML
UIs, with the same techniques being used to manage mobile app UIs,
terminal UIs, and many other scenarios

Build tools and interactive UIs are on one hand different, but on the
other hand very similar: you are trying to update a large stateful
system (whether a HTML page or filesystem build artifacts) to your
desired state in response to change in inputs (whether user-clicks or
source-file-edits). Like with React in 2014, these two ideas are not
widespread among build tools today in 2024. But many of the same
downstream benefits apply, and these ideas give Mill some unique
properties as a build tool.

 Direct-Style Builds

One key aspect of React.js is that you wrote your code to generate
your web UI "directly":

  * Before React, you would write Javascript code whose purpose was
    to mutate some HTML properties to set up a forest of callbacks
    and event handlers. These would then be executed when a user
    interacted with your website, causing further mutations to the
    HTML UI. This would often recursively trigger other callbacks
    with further mutations, and you as the developer would somehow
    need to ensure this all converges to the UI state that you
    desire.

  * In React, you had normal functions containing normal code that
    executed top-to-bottom, each returning a JSX HTML snippet -
    really just a Javascript object - with the top-level component
    eventually returning a snippet representing the entire UI. React
    would handle all the update logic for you in an efficient manner,
    incrementally caching and optimizing things automatically. The
    developer just naively returns the UI structure they want from
    their React code and React.js does all the rest

Before React you always had a tradeoff: do you re-render the whole UI
every update (which is easy to implement naively, but wasteful and
disruptive to users) or do you do fine-grained UI updates (which was
difficult to implement, but efficient and user-friendly). React
eliminated that tradeoff, letting the developer write "naive" code as
if they were re-rendering the entire UI, while automatically
optimizing it to be performant and provide a first-class user
experience.

Mill's approach as a build tool is similar:

  * Most existing build tools involve registering "task" callbacks to
    tell the build tool what to do when certain actions happen or
    certain files change. These callbacks mutate the filesystem in an
    ad-hoc manner, often recursively triggering further callbacks. It
    is up to the developer to make sure that these callbacks and
    filesystem updates end up converging such that your build outputs
    ends up containing the files you want.

  * With Mill, you instead write "direct-style" code: normal
    functions that call other functions and end up returning the
    final metadata or files that were generated. Mill handles the
    work of computing these functions efficiently: automatically
    caching, parallelizing, and optimizing your build. The developer
    writes naive code computing and returning the files they want,
    and Mill does all the rest to make it efficient and performant

Earlier we saw a hello-world Mill build using the built in module
types like JavaModule, but if we remove these built in classes we can
see how Mill works under the hood. Consider the following Mill tasks
that define some source files, use the javac executable to compile
them into classfiles, and then the jar executable to package them
together into an assembly:

def mainClass: T[Option[String]] = Some("foo.Foo")

def sources = Task.Source("src")
def resources = Task.Source("resources")

def compile = Task {
  val allSources = os.walk(sources().path)
  os.proc("javac", allSources, "-d", Task.dest).call()
  PathRef(Task.dest)
}

def assembly = Task {
  for(p <- Seq(compile(), resources())) os.copy(p.path, Task.dest, mergeFolders = true)

  val mainFlags = mainClass().toSeq.flatMap(Seq("-e", _))
  os.proc("jar", "-c", mainFlags, "-f", Task.dest / "assembly.jar", ".")
    .call(cwd = Task.dest)

  PathRef(Task.dest / "assembly.jar")
}

This code defines the following task graph, with the boxes being the
tasks and the arrows representing the data-flow between them:

            sources compile assembly resources mainClass

This example does not use any of Mill's builtin support for building
Java or Scala projects, and instead builds a pipeline "from scratch"
using Mill tasks and javac/jar subprocesses. We define Task.Source
folders and plain Tasks that depend on them, implementing entirely in
our own code.

Two things are worth noting about this code:

 1. It looks almost identical to the equivalent "naive" code you
    would write without using a build tool! If you remove the Task
    {... } wrappers, you could run the code and it would behave as a
    naive script running top-to-bottom every time and generating your
    assembly.jar from scratch. But Mill allows you to take such naive
    code and turn it into a build pipeline with parallelism, caching,
    invalidation, and so on.

 2. You do not see any logic at all related to parallelism, caching,
    invalidation in the code at all! No mtime checks, no computing
    cache keys, no locks, no serializing and de-serializing of data
    on disk. Mill handles all this for you automatically, so you just
    need to write your "naive" code and Mill will provide all the
    "build tool stuff" for free.

This direct-style code has some surprising benefits: IDEs often not
understand how registered callbacks recursively trigger one another,
but they do understand function calls, and so they should be able to
seamlessly navigate up and down your build graph just by following
those functions. Below, we can see IntelliJ resolve compile to the
exact def compile definition in build.foo, allowing us to jump to it
if we want to see what it does:

IntellijDefinition

In the JavaModule example earlier, IntelliJ is able to see the def
mvnDeps configuration override, and find the exact override
definitions in the parent class hierarchy:

IntellijOverride

This "direct style" doesn't just make navigating your build easy for
IDEs: human programmers are also used to navigating in and out of
function calls, up and down class hierarchies, and so on. Thus for a
developer configuring or maintaining their build system, Mill's
direct style means they easier time understanding what is going on,
especially compared to the classic "callbacks forests" you may have
come to expect from build tools. However, both of these benefits
require that the IDE and the human understands the code in the first
place, which leads to the second major design decision:

 Using a Single General Purpose Language

React.js makes users use Javascript to implement their HTML UIs.
While a common approach now in 2024, it is hard to overstate how
controversial and unusual this design decision was at the time.

In 2014, web UIs were implemented in some HTML templating language
with separate CSS source files, and "code behind" Javascript logic
hooked in. This allowed separation of concerns: a graphic designer
could edit the HTML and CSS without needing to know Javascript, and a
programmer could edit the Javascript without needing to be an expert
in HTML/CSS. And so writing frontend code in three languages in three
separate files was the best practice, and so it was since the
inception of the web two decades prior.

React.js flipped all that on its head: everything was Javascript! UI
components were Javascript objects first, containing Javascript
functions that returned HTML snippets (which were really also
Javascript objects). CSS was often in-lined at the use site, perhaps
with constants fetched from a CSS-in-JS library. This was a total
departure from the previous two decades of web development best
practices.

While controversial, this approach had two huge advantages:

 1. It broke the hard language barriers between HTML/CSS/JS, allowing
    more flexible ways of organizing and grouping code in order to
    meet the needs of the particular UI. While seemingly trivial, it
    makes a huge difference to have one file in one language
    containing everything you need to know about a UI component,
    rather than needing to tab between three files in three different
    languages.

 2. It removed the separate second-class "templating language". While
    the "platonic ideal" was people writing HTML/CSS/JS, the HTML
    often ended up being Jinja2, HAML, or Mustache templates instead,
    and the CSS usually ended up being replaced by SASS or LESS.
    While Javascript was by no means perfect, having everything in a
    single "real" programming language was a breath of fresh air over
    tabbing between three different languages each with their own
    half-baked version of language features like if-else, loops,
    functions, etc.

The story for build tools is similar: the traditional wisdom has been
to implement your build logic in some limited "build language", in
the past often XML (e.g. for Maven, MSBuild), nowadays often JSON/
TOML/YAML (e.g. Cargo), with logic split out into separate shell
scripts or plugins. While this worked, it always had issues:

 1. Like web development, build tools also had the logic split
    between multiple languages. Templated-Bash-in-Yaml is a common
    outcome, Bazel makes you write make-interpolated Bash in
    pseudo-Python, Maven makes you choose between XML+Java to write
    plugins or Bash-in-XML Ant scripts. Most build tools using
    "simple" config languages would inevitably find logic pushed into
    shell scripts within the build, or the entire build tool itself
    wrapped in a shell script to provide the flexibility a project
    needs

 2. These "simple build languages" would always start off simple, but
    eventually grow real programming language features: not just
    if-else, loops, functions, inheritance, but also package
    managers, package repositories, profilers, debuggers, and more.
    These were always ad-hoc, designed and implemented in their own
    weird and idiosyncratic ways, and generally inferior to the same
    feature or tool provided by a real programming language.

"Config metadata turns into templating language turns into
general-purpose language" is a tale as old as time. Whether it's HTML
templating using Jinja2, CI configuration using Github Actions Config
Expressions, or infrastructure-as-code systems like Cloudformation
Functions or Helm Charts. While the allure of using a "simple" config
language is strong, many systems inevitably end up growing so many
programming-language features that you would have been better off
using a general-purpose language to start off with.

Mill follows React.js with its "One General-Purpose Language"
approach:

 1. Mill tasks are just method definitions

 2. Mill task dependencies are just method calls

 3. Mill modules are just objects

While this is not strictly true - Mill tasks and Mill modules have a
small amount of extra logic necessary to handling caching
parallelization and other build tool necessities - it is true enough
that these details are often completely transparent to the user.

This has the same benefits that React.js had from using a
general-purpose language throughout:

 1. You can directly write code to wire up and perform your build
    logic all in one language, without the nested
    Bash-nested-in-Mustache-templates-nested-in-YAML monstrosities
    common when insufficiently flexible config languages are chosen.

 2. You already know how programming languages works: not just
    conditionals loops and functions, but also classes, inheritance,
    overrides, typechecking, IDE navigation, package repositories and
    library ecosystem (in Mill's case, you can use everything on
    Java's Maven Central repository). Rather than dealing with
    half-baked versions of these features that specialized languages
    inevitable grow, Mill lets you use the real thing right off the
    bat.

For example, in Mill you may not be familiar with the bundled
libraries and APIs, but your IDE can help you understand them:

IntellijDocs

And if you make an error, e.g. you typo-ed resources as reources,
your IDE will immediately flag it for you even before you run the
build:

IntellijError

While all IDEs have good support for understanding JSON/TOML/YAML/
XML, the support for understanding a particular tool's dialect of
templated-bash-in-yaml is much more spotty. Even IntelliJ, the gold
standard, usually cannot provide more than basic assistance editing
templated-bash-in-yaml configs file. In contrast, IDE support for a
widely-used general purpose programming language is much more solid.

As another example, if you need a production-quality templating
engine to use in your build system, you have a buffet of options. The
common Java Thymeleaf templating engine is available with a single
import, as is the popular Scalatags templating engine. Rather than
being limited to what the build tool has built-in or what third-party
plugins someone on the internet has published, you have at your
fingertips any library in the huge JVM ecosystem, and can use them in
exactly the same way you would in any Java/Scala/Kotlin application.

 What About Other Build Tools?

There are existing build tools that use some of the ideas above, but
perhaps none of them have both, which is necessary to take full
advantage:

  * Tools like Gradle, Rake, or Gulp may be written in a single
    language, but are not direct-style: they still rely on you
    registering a forest of callbacks performing filesystem
    mutations, and manually ensuring that they are wired up to
    converge to the state you want. This means that although that a
    human programmer or an IDE like IntelliJ may be able to navigate
    around the Groovy/Kotlin/Ruby code used to configure the build,
    both human and machine often have trouble tracing through the
    forest of mutating callbacks to figure out what is actually
    happening

  * Tools like Cargo, Maven, or go build are very inflexible. This
    leads either to embedded shell scripts (or
    embedded-shell-scripts-as-XML such as the Maven AntRun Plugin!),
    or having the build tool mvn/cargo/go being itself wrapped in
    shell scripts (or even another build tool like Bazel!)

Mill's direct style code and use of a general-purpose language makes
it unique among build tools, just like how React.js was unique among
UI frameworks when it was first released in 2014. With these two key
design features, Mill makes understanding and maintaining your build
an order of magnitude easier than traditional tools, democratizing
project builds so anyone can contribute without needing to be
experts.

 Where can Mill Go?

Above, we discussed some of the unique design decisions of Mill, and
the value they provide to users. In this section we will discuss
where Mill can fit into the larger build-tool ecosystem. I think Mill
has legs to potentially grow 10x to 100x bigger than it is today.
There are three main areas where I think Mill can grow into:

 A Modern Java/JVM Build Tool

Mill is a JVM build tool, and the JVM platform hosts many rich
communities and ecosystems: the Java folks, offshoots like Android,
other languages like Kotlin and Scala. All these ecosystems rely on
tools like Maven or Gradle to build their code, and I believe Mill
can provide a better alternative. Even today, there are already many
advantages of using Mill over the incumbent build tools:

 1. Mill today runs the equivalent local workflows 3-6x faster than
    Maven and 2-4x faster than Gradle, with automatic parallelization
    and caching for every part of your build

 2. Mill today provides better ease of use than Maven or Gradle, with
    IDE support for navigating your build graph and visualizing what
    your build is doing

 3. Mill today makes extending your build 10x easier than Maven or
    Gradle, directly using the same JVM libraries you already know
    without being beholden to third-party plugins

The JVM is a flexible platform, and although Java/Kotlin/Scala/
Android are superficially different, underneath there is a ton of
similarity. Concepts like classfiles, jars, assemblies, classpaths,
dependency management and publishing artifacts, IDEs, debuggers,
profilers, many third-party libraries, are all shared and identical
between the various JVM languages. Mill provides a first class Java
and Scala experience, with growing support for Kotlin and Android.
Mill's easy extensibility means integrating new tools into Mill takes
hours rather than days or weeks.

In the last 15-20 years, we have learned a lot about build tooling,
and the field has developed significantly:

  * Bazel, Buck, Pants have emerged to manage large codebases

  * Webpack, Snowpack, ESBuild, Nx, TurboRepo, Vite have emerged for
    Javascript

  * Astral, Poetry, and others have emerged for Python

  * We have seen papers published like Build Systems A La Carte, that
    thoroughly explore the design space for how a build tool might
    work.

But there are no build tools in the Java/JVM ecosystem that really
take advantage of these newer designs and techniques: ideas like
having a build graph, automatic caching, automatic parallelization,
side-effect-free build tasks, and so on. While Maven (from 2004) and
Gradle (2008) have been slowly trying to move in these directions,
they are also constrained by their two decades of legacy that limits
how fast they can evolve.

Mill could be the modern Java/JVM build tool: providing 10x speedups
over Maven or Gradle, 10x better ease of use, 10x better
extensibility. Today Mill already provides a compelling Java build
experience. With some focused effort, I think Mill can be not just a
good option, but the better option for Java projects going forward!

 An Easier Monorepo Build Tool

Many companies are using Bazel today. Of the companies I interviewed
from my Silicon Valley network, 25 out of 30 are using or trying to
use Bazel. Bazel is an incredibly powerful tool: it provides
sandboxing, parallelization, remote caching, remote execution. These
are all things that are useful or even necessary as your organization
and codebase grows. I even wrote about the benefits on my company
blog at the time:

  * Speedy Scala Builds with Bazel at Databricks

  * Fast Parallel Testing with Bazel at Databricks

There is no doubt that if set up correctly, Bazel is a great
experience that "just works", and with a single command you can do
anything that you could want to do in a codebase.

But of those 25 companies I interviewed, basically everyone was
having a hard time adopting Bazel. From my own experience, both of my
prior employers (Dropbox and Databricks) both took O(1 person decade)
of work to adopt Bazel. I have met multiple Silicon Valley dev-tools
teams that spent months doing a Bazel proof-of-concept only to give
up due to the difficulty. Bazel is a ferociously complex tool, and
although some of that complexity is inherent, much of it is
incidental, and some of it is to support projects at a scale beyond
what most teams would encounter.

I think there is room for a lightweight monorepo build tool that
provides maybe 50% of Bazel's functionality, but at 10% the
complexity:

  * Most companies are not Google, don't operate at Google-scale, do
    not have Google-level problems, and may not need all the most
    advanced features that Bazel provides

  * Bazel itself is not getting any simpler over time - instead is
    getting more complex with additional features and functionality,
    as tends to happen to projects over time

Mill provides many of the same things Bazel does: automatic caching,
parallelization, sandboxing, extensibility. Mill can already work
with a wide variety of programming languages, from JVM languages like
Java/Scala/Kotlin to Typescript and Python. Mill's features are not
as highly-scalable as their Bazel equivalents, but they are provided
in a lighter-weight, easier-to-use fashion suitable for organizations
with less than 1,000 engineers who cannot afford the O(1 person
decade) it takes to adopt Bazel in their organization.

For most companies, their problems with Bazel aren't its scalability
or feature set, but its complexity. While Mill can never compete with
Bazel for the largest-scale deployments by its most sophisticated
users, the bulk of users operate at a somewhat smaller scale and need
something easier than Bazel. Mill could be that easy monorepo build
tool for them to use.

 Next Steps For Mill Going Forward

10 years ago React.js democratized front-end Web UIs: what previously
took intricate surgery to properly wire up event handlers and UI
mutations in three separate languages became a straightforward task
of naively returning the UI you want to render. Previously
challenging tasks (e.g. "make a loading bar that is kept in sync with
the text on screen as a file is uploaded") became trivial, and now
anyone can probably fumble through a basic interactive website
without getting lost in callback hell.

I think Mill has a chance to do the same thing for build systems.
Like Web UIs 10 years ago, configuring and maintaining a build-system
often requires juggling multiple different templating/config/
scripting languages in an intricate dance of callbacks and filesystem
mutations. Like React.js, Mill collapses all this complexity, letting
you write naive "direct-style" code in a single language while
getting all the benefits of caching and parallelism, making
previously challenging build pipelines implementations trivial.

Fundamentally, there are holes in the build-tool market that are not
well served: the Java folks deserve something more modern than Maven
or Gradle, and the Monorepo folks need something easier to use than
Bazel. I think Mill has a decent shot at occupying each of these two
niches, and even if it is only able to succeed in one that would
still be significant.

Going forward, I expect to pursue both paths: Mill as a better Java
build tool, Mill as an easier Monorepo build tool, leveraging Mill's
unique direct style to make the build tool experience easier. If
anyone out there is interested in collaborating on improving the
build tool, reach out on Github or Discord and let me know!!

The Mill Build Engineering Blog Strategies for Efficiently
Parallelizing JVM Test Suites