June 10, 2024

Whirlwind Tour of Mill Build

Lets start a whirlwind tour. Last time we stopped at a basic Java app. Yes, Mill does of course support building Scala projects as well, but Mill works well for other builds as well.

src/HelloWorld.java:
class HelloWorld{
    public static void main(String[] args){
        System.out.println("I'm built with Mill")
    }
}
build.sc:
import mill._
import scalalib._

object demo extends RootModule with JavaModule{
}

Add Dependencies!

The JavaModule supports adding Maven Dependencies. The word 'ivy' comes from the Apache Ivy dependency manager. However, Mill uses another library to fetch Maven dependencies meanwhile.

build.sc:
import mill._
import scalalib._

object demo extends RootModule with JavaModule{
    override def ivyDeps = Agg(
        ivy"org.json:json:20240303",
        ivy"com.h2database:h2:2.2.224")
}
Angler fish as transitive dependencies
Figure 1. Shiny dependencies!

Splitting Code Into Modules

I said Mill makes sense for complicated builds. In Mill you use modules to split things up. A module is any instance of a Module sub-class. For example, we split up our project into an api and backend like this:

build.sc
object demo extends RootModule{
   object api extends JavaModule{
    override def ivyDeps = Agg(ivy"org.json:json:20240303")
   }
   object backend extends JavaModule{
    override def moduleDeps = Seq(demo.api)
    override def ivyDeps = Agg(ivy"com.h2database:h2:2.2.224")
   }
}

This creates two modules. By default, the source code also moves to the sub-directory with the same name as the sub-module. So, the api module source will be in the 'api' directory, and the 'backend' module source in the backend directory. This will also be reflected in the command line. Each module will get its own sub-commands. In our example:

$ mill resolve _ # Show the top level tasks
api     # our api module
backend # our backend module
clean   # built-in
init    # built-in
# more built-in tasks

$ mill resolve api._ # Show the tasks for our api module
api.allIvyDeps
api.allSourceFiles
# ...

$ mill resolve backend._ # Show the tasks for our backend module
backend.allIvyDeps
backend.allSourceFiles
# ...

Note: The RootModule places its commands at the top level. If we use another module type, commands would be under demo._ etc. Modules build a strict tree hierarchy:

build.sc
object demo extends Module{
   // the api and backend, as above
}

Then it does result in:

$ mill resolve _ # Show the top level tasks
clean # built-in
demo  # our module
init  # built-in
# more built-in tasks

$ mill resolve demo._ # Show the tasks for our demo module
demo.api
demo.backend
# ...

$ mill resolve demo.api._ # Show the tasks for our api module
demo.api.allIvyDeps
demo.api.allSourceFiles
# ...

You also can list things across module boundaries by using a double underscore:

$ mill resolve demo.__ # Show everything from the demo module and its sub-modules
demo
demo.api
demo.api.allIvyDeps
# long list of demo.api tasks.
demo.backend
demo.backend.allIvyDeps
demo.backend.allSourceFiles
# long list of demo.backend tasks.

Anyway, in the rest of the this blog post we got back to use a RootModule

Adding Tests

So far Mill looks quite similar to Maven (with Scala instead of XML) or SBT. Testing is where starts to Mill diverge, because Mill has no 'scope' concept. Missing scopes is a 'feature' [1] to keep the amount of concepts low.

What you do in Mill is to create a tests module:

build.sc
import mill.scalalib.TestModule.Junit5 // Import JUnit 5 support
// ...existing stuff
object demo extends RootModule{
   object api extends JavaModule{
    override def ivyDeps = Agg(ivy"org.json:json:20240303")
    object tests extends JavaModuleTests with Junit5{}
   }
   // The backend module as above
}

Where should the test source be? Again, we can ask Mill for that:

./mill show api.tests.sources
[
  "ref:v0:c984eca8:/home/roman/dev/private-dev/mill-demo/api/tests/src"
]

But what if you migrate and maybe have a Maven directory structure? Then use the MavenModule module instead:

build.sc
object demo extends RootModule{
  object api extends MavenModule {
    override def ivyDeps = Agg(ivy"org.json:json:20240303")
    object tests extends MavenModuleTests with Junit5{}
  }
   // The backend module as above
}
./mill show api.tests.sources
[1/1] show > [1/1] api.tests.sources
[
  "ref:v0:c984eca8:/home/roman/dev/private-dev/mill-demo/api/src/test/java"
]

Of course, you can overwrite it yourself for any other directory: .build.sc

object api extends JavaModule{
  override def ivyDeps = Agg(ivy"org.json:json:20240303")
  object tests extends JavaModuleTests with Junit5{
    override def sources = T.sources(api.millSourcePath / "test-src")
  }
}

How do you run the tests? You call the test task on that module:

$ ./mill api.tests.test
[62/62] api.tests.test
Test run started (JUnit Jupiter)
Test MyTest#myTest() started
Test MyTest#myTest() finished, took 0.01
....

For tests you might also want to use the --watch flag. That flag will start watching the source code, and re-run the specified task if the source changes:

$ ./mill --watch api.tests.test
[62/62] api.tests.test
Test run started (JUnit Jupiter)
# ... test results ...
Test run finished: 0 failed, 0 ignored, 2 total, 0.052s
Watching for changes to 11 paths and 7 other values... (Enter to re-run, Ctrl-C to exit)
# ..this changed, then:
[55/62] api.tests.compile
# compiles changes stuff, reruns tests etc
Test run finished: 1 failed, 0 ignored, 2 total, 0.056s
1 targets failed
api.tests.test 1 tests failed:
  MyTest myTest()
Watching for changes to 11 paths and 7 other values... (Enter to re-run, Ctrl-C to exit)

Adding a front end build

Let’s say we need a web front-end, based on NodeJS tooling.

New Adventures in NodeJS land!
Figure 2. New Adventures in NodeJS land!

We want to bundle it with our backend. First, we create a build for the front end, which runs the external tooling:

build.sc:
object `front-end` extends Module{
  import os.Inherit
  // Declare the sources etc, This will allow mill do skip the front-end build if nothing changes
  def sources = T.sources(millSourcePath / "src") // The 'millSourcePath' is by default ./front-end, like the module name.
  def nodePackages = T.sources(
    millSourcePath / "package.json",
    millSourcePath / "package-lock.json")

  def installNodeModules = T{
    // npm will implicitly use the package.json / package-lock.json. Therefore, it depends on these source files
    val config = nodePackages()
    os.proc("npm","ci").call(cwd=millSourcePath, stdout = Inherit)
    millSourcePath/"node_modules"
  }

  /**
   * Build our front end with our favorite framework of the month ;)
   */
  def build = T{
    val nodeModules = installNodeModules() // install node packages
    val sourceCode = sources() // Depends on the source code
    os.proc("npm","run","build-frontend").call(cwd=millSourcePath, stdout = Inherit)
    PathRef(millSourcePath/"public")
  }
}

This allows use to build the front-end with Mill:

$ ./mill front-end.build
[2/4] front-end.installNodeModules

added 1 package, and audited 2 packages in 532ms

[4/4] front-end.build

> build-frontend
> mkdir -p ./output && echo 'Building front end...' && sleep 10 && echo "building hi" > ./output/output.js && echo 'Front end done'

Building front end...
Front end build complete!

The best thing is that Mill will track changes to the source and our package definitions. So, if we run the front-end build again, it will be near instant!

$ time ./mill front-end.build
[3/4] front-end.sources
________________________________________________________
Executed in  358.69 millis    fish           external
   usr time  321.47 millis  403.00 micros  321.06 millis
   sys time   65.86 millis  206.00 micros   65.66 millis

Based on this, we then easily integrate it into the rest of our build. Let’s package the front-end as part of the backend, which will serve it as static resources:

build.sc:
object backend extends JavaModule{
  override def moduleDeps = Seq(demo.api)
  override def ivyDeps = Agg(ivy"com.h2database:h2:2.2.224")

  override def resources = T{
    val frontEndResult = `front-end`.build()
    super.resources().appended(frontEndResult)
  }
}

Now if you build your backend, the front-end will be built and packaged into it. Repeating a build step will without changing source code will be nearly instant, because the cached results from from previous steps are used.

IDE Support

Let’s get some IDE support. That is certainly where Mill is weak, compared to Maven. Anyway, there is some support via the BSP or direct support for IntelliJ:

# Generate a IntelliJ idea project
$ ./mill mill.idea.GenIdea/idea

# Or, create a BSP project, which can be imported into an IDE:
./mill mill.bsp.BSP/install

If you are using IntelliJ, I would try the mill.idea.GenIdea/idea first. However, I had to use the mill.bsp.BSP/install on complex builds, and ran into some trouble there with IntelliJ idea. If you are using the BSP import, I would consider changing the mill start script to fix the JDK used for the mill command:

mill start script
#!/usr/bin/env sh

# For example, using sdk man to enforce a JDK.
# Otherwise if BSP and your CLI commands use different SDK, you will invalide each others caches.
sdk env
# Original mill script from down below

Inspecting Things

./mill inspect <task> will tell you details about the task. It tells the task name, the source file and line where it is defined, and what its inputs are. Some tasks have arguments, these show up as well. The explanation is the JavaDoc comment on the given task. So, if you add JavaDoc on a task, it will sho

$ ./mill inspect front-end.build
front-end.build(build.sc:39)
    Build our front end with our favorite framework of the month ;)

Inputs:
    front-end.installNodeModules
    front-end.sources

$ ./mill inspect backend.ivyDepsTree
    Command to print the transitive dependency tree to STDOUT.

    --inverse                Invert the tree representation, so that the root is on the bottom val
                             inverse (will be forced when used with whatDependsOn)
    ...

Inputs:
    backend.transitiveIvyDeps
Inspect your Mill ;)
Figure 3. Inspect your Mill ;)

There is also the ./mill show visualizePlan <task> command, which generate the dependencies in your build plan as image.

$ ./mill show visualizePlan front-end.build
# Will give you a paths to the task graph in different file formats

In our example:

Our Basic Task Graph
Figure 4. Task Graph

Profile and Concurrent Builds

Your builds start to take longer in CI. How does Mill help with long builds?

Mill supports running the tasks concurrently. By default, only 1 task runs at a time. You can specify with the -j <number> flag how many tasks should run concurrently. Zero has the meaning of running as many tasks as there are CPUs on your system:

Mill also gives you a profile you can analyze. There is a ./out/mill-chrome-profile.json file, which you can analyze with the Chrome profiler:

  1. Open Chrome/Chromium

  2. Type in chrome://tracing/

  3. Open that file

Here’s an example from a real build:

Build Profile
Figure 5. Build Profile

Run Mill on Build Servers

Last, if you run Mill on your build server, GitHub actions, whatever: Add --no-server to the command line. Mill spawns a background server task that keeps hanging around by default. You do not want that on a build server.

Summary:

So, we saw that with Mill, we can: - Download dependencies easily - Split things up easily into modules - Tests are usually put into separate modules. --watch helps you rerun a build step on each change. - Integrating external tools for the build isn’t hard - Build results are cached and reused - There are command lines to give you information about your tasks, like inputs, arguments, task graphs, etc - Concurrent builds and profiling are integrated - Use --no-server on your CI system

Next time we take a step back and look at the fundamentals of Mill, tasks and modules.

In this series:
1. If you ever debugged issues with SBTs scoping, you might be thankful for the "not having scopes" feature
Tags: Scala Mill Build Java Development