July 15, 2024

Mill Basic Building Blocks

Lets explore the Mill building blocks. We start with a hello world:

import mill._
import scalalib._

def buildGreetings = T{ // T stands for Target ;)
  val greeting = "Hello Mill!"
  val outFile = T.dest / "greeting.txt" // Note, we use T.dest as our output directory
  os.write(outFile, greeting)
  println(s"File is in $outFile")
  PathRef(outFile)
}

Then we build it:

$ ./mill buildGreetings
...
[1/1] buildGreetings
File is in /home/roman/dev/private-dev/mill-demo/out/buildGreetings.dest/greeting.txt
$ cat /home/roman/dev/private-dev/mill-demo/out/buildGreetings.dest/greeting.txt
Hello Mill!

That works. Note that we used T.dest to give each target its own directory to write to disk without trampling over other targets. The location in the 'out' directory is always the task names with a .dest suffix.

Ok, but in the real world we have source code! Lets read the name for the greeting from source code.

def readName() = os.read(os.pwd / "name.txt")

def buildGreetings = T{
  val name = readName()
  val greeting = s"Hello $name!"
  val outFile = T.dest / "greeting.txt"
  os.write(outFile, greeting)
  println(s"File is in $outFile")
  PathRef(outFile)
}

Ah, now we can change the name, right?

echo Gamlor > name.txt
$ ./mill buildGreetings
...
File is in /home/roman/dev/private-dev/mill-demo/out/buildGreetings.dest/greeting.txt
$ cat /home/roman/dev/private-dev/mill-demo/out/buildGreetings.dest/greeting.txt
Hello Gamlor!
$ echo Roman > name.txt
$ ./mill buildGreetings
...
$ cat /home/roman/dev/private-dev/mill-demo/out/buildGreetings.dest/greeting.txt
Hello Gamlor! #OOhh, no!

Nope! When we change the name the build output is stuck.

The reason is that Mill only re-builds stuff when the inputs did change. Inputs for a Mill tasks are results from other tasks. However, if you try to turn readName into a task, it still will not work:

def readName: Target[String] = T {os.read(os.pwd / "name.txt")}

def buildGreetings = T {
  val name = readName()
  // ... existing code from before
 }

You need to tell Mill that a something might change, like source code ;)

Track Changes With Source Tasks

The most common thing for a build tool is that source code changed and therefore some things must be rebuild. For that Mill has a T.source and T.sources target. It declares files as source files. When a file changes, the targets depending on will be rebuild, otherwise not.

def nameSource: Target[PathRef] = T.source(os.pwd / "name.txt")

def readName: Target[String] = T {
  os.read(nameSource().path)
}

// ... existing code from before

Now the build works as you expect. ./mill buildGreetings run when the name.txt changes, otherwise it will use the cached results.

Note: Mill creates an actual hash of the content over the source code (PathRef.sig). If the path points to a directory, the hash is build over the whole directory content, to track all files in it. You’ll see the hash if you run the source code target.

$ ./mill show nameSource
"ref:v0:fdf27884:/home/roman/dev/private-dev/mill-demo/name.txt"
Luckily Digital Artifacts Do Not Get Stale
Figure 1. Luckily Digital Artifacts Do Not Get Stale

Lower level change tracking.

The T.source/T.sources covers most cases. However, sometimes you want to track some other value. For that there is a T.inputs target. It’s main use is to fetch some value relevant to the build from the environment. For example get the build version number which then is embedded into the build results / artifacts:

def buildVersion = T.input {
  os.proc("git","describe")
    .call()
    .out.text()
}

def buildGreetings = T {
  val name = readName()
  val version = buildVersion()
  val greeting = s"Hello $name! from version ${version}"
  // ... existing code from before
}

Commands & Tasks

So far we saw Targets, which is Task type you usually use. Targets only run when the input changes. However, some things you want to run every time: Like a utility method, or running test to investigating flaky tests. Commands are intended to be invoked from the CLI and do run every time.

def printGreetings() = T.command {
  val greeting = buildGreetings()
  val fileContent = os.read(greeting.path)
  println(fileContent)
}
def printAdvancedGreetings(
                           @arg(doc = "The Style of the greeting. Defaults to 'plain'. Supports ....")
                           style:String,
                           @arg(doc = "Font Size, ignored for the 'plain' style")
                           fontSize: Int = 13) = T.command{
  println(s"Style is ${style} and ${fontSize}")
}

Commands can use other tasks results. Plus they can have arguments declared, which then are passed via the command line:

$ ./mill printGreetings
[5/5] printGreetings
Hello Roman
! from version 2024.07.01-1-g788291f

$ ./mill printAdvancedGreetings
Missing argument: --style <str>
Expected Signature: printAdvancedGreetings
  --fontSize <int>  Font Size, ignored for the 'plain' style
  --style <str>     The Style of the greeting. Defaults to 'plain'. Supports ....

$ ./mill printAdvancedGreetings --style plain
[1/1] printAdvancedGreetings
Style is plain and 13

A task is somewhat similar to a command, but it is intended as a internal building block to a build more targets. For example a task which compiles things, and then is invoked by a target to build a specific part. It cannot be directly invoked from the CLI.

def buildTask(source:Task[PathRef]) = T.task{
   println("Compiling")
   os.copy(source().path, T.dest / "compiled.bin")
}

def compile = T{
  buildTask(sources)
}
def testSources = T.source( os.pwd / "test.txt")
def testCompile = T{
  buildTask(testSources)
}

Super Specialized Tasks: Workers & Persistent Targets

I’m just mentioning these tasks here, but won’t provide examples.

By default Mill ensures that each Target gets an empty T.dest directory when the Target task starts running. This ensures that each task runs cleanly and doesn’t break on left over pieces from previous runs. A persistent task will not have its directory cleaned between runs. This is for specialized tasks do implement finer grained cache control and manage their directory themself.

Even more advanced are worker tasks. Workers keep running between builds and keep running. These are used for example to keep for complicated compilers warm in memory or run processes like dev-servers which should stay up.

Task Overview

Here is the task overview again.

Task Overview

Modules!

Tasks would be enough ti building everything. To run builds, Mill only cares about the tasks. However, with only tasks it is hard to organize builds, as the number of tasks increases.

Here, Modules come in. Modules allow you to structure the build.

object `mill-demo` extends Module{
  def source = T.source(millSourcePath)
  object backend extends Module{
    object customers extends Module{
      def source = T.source(millSourcePath)
    }
    object `shopping-card` extends Module{
      def source = T.source(millSourcePath)
    }
  }
  object website extends Module{
    object mobile extends Module{
      def source = T.source(millSourcePath)
    }
    object desktop extends Module{
      object `admin-site` extends Module{
        def source = T.source(millSourcePath)
      }
      object site extends Module{
        def source = T.source(millSourcePath)
      }
    }
  }
}

What a module does is to give the build a tree structure, to group relevant things together. The target/command name is always the path down the module names, separated by dots:

$ ./mill resolve __
clean
init
inspect
mill-demo
mill-demo.backend
mill-demo.backend.customers
mill-demo.backend.customers.source
mill-demo.backend.shopping-card
mill-demo.backend.shopping-card.source
mill-demo.source
mill-demo.website
mill-demo.website.desktop
mill-demo.website.desktop.admin-site
mill-demo.website.desktop.admin-site.source
mill-demo.website.desktop.site
mill-demo.website.desktop.site.source
mill-demo.website.mobile
mill-demo.website.mobile.source
path
...

This is the also used for the default millSourcePath of a module. It can be overridden of course.

./mill show mill-demo.source
"ref:v0:c984eca8:/home/roman/dev/private-dev/mill-demo/mill-demo"
./mill show mill-demo.backend.customers.source
"ref:v0:c984eca8:/home/roman/dev/private-dev/mill-demo/mill-demo/backend/customers"
./mill show mill-demo.website.desktop.site.source
"ref:v0:c984eca8:/home/roman/dev/private-dev/mill-demo/mill-demo/website/desktop/site"

This is also applied to the targets output directory (T.dest):

  • mill-demo.compile → out/mill-demo/compile.dest

  • mill-demo.backend.compile → out/mill-demo/backend/compile.dest

  • etc.

Most of the time you want the top level module to have all its target at the top level. For that, declare it as the RootModule:

object `mill-demo` extends RootModule{
// as before
}

This results in the tasks of the root module being at the top without any module prefix:

$ ./mill resolve __
backend
backend.customers
backend.customers.source
backend.shopping-card
backend.shopping-card.source
...

Inherited Modules

Ok, with tasks and modules we are able to structure our build and have a the task based incremental build. However, how do we get any convenience and standard builds? By inheriting from Module traits! (Java Devs: trait are more or less interfaces with default methods).

Like this:

object `mill-demo` extends RootModule with JavaModule {
}

That results in all the default tasks for a Java build:

$ ./mill resolve __
allIvyDeps
allSourceFiles
allSources
artifactId
artifactName
artifactNameParts
artifactSuffix
assembly
...

Because these are regular traits added to a module, you can do the usual Scala things. You can override things and combine multiple traits, etc. For example adding resources directories to a Java build is plain override away:

object `mill-demo` extends RootModule with JavaModule {
  def myMagicGeneratedFile = T{
    os.write(T.dest/"build-time-resource.txt", "some file generated at compile time")
    PathRef(T.dest)
  }
  override def resources =T{
    val buildTimeResource = myMagicGeneratedFile()
    super.resources()
      .appended(PathRef(millSourcePath/"my-special-resources"))
      .appended(buildTimeResource)
  }
}

You can create your own traits for your builds to be reused:

trait OurJavaBackendModule extends JavaModule {
 // overrides and extra targets we use for our Java Backends
}
trait OurPHPModule extends Module{
  // targets etc relevant for a PHP module
}
object `mill-demo` extends RootModule with JavaModule {
  object `search-service` extends OurJavaBackendModule{

  }
  object catalogue extends OurPHPModule{

  }
  object `admin-panel` extends OurPHPModule
}

In fact, the Mill provided modules are defined exactly like this. I tend to look up the standard modules on how to do things when I want to get a similar behavior for my own modules.

.Inherit a great build
Figure 2. Inherit a great build

Summary:

We learned:

  • Mill is based on tasks, where there are a flavors for specific use cases.

  • Modules give the build structure

  • Because Modules are build our of regular Scala traits, inheritance with overrides is used for providing reusable build blocks.

Tags: Scala Mill Build Java Development