November 4, 2019

OpenJDK's Flight-Recorder and Mission Control

Flight Recorder is a low overhead profiler and diagnostics events recorder for the Hotspot JVM. Java Mission Control is a software to visualize these events. It has been included for years in the Oracle JDK but used to be a commercial feature. To use it you needed to add -XX:+UnlockCommercialFeatures -XX:+FlightRecorder flags and you were not allowed to use it in production unless you paid a license.

Oracle donated the code to the OpenJDK project and it is now included in OpenJDK 11 or newer builds. However, if you look at your JDK there is no Mission Control, so where it is? It is part of the OpenJDK but is built separately. You can download binaries from AdoptOpenJDK or Azul, which rebranded it as Zulu Mission Control.

// Example, after downloading the AdoptOpenJDK version
mkdir -p ./mission-control && tar xfv org.openjdk.jmc-linux.gtk.x86_64.tar.gz -C ./mission-control
cd ./mission-control
./jmc

If your app runs on Java 8 without flags or on a pure OpenJDK 8 release you get an error if you try to attach the flight recorder.

Java 8 needs flags to work
Figure 1. Java 8 needs Oracle builds and extra flags
No Flight Recorder in OpenJDK 8
Figure 2. No Flight Recorder in OpenJDK 8

On Java 11 it works:

On Java 11 it works
Figure 3. On Java 11 it works
No Flight Recorder in OpenJDK 8
Figure 4. Flight Recorder built in OpenJDK 11+

Start Recording With Command Line Flags

The flight recorder is suited for running in production system due to its low overhead. For that, you can start the flight recorder with you JVM. For example.

java -XX:StartFlightRecording=name=my-recording,maxage=2h,disk=true,filename=/tmp/jfr/my-recording.jfr,dumponexit=true ....

The name gives a nice name to the recording. maxage specified how much data is kept before it’s discarded. disk specifies that we want to store the data to disk. The filename specifies where the recording is stored. The dumponexit flag ensures the JVM flushes the recording to disk on exit.

These flags don’t seem to have a good place where they are documented. The documentation out there often refers to flags that do not exist anymore. The best way to get the list of the flags and their meaning seems to use the jcmd command. For that start a JVM running your app. Then use the jcmd command to ask for help on the 'JSR.start' command.

gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd <pid-of-java-11-or-newer-process> help JFR.start
1873:
JFR.start
Starts a new JFR recording

Impact: Medium: Depending on the settings for a recording, the impact can range from low to high.

Permission: java.lang.management.ManagementPermission(monitor)

Syntax : JFR.start [options]

Options: (options must be specified using the <key> or <key>=<value> syntax)
name : [optional] Name that can be used to identify recording, e.g. \"My Recording\" (STRING, no default value)
settings : [optional] Settings file(s), e.g. profile or default. See JRE_HOME/lib/jfr (STRING SET, no default value)
delay : [optional] Delay recording start with (s)econds, (m)inutes), (h)ours), or (d)ays, e.g. 5h. (NANOTIME, 0)
duration : [optional] Duration of recording in (s)econds, (m)inutes, (h)ours, or (d)ays, e.g. 300s. (NANOTIME, 0)
disk : [optional] Recording should be persisted to disk (BOOLEAN, no default value)
filename : [optional] Resulting recording filename, e.g. \"/home/user/My Recording.jfr\" (STRING, no default value)
maxage : [optional] Maximum time to keep recorded data (on disk) in (s)econds, (m)inutes, (h)ours, or (d)ays, e.g. 60m, or 0 for no limit (NANOTIME, 0)
maxsize : [optional] Maximum amount of bytes to keep (on disk) in (k)B, (M)B or (G)B, e.g. 500M, or 0 for no limit (MEMORY SIZE, 0)
dumponexit : [optional] Dump running recording when JVM shuts down (BOOLEAN, no default value)
path-to-gc-roots : [optional] Collect path to GC roots (BOOLEAN, false)

JCMD and JFR.start, JFR.dump, JFR.stop

JCMD allows to start, dump and stop a the Java Flight recorder on a running VM. Start with listing the running JVM processes:

gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd -l
1873 com.intellij.idea.Main
14452 clojure.main -i /tmp/form-init9389737953409452528.clj
5237 org.jetbrains.kotlin.daemon.KotlinCompileDaemon --daemon-runFilesPath /home/gamlor/.local/share/kotlin/daemon --daemon-autoshutdownIdleSeconds=7200 --daemon-compilerClasspath /home/gamlor/progs/idea/plugins/Kotlin/kotlinc/lib/kotlin-compiler.jar
15240 org.jetbrains.jps.cmdline.Launcher /home/gamlor/progs/idea-IU-192.6262.58/lib/oro-2.0.8.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/netty-resolver-4.1.38.Final.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/platform-api.jar:/home/gamlor/progs/idea/plugins/java/lib/maven-repository-metadata-3.3.9.jar:/home/gamlor/progs/idea/plugins/java/lib/plexus-utils-3.0.22.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/httpcore-4.4.11.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/forms-1.1-preview.jar:/home/gamlor/progs/idea/plugins/java/lib/jps-builders-6.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/lz4-java-1.6.0.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/protobuf-java-3.5.1.jar:/home/gamlor/progs/idea/plugins/java/lib/aether-dependency-resolver.jar:/home/gamlor/progs/idea/plugins/java/lib/maven-artifact-3.3.9.jar:/home/gamlor/progs/idea/plugins/java/lib/aether-transport-file-1.1.0.jar:/home/gamlor/progs/idea/plugins/java/lib/jps-builders.jar:/home/gamlor/progs/idea-IU-192.6262.58/lib/commons-logging-1.2.jar:/h
15866 jdk.jcmd/sun.tools.jcmd.JCmd -l
14523 clojure.main -i /tmp/form-init3939668947907373363.clj -m build
2334 org.jetbrains.idea.maven.server.RemoteMavenServer36
15247 info.gamlor.testing.Main2

Then, use the PID to start the Java Flight recorder. There are three main commands: JFR.start, JFR.dump and JFR.dump. If you want see all available command use jcmd <pid> help. For details on a command use jcmd <pid> help <command>. Anyway, let’s start recording:

gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd 15247 JFR.start name=our-recording
15247:
Started recording 2. No limit specified, using maxsize=250MB as default.

Use jcmd 15247 JFR.dump name=our-recording filename=FILEPATH to copy recording data to file.

Once the recording is running we can dump the current recording to a file:

gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd 15247 JFR.dump name=our-recording filename=/home/gamlor/latest-recording.jfr
15247:
Dumped recording "our-recording", 3.2 MB written to:

/home/gamlor/latest-recording.jfr

And to stop the recording using JFR.stop

gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd 15247 JFR.stop name=our-recording
15247:
Stopped recording "our-recording".

Settings / Profiles

When you start a flight recording with the Java Mission Control you have all these options on what you want to record. There are two built-in settings, default' which is the default and `profile which does more detailed profiling with more overhead.

To create your own you copy one of these settings files from JAVA_HOME/lib/jfr/default.jfc to another location and edit it. Alternatively you can use Java Mission Controls Template Manager (Window→Flight Recorder Template Manager) to create a profile and export it to a file. Once you have your template file, you can use it in the command line or JFR.start command like this:

java -XX:StartFlightRecording=name=my-recording,maxage=2h,disk=true,filename=/tmp/jfr/my-recording.jfr,dumponexit=true,settings=/home/gamlor/my-profile-settings.jfc ....
gamlor@gamlor-t470p ~> $JAVA_HOME/bin/jcmd 17578 JFR.start name=my-recording settings=/home/gamlor/my-profile-settings.jfc
17578:
Started recording 3. No limit specified, using maxsize=250MB as default.

Use jcmd 17578 JFR.dump name=my-recording filename=FILEPATH to copy recording data to file.

Happy Flight Recording!

Happy Flight Recording. I hope your project already runs on Java 11 or newer and you can use flight recording out of the box!

Tags: Java