April 25, 2017

Deploying JVM in tiny containers, be careful!

I`ve deployed my https://rohrli.gamlor.info service into the smallest possible 2017-04-25-jvm-in-containers[Triton] container. Except me, noone is going to use it anyway -. This container has 128Mbyte of memory, and 1/16th CPU time. The OpenJDK JVM is known for consuming tons of memory. And it is somewhat true. However, the JVM also runs in a small container. In this blog post I give some recommendations. The problems exists in small Docker and Triton containers. I show everything for both.

Chrome and JVM Eating Memory
Figure 1. Chrome & JVM eat memory generously

Web App Example

The web app is just a ‘Hello World’ app. For better explanation, it also consumes tons of memory. I created the example in Scala, but the other JVM languages have the same issue. The source code for the example is here.

Hello-Basics.scala
/**
 * Snippet, memory limits
 */
// Limit the amount of items in memory for this example.
// So, max ~192*256k=48MBytes
val maxMemoryItems = 192
val memoryPerItem = 256 * 1024
val memoryWasting = new ArrayBlockingQueue[Array[Byte]](maxMemoryItems)

/**
 * Snippet, request handling
 */
// On each request, consume some memory.
// And keep things in memory to simulate memory use
case "/" :: Nil => {
// On each request, consume some memory.
// And keep things in memory to simulate memory use
val chunkOfMemory = Array.ofDim[Byte](memoryPerItem)
var addedToQueue = memoryWasting.offer(chunkOfMemory)

    // Throw things away until we've space
    while (!addedToQueue) {
        memoryWasting.poll()
        addedToQueue = memoryWasting.offer(chunkOfMemory)
    }

    val createdSoFar = createdItemsCounter.incrementAndGet()
    response.getWriter.write(
        s"""
            |{
            |"title":"Hello World! Let's use some memory",
            |"created-overall":"$createdSoFar",
            |"queue-size":"${memoryWasting.size()}",
            |"":"Hello. I'm so memory hungry. kthxbye!"
            |}
        """.stripMargin)
}

We compile and package the program into a Docker container. The Docker file is simple:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

EXPOSE 8080
CMD java -jar /app/useless-demo-service.jar

The we create the image with build and run it with run. With regular Docker we restrict the memory with the -m 128m option to 128Mbyte. And with –cpus 0.065 we only give very little CPU time. When using Triton, the -m 128m option will find a sufficient large compute instance. Alternatively, you also can use –label com.joyent.package=g4-highcpu-128M to create a tiny instance. Once we started out container, we create some load with the watch and curl commands. Then, let`s watch the logs.

##
# Local docker run
##
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
# Start to follow the logs
sudo docker logs -f jvm-on-small-container
# Then, start the simple load, in another termial
watch -n 0.01 curl http://localhost:8080/

##
# Joyent Triton run
##
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
# Get the IP
docker inspect --format '{{ .NetworkSettings.IPAddress }}' jvm-on-small-container
# Start to follow the logs
docker logs -f jvm-on-small-container
# Then, start the simple load, in another termial
watch -n 0.01 curl http://{ip-from-above}:8080/

First Issue: Program Dies

The Triton instance won`t experience any issue. But the plain Docker instance (Linux 4.10, Docker 17.04) suddenly stops working. =( The program got killed.

[root@gamlor-manjaro useless-demo-service]# docker run -m 128m --cpus 0.065 -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
2017-04-20 12:58:20.248:INFO::main: Logging initialized @5900ms to org.eclipse.jetty.util.log.StdErrLog
2017-04-20 12:58:23.348:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT
2017-04-20 12:58:30.650:INFO:oejs.AbstractConnector:main: Started ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-04-20 12:58:30.748:INFO:oejs.Server:main: Started @16602ms
Killed

Hmmm….Why did it got killed? Aha…the memory is small, so we might ran out of memory and got killed. We can, while the container runs, inspect the memory usage of the JVM with jstat:

##
# Local docker run
##
# Get memory usage of the running java process
sudo docker exec jvm-on-small-container sh -c 'jstat -gccapacity `pgrep java`'
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
84480.0 1354240.0 84480.0 10240.0 10240.0 64000.0 169472.0 2708992.0 169472.0 169472.0 0.0 1056768.0 4480.0 0.0 1048576.0 384.0 1 0


##
# Joyent Triton run
##
# Get memory usage of the running java process
eval "$(triton env)"
docker exec jvm-on-small-container sh -c 'jstat -gccapacity `pgrep java`'
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
  2688.0  21824.0  21824.0 2176.0 2176.0  17472.0     5504.0    43712.0    43712.0    43712.0      0.0 1060864.0  12544.0      0.0 1048576.0   1536.0     15    69
Linux killing JVM
Figure 2. The penguin and whale are dangerous. Be careful!

jstat is a bit confusing. All numbers are in KByte. The JVM has a few memory pools and each memory pool has a minimum (--MN), maximum (--MX) and current size (--C).

So, inside the regular Docker container it create really big memory pools. On my computer (16GB, Linux 4.10, Docker 17.04), OGCMX (old generation heap) memory is about 2.5GByte. However, on the Triton instance it is about 40Mybte large. So, with a 2.5GByte pool, if it actually uses more than 128MByte, it will die. For now, Docker just kills processes which use too much memory.

However, the JVM sees all the hosts memory, so it creates giant memory pools. However, the Triton container lies more convincingly, so the JVM sees the containers memory sizes and creates a sufficient small memory pool.

Well, there a solution to this issue. We just configure a small memory size with the -Xmx option. Plus I configure the JVM such that when it runs out of memory, it dumps the memory and dies.

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m"

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -jar /app/useless-demo-service.jar

After doing this, the JVM runs in the Docker container without any issue.

So Many Threads

Now, we configured the memory properly. However, there maybe more things we can configure properly? So, let`s see what threads are running in the container with jstack:

##
# Local docker run
##
# Run second version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x000056473bc03800 nid=0x42 waiting on condition [0x00007fefe4931000]
"Attach Listener" #22 daemon prio=9 os_prio=0 tid=0x000056473bf72800 nid=0x41 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #21 prio=5 os_prio=0 tid=0x000056473bcd1800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-0@27b0949f-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x000056473bf75000 nid=0x24 runnable [0x00007fefe4b33000]
"qtp791452441-17" #17 prio=5 os_prio=0 tid=0x000056473bc2d800 nid=0x22 runnable [0x00007fefe8160000]
"qtp791452441-16" #16 prio=5 os_prio=0 tid=0x000056473bc2b000 nid=0x21 runnable [0x00007fefe8261000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000056473bc29000 nid=0x20 waiting for monitor entry [0x00007fefe8363000]
"qtp791452441-14-lowPrioritySelector" #14 prio=1 os_prio=0 tid=0x000056473bc8c000 nid=0x1f waiting for monitor entry [0x00007fefe8464000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x000056473bc8a000 nid=0x1e waiting for monitor entry [0x00007fefe8565000]
"qtp791452441-12-lowPrioritySelector" #12 prio=1 os_prio=0 tid=0x000056473bc86800 nid=0x1d waiting for monitor entry [0x00007fefe8666000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x000056473bd8a800 nid=0x1c runnable [0x00007fefe8766000]
"qtp791452441-10" #10 prio=5 os_prio=0 tid=0x000056473bd84800 nid=0x1b runnable [0x00007fefe8867000]
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000056473bc63800 nid=0x19 runnable [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000056473bc14000 nid=0x18 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x000056473bbf9000 nid=0x17 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x000056473bbea000 nid=0x16 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x000056473bbe7000 nid=0x15 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x000056473bbe4800 nid=0x14 runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x000056473bbb5000 nid=0x13 in Object.wait() [0x00007fefe92cb000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x000056473bbb2000 nid=0x12 in Object.wait() [0x00007fefe93cc000]
"VM Thread" os_prio=0 tid=0x000056473bba8000 nid=0x11 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000056473b9f0000 nid=0x9 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000056473b9f2000 nid=0xa runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000056473b9f3800 nid=0xb runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000056473b9f5800 nid=0xc runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x000056473b9f7000 nid=0xd runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x000056473b9f9000 nid=0xe runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x000056473b9fa800 nid=0xf runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000056473b9fc800 nid=0x10 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x000056473bc59800 nid=0x1a waiting on condition

##
# Joyent Triton run
##
# Run second version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"qtp791452441-44" #44 prio=5 os_prio=0 tid=0x000000000215e000 nid=0x12de4 runnable [0x00007fffe51f7000]
"qtp791452441-43" #43 prio=5 os_prio=0 tid=0x0000000000262000 nid=0x12aa4 waiting for monitor entry [0x00007fffe52fa000]
"qtp791452441-42" #42 prio=5 os_prio=0 tid=0x000000000068a000 nid=0x11808 waiting on condition [0x00007fffe7c7a000]
"Scheduler-1896277646" #41 prio=5 os_prio=0 tid=0x0000000000145000 nid=0x1176f waiting on condition [0x00007fffe4e34000]
"Attach Listener" #39 daemon prio=9 os_prio=0 tid=0x0000000000717800 nid=0x116e2 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #33 prio=5 os_prio=0 tid=0x0000000000716000 nid=0x10796 waiting on condition [0x0000000000000000]
"qtp791452441-32-acceptor-3@6dbc83d8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #32 prio=3 os_prio=0 tid=0x0000000000714800 nid=0x1082a runnable [0x00007fffe5cff000]
"qtp791452441-31-acceptor-2@1ac08473-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #31 prio=3 os_prio=0 tid=0x0000000000713000 nid=0x10829 waiting for monitor entry [0x00007fffe6000000]
"qtp791452441-30-acceptor-1@34228b2c-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #30 prio=3 os_prio=0 tid=0x000000000070d800 nid=0x10828 waiting for monitor entry [0x00007fffe67fe000]
"qtp791452441-29-acceptor-0@467e71fc-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #29 prio=3 os_prio=0 tid=0x000000000070b000 nid=0x10827 waiting for monitor entry [0x00007fffe6f72000]
"qtp791452441-28-lowPrioritySelector" #28 prio=1 os_prio=0 tid=0x0000000000695800 nid=0x10818 runnable [0x00007fffe72b3000]
"qtp791452441-27" #27 prio=5 os_prio=0 tid=0x000000000068d800 nid=0x10816 waiting on condition [0x00007fffe75f6000]
"qtp791452441-26-lowPrioritySelector" #26 prio=1 os_prio=0 tid=0x000000000068b800 nid=0x10815 runnable [0x00007fffe7937000]
"qtp791452441-24-lowPrioritySelector" #24 prio=1 os_prio=0 tid=0x0000000000687800 nid=0x10812 waiting for monitor entry [0x00007fffe7fbc000]
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x0000000000685800 nid=0x10810 waiting for monitor entry [0x00007fffe82fe000]
"qtp791452441-22-lowPrioritySelector" #22 prio=1 os_prio=0 tid=0x0000000000683800 nid=0x1080f waiting for monitor entry [0x00007fffe85ff000]
"qtp791452441-21" #21 prio=5 os_prio=0 tid=0x000000000067d000 nid=0x1080e runnable [0x00007fffe8b1d000]
"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00000000001ee800 nid=0x107c5 runnable [0x0000000000000000]
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00000000001e1800 nid=0x107c3 waiting on condition [0x0000000000000000]
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00000000001d7000 nid=0x107c1 waiting on condition [0x0000000000000000]
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00000000001d4000 nid=0x107bf waiting on condition [0x0000000000000000]
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00000000001b7800 nid=0x107bd waiting on condition [0x0000000000000000]
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00000000001ad800 nid=0x107bc waiting on condition [0x0000000000000000]
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00000000001a2000 nid=0x107bb waiting on condition [0x0000000000000000]
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x0000000000197800 nid=0x107b9 waiting on condition [0x0000000000000000]
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x0000000000172000 nid=0x107b8 waiting on condition [0x0000000000000000]
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x0000000000186800 nid=0x107b7 waiting on condition [0x0000000000000000]
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x000000000017c000 nid=0x107b6 waiting on condition [0x0000000000000000]
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x0000000000165800 nid=0x107b5 waiting on condition [0x0000000000000000]
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000000000010f000 nid=0x107b3 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x0000000000104800 nid=0x107b2 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00000000000f5000 nid=0x107b1 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x107b0 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x107af runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x107a3 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x107a1 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x1079d runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00000000001f3800 nid=0x107c6 waiting on condition

Howly! More than 30 threads are running. On my computer the regular Docker container has 8 GC task threads, and 4 CompilerThreads. On Triton there are 15! The JVM saw a lot of CPU cores and created many threads to take advantage of it. However, our container is restricted to 0.065 cores. Those many threads won`t help.

Triton shows all the cores
Figure 3. Triton isn`t a fun guy!

Of course, we can configure this:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m" \
    CPUS=1

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \
    -jar /app/useless-demo-service.jar

In this really small container the ParallelGC is not really needed and we can use the SerialGC. In case you use a bit larger container, you can use -XX:-XX:+ParallelGCThreads=$CPU for ParallelGC. Or -XX:G1ConcRefinementThreads=$CPU in case you use -XX:+UseG1GC. And we can use -XX:CICompilerCount=$CPU to reduce the compiler threads. However, it needs to be at least 2.

Still Many Threads

We still have many ‘acceptor’ and ‘selector’ threads. On Triton we’ve 4 acceptor and 4 selector threads:

##
# Local docker run
##
# Run third version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"Attach Listener" #20 daemon prio=9 os_prio=0 tid=0x0000564e7cd17000 nid=0x37 waiting on condition [0x0000000000000000]
"qtp791452441-19" #19 prio=5 os_prio=0 tid=0x0000564e7cd15000 nid=0x36 waiting on condition [0x00007fdcc863f000]
"qtp791452441-17-acceptor-0@77fe3be4-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x0000564e7cc25000 nid=0x34 runnable [0x00007fdcc8841000]
"DestroyJavaVM" #16 prio=5 os_prio=0 tid=0x0000564e7c922800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x0000564e7ce14000 nid=0x31 waiting for monitor entry [0x00007fdccbd6e000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x0000564e7ce12000 nid=0x30 runnable [0x00007fdccbe6e000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000564e7ce0f800 nid=0x2f waiting for monitor entry [0x00007fdccbf70000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000564e7ce0e000 nid=0x2e runnable [0x00007fdccc070000]
"qtp791452441-11-lowPrioritySelector" #11 prio=1 os_prio=0 tid=0x0000564e7ce0c000 nid=0x2d waiting for monitor entry [0x00007fdccc172000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000564e7ce0a000 nid=0x2b waiting for monitor entry [0x00007fdccc273000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000564e7ce07800 nid=0x2a runnable [0x00007fdccc373000]
"qtp791452441-8" #8 prio=5 os_prio=0 tid=0x0000564e7ce01000 nid=0x29 runnable [0x00007fdccc474000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000564e7c8c2800 nid=0xf runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000564e7c8b7800 nid=0xe waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x0000564e7c8ac800 nid=0xd waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x0000564e7c8aa800 nid=0xc runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000564e7c87d800 nid=0xb in Object.wait() [0x00007fdccccd4000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000564e7c873000 nid=0xa in Object.wait() [0x00007fdcccdd5000]
"VM Thread" os_prio=0 tid=0x0000564e7c869000 nid=0x9 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000564e7c8c5800 nid=0x10 waiting on condition


##
# Joyent Triton run
##
# Run third version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x000000000028c000 nid=0x732a waiting on condition [0x0000000000000000]
"qtp791452441-22" #22 prio=5 os_prio=0 tid=0x00000000006eb000 nid=0x7178 waiting on condition [0x00007fffe81fe000]
"qtp791452441-21-acceptor-3@161ab1a8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #21 prio=3 os_prio=0 tid=0x00000000006ea000 nid=0x7177 waiting for monitor entry [0x00007fffe84ff000]
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00000000001df000 nid=0x7109 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-2@70b4bcd9-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x00000000006e7000 nid=0x7175 waiting for monitor entry [0x00007fffe8800000]
"qtp791452441-18-acceptor-1@57033676-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #18 prio=3 os_prio=0 tid=0x0000000000747800 nid=0x7174 waiting for monitor entry [0x00007fffe97f6000]
"qtp791452441-17-acceptor-0@67e0dd84-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x00000000006e4800 nid=0x7173 runnable [0x00007fffe8ffe000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000000000066b000 nid=0x7159 waiting for monitor entry [0x00007fffe9af7000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x000000000066c800 nid=0x7157 runnable [0x00007fffe9df7000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000000000668800 nid=0x7156 waiting for monitor entry [0x00007fffea0f9000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000000000666800 nid=0x7155 runnable [0x00007fffea3f9000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x0000000000664800 nid=0x7154 runnable [0x00007fffea6fa000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000000000662800 nid=0x7153 waiting for monitor entry [0x00007fffea9fc000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000000000660800 nid=0x7152 runnable [0x00007fffeacfc000]
"qtp791452441-8-lowPrioritySelector" #8 prio=1 os_prio=0 tid=0x0000000000659800 nid=0x7151 waiting for monitor entry [0x00007fffeaffe000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000000000130800 nid=0x7120 runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000000000104000 nid=0x711e waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x711d waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x711c runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x7115 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x7113 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x710f runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000000000128800 nid=0x7121 waiting on condition

Those threads are created by the web server, so you have to configure the web server you are using. In this example Jetty is used, and we configure it via simple Java properties:

DemoService.scala
// Version 1 did just start Jetty
// val server = new Server(8080)
// It creates many acceptors, selectors and threads on big machines, even when running in small container
// So, let's explicitly configure it.
val minThreads = System.getProperty("jetty.min-threads", "8").toInt
val maxThreads = System.getProperty("jetty.max-threads", "200").toInt
var threadPool = new QueuedThreadPool(minThreads, maxThreads)

val server = new Server(threadPool)
val acceptorCount = System.getProperty("jetty.acceptor-threads", "-1").toInt
val selectorCount = System.getProperty("jetty.selector-threads", "-1").toInt
server.setConnectors(Array(new ServerConnector(server, acceptorCount, selectorCount)))

server.setHandler(new OurHandling)
server.start()

And also change the Docker file:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m" \
    CPUS=1\
    ACCEPTORS=1 \
    SELECTORS=1\
    JETTY_MIN_THREADS=4 \
    JETTY_MAX_THREADS=20

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \
    -Djetty.acceptor-threads=$ACCEPTORS \
    -Djetty.selector-threads=$SELECTORS \
    -Djetty.min-threads=$JETTY_MIN_THREADS \
    -Djetty.max-threads=$JETTY_MAX_THREADS \
    -jar /app/useless-demo-service.jar

And now, it starts to look fine, right?

##
# Local docker run
##
# Run forth version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:4
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"DestroyJavaVM" #13 prio=5 os_prio=0 tid=0x000055927f487000 nid=0x8 waiting on condition [0x0000000000000000]
"qtp2096171631-12-acceptor-0@38ffe3b-ServerConnector@457e2f02{HTTP/1.1,[http/1.1]}{0.0.0.0:45611}" #12 prio=3 os_prio=0 tid=0x000055927f4b9800 nid=0x30 runnable [0x00007ff4bbd45000]
"qtp2096171631-11-lowPrioritySelector" #11 prio=1 os_prio=0 tid=0x000055927f4b4800 nid=0x2f waiting for monitor entry [0x00007ff4bbe46000]
"qtp2096171631-10" #10 prio=5 os_prio=0 tid=0x000055927f4b3000 nid=0x2e runnable [0x00007ff4bbf46000]
"qtp2096171631-9" #9 prio=5 os_prio=0 tid=0x000055927ec6d800 nid=0x2d waiting on condition [0x00007ff4bc048000]
"Attach Listener" #8 daemon prio=9 os_prio=0 tid=0x000055927f7f0000 nid=0x2c waiting on condition [0x0000000000000000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x000055927ea74000 nid=0xf runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x000055927ea59000 nid=0xe waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x000055927ea56800 nid=0xd waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x000055927ea54800 nid=0xc runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x000055927ea24000 nid=0xb in Object.wait() [0x00007ff4bc9a7000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x000055927ea19800 nid=0xa in Object.wait() [0x00007ff4bcaa8000]
"VM Thread" os_prio=0 tid=0x000055927ea0f000 nid=0x9 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x000055927ea71800 nid=0x10 waiting on condition


##
# Joyent Triton run
##
# Run forth version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:4
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
Attach Listener" #13 daemon prio=9 os_prio=0 tid=0x0000000000a5f000 nid=0x91c2 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #12 prio=5 os_prio=0 tid=0x0000000000f3f800 nid=0x8e75 waiting on condition [0x0000000000000000]
"qtp2096171631-11-acceptor-0@5230b90f-ServerConnector@5cf74d39{HTTP/1.1,[http/1.1]}{0.0.0.0:54428}" #11 prio=3 os_prio=0 tid=0x0000000001075000 nid=0x8ead runnable [0x00007fffea6fb000]
"qtp2096171631-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000000001073000 nid=0x8eac waiting for monitor entry [0x00007fffea9fc000]
"qtp2096171631-9" #9 prio=5 os_prio=0 tid=0x0000000001070800 nid=0x8eab waiting on condition [0x00007fffeacfd000]
"qtp2096171631-8" #8 prio=5 os_prio=0 tid=0x0000000001070000 nid=0x8eaa runnable [0x00007fffeaffd000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000000000179000 nid=0x8e81 runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00000000000f5000 nid=0x8e7f waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f3000 nid=0x8e7e waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f1000 nid=0x8e7d runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c4800 nid=0x8e78 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be800 nid=0x8e77 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x8e76 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000000000117800 nid=0x8e82 waiting on condition

Be Still Careful of Many Threads

Our example web app now runs fine. But keep an eye on the thread count. Other Java/programming language library might create a thread pool and many threads. However, you can configure these libraries too.

Update 21th June, 2017

Well, first the OpenJDK team is improving the situation. As of JDK 8u131, the JVM does limit the CPU’s in Docker. And with the experimental flags ‘-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap’ it does limit the memory.

Also, Matt Rasband has a good blog post on how to estimate the memory usage of the JVM and set the flags properly.

Have fun running the JVM in a tiny container =)

Tags: Containers Java