Deploying JVM in tiny containers, be carefull!

By gamlerhart April 25, 2017 English, Technical Wibbly Wobbly

I`ve deployed my rohrli.gamlor.info service into the smallest possible triton container. Except me, noone is going to use it anyway ^-^. This container has 128Mbyte of memory, and 1/16th CPU time. The OpenJDK JVM is known for consuming tons of memory. And it is somewhat true. However, the JVM also runs in a small container. In this blog post I give some recommendations. The problems exists in small Docker and Triton containers. I show everything for both.

Chrome & JVM eat memory generously

Web App Example

The web app is just a ‘Hello World’ app. For better explaination, it also consumes tons of memory. I created the example in Scala, but the other JVM languages have the same issue. The source code for the example is here.

We compile and package the program into a Docker container. The Docker file is simple:

The we create the image with `build` and run it with `run`. With regular Docker we restrict the memory with the ‘-m 128m’ option to 128Mbyte. And with ‘–cpus 0.065’ we only give very little CPU time.
When using Triton, the ‘-m 128m’ option will find a sufficient large compute instance. Alternativly, you also can use ‘–label com.joyent.package=g4-highcpu-128M’ to create a tiny instance.
Once we started out container, we create some load with the ‘watch’ and ‘curl’ commands. Then, let`s watch the logs.

First Issue: Program Dies

The Triton instance won`t expirience any issue. But the plain Docker instance (Linux 4.10, Docker 17.04) suddenly stops working. =( The program got killed.

Hmmm….Why did it got killed? Aha…the memory is small, so we might ran out of memory and got killed. We can, while the container runs, inspect the memory usage of the JVM with `jstat`:

The penguin and whale are dangerous. Be careful!

jstat is a bit confusing. All numbers are in KByte. The JVM has a few memory pools and each memory pool has a minimum (**MN), maximum (**MX) and current size (**C).
So, inside the regular Docker container it create really big memory pools. On my computer (16GB, Linux 4.10, Docker 17.04), OGCMX (old generation heap) memory is about 2.5GByte. However, on the Triton instance it is about 40Mybte large. So, with a 2.5GByte pool, if it actually uses more than 128MByte, it will die. For now, Docker just kills processes which use too much memory. However, the JVM sees all the hosts memory, so it creates giant memory pools.
However, the Triton container lies more convincingly, so the JVM sees the containers memory sizes and creates a sufficient small memory pool.
Well, there a solution to this issue. We just configure a small memory size with the `-Xmx` option. Plus I configure the JVM such that when it runs out of memory, it dumps the memory and dies.

FROM openjdk:8u121-jdk-alpine # Assume already compiled for now COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar # Defaults for a tiny container ENV MEMORY="64m" EXPOSE 8080 CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \ -jar /app/useless-demo-service.jar

After doing this, the JVM runs in the Docker container without any issue.

So Many Threads

Now, we configured the memory properly. However, there maybe more things we can configure properly? So, let`s see what threads are runing in the container with `jstack`:

Triton isn`t a fun guy!

Howly! More than 30 threads are running. On my computer the regular Docker container has 8 `GC task thread`s, and 4 `CompilerThread`s. On Triton there are 15! The JVM saw a lot of CPU cores and created many threads to take advantage of it. However, our container is restricted to 0.065 cores. Those many threads won`t help. Of course, we can configure this:

FROM openjdk:8u121-jdk-alpine # Assume already compiled for now COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar # Defaults for a tiny container ENV MEMORY="64m" \ CPUS=1 EXPOSE 8080 CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \ -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \ -jar /app/useless-demo-service.jar

In this really small container the `ParallelGC` is not really needed and we can use the `SerialGC`. In case you use a bit larger container, you can use `-XX:-XX:+ParallelGCThreads=$CPU` for `ParallelGC`. Or `-XX:G1ConcRefinementThreads=$CPU` in case you use `-XX:+UseG1GC`.
And we can use `-XX:CICompilerCount=$CPU` to reduce the compiler threads. However, it needs to be at least 2.

Still Many Threads

We still have many ‘acceptor’ and ‘selector’ threads. On Triton we’ve 4 acceptor and 4 selector threads:

Those threads are created by the web server, so you have to configure the web server you are using. In this example Jetty is used, and we configure it via simple Java properties:

And also change the Docker file:

And now, it starts to look fine, right?

Be Still Carefull of Many Threads

Our example web app now runs fine. But keep an eye on the thread count. Other Java/programming language library might create a thread pool and many threads. However, you can configure these libraries too.

Update 21th June, 2017

Well, first the OpenJDK team is improving the situation. As of JDK 8u131, the JVM does limit the CPU’s in Docker. And with the experimental flags ‘-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap’ it does limit the memory.

Also, Matt Rasband has a good blog post on how to estimate the memory usage of the JVM and set the flags properly.

Have fun running the JVM in a tiny container =)

Tagged on: docker, java, joyent, JVM, triton

Gamlor

thoughts about programming, tv-series and other geeky stuff.