April 24, 2017

JVM는 작은 컨테이너 안에 배포하면 주의!

저는 저의 https://rohrli.gamlor.info 서비스가 재일 작은 트리톤(Triton) 컨테이너 안에 배포됐어요. 저 말고 아무도 사용해서 괜찮아요 -. 이 컨테이너는 128MB 메모리만 있고, 1/16CPU만 있어요. OpenJDK JVM은 메모리를 많이 사용하기 잘 알려져요. 부분적으로 사실이에요. 근데 JVM이 작은 컨테이너 안에 도 작동해요. 이 글에는 조언을 제공할 거예요. 모은 작은 도커(Docker)와 트리톤 컨테이너가 이 문제 있어요. 제가 둘 다 보여줄 거예요 =)

Chrome와 JVM 메머리 잘 먹어요
Figure 1. Chrome와 JVM 메머리 잘 먹어요.

웹앱 예

이 웹앱 예는 그냥 ‘안녕 세상’ 프로그램 이에요. 그리고 설명 때문에 메모리 많이 사용해요. 제가 이 예를 Scala로 만들어요, 근데 모든 JVM의 프로그래밍 언어들이 도같는 문제 있어요. 소스 코드가 여기 있어요.

Hello-Basics.scala
/**
 * Snippet, memory limits
 */
// Limit the amount of items in memory for this example.
// So, max ~192*256k=48MBytes
val maxMemoryItems = 192
val memoryPerItem = 256 * 1024
val memoryWasting = new ArrayBlockingQueue[Array[Byte]](maxMemoryItems)

/**
 * Snippet, request handling
 */
// On each request, consume some memory.
// And keep things in memory to simulate memory use
case "/" :: Nil => {
// On each request, consume some memory.
// And keep things in memory to simulate memory use
val chunkOfMemory = Array.ofDim[Byte](memoryPerItem)
var addedToQueue = memoryWasting.offer(chunkOfMemory)

    // Throw things away until we've space
    while (!addedToQueue) {
        memoryWasting.poll()
        addedToQueue = memoryWasting.offer(chunkOfMemory)
    }

    val createdSoFar = createdItemsCounter.incrementAndGet()
    response.getWriter.write(
        s"""
            |{
            |"title":"Hello World! Let's use some memory",
            |"created-overall":"$createdSoFar",
            |"queue-size":"${memoryWasting.size()}",
            |"":"Hello. I'm so memory hungry. kthxbye!"
            |}
        """.stripMargin)
}

그리고 우리는 이 프로그램은 컴파일하고 도커 컨테이너를 만들어요. 도커 파일가 단순해요:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

EXPOSE 8080
CMD java -jar /app/useless-demo-service.jar

그리고 도커 build로 이미지 만들고 run로 실행해요. 보통 도커는 -m 128m 명령 행 옵션으로 메모리를 128MB 줘요. 그리고 –cpus 0.065로 조금만 CPU 시간 제공해요. 트리톤 사용하면 -m 128m가 매미로 충분한 트리톤 컨테이너 찾고 사용할 거예요. 아니면 –label com.joyent.package=g4-highcpu-128M도 작은 컨테이너를 사용할 수 있어요. 우리 컨테이너 시작했는데 그냥 watch 명령과 curl 명령으로 이 웹앱 예 사용해요. 그리고 로그 파일을 도 보자!

##
# Local docker run
##
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
# Start to follow the logs
sudo docker logs -f jvm-on-small-container
# Then, start the simple load, in another termial
watch -n 0.01 curl http://localhost:8080/

##
# Joyent Triton run
##
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
# Get the IP
docker inspect --format '{{ .NetworkSettings.IPAddress }}' jvm-on-small-container
# Start to follow the logs
docker logs -f jvm-on-small-container
# Then, start the simple load, in another termial
watch -n 0.01 curl http://{ip-from-above}:8080/

처음 문제. 프로세스가 죽었어요.

트리톤 컨테이너가 문제 없고 그냥 작동해요. 근데 보통 도커 (Linux 4.10, Docker 17.04) 컨테이너가 갑자기 작동하지 않아요. =(. 프로세스를 죽였어요:

[root@gamlor-manjaro useless-demo-service]# docker run -m 128m --cpus 0.065 -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:1
2017-04-20 12:58:20.248:INFO::main: Logging initialized @5900ms to org.eclipse.jetty.util.log.StdErrLog
2017-04-20 12:58:23.348:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT
2017-04-20 12:58:30.650:INFO:oejs.AbstractConnector:main: Started ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-04-20 12:58:30.748:INFO:oejs.Server:main: Started @16602ms
Killed

~. 왜 죽였어요? 아…메모리가 작고 아마 메모리 없었어서 죽었어요. 우리는 컨테이너 작동하는 동안 ‘jstat’로 JVM의 메모리 할당 볼 수 있어요:

##
# Local docker run
##
# Get memory usage of the running java process
sudo docker exec jvm-on-small-container sh -c 'jstat -gccapacity `pgrep java`'
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
84480.0 1354240.0 84480.0 10240.0 10240.0 64000.0 169472.0 2708992.0 169472.0 169472.0 0.0 1056768.0 4480.0 0.0 1048576.0 384.0 1 0


##
# Joyent Triton run
##
# Get memory usage of the running java process
eval "$(triton env)"
docker exec jvm-on-small-container sh -c 'jstat -gccapacity `pgrep java`'
 NGCMN    NGCMX     NGC     S0C   S1C       EC      OGCMN      OGCMX       OGC         OC       MCMN     MCMX      MC     CCSMN    CCSMX     CCSC    YGC    FGC
  2688.0  21824.0  21824.0 2176.0 2176.0  17472.0     5504.0    43712.0    43712.0    43712.0      0.0 1060864.0  12544.0      0.0 1048576.0   1536.0     15    69
펭귄과 고래 위험 조심해주세요
Figure 2. 펭귄과 고래 위험 조심해주세요.

jstat가 조금 혼란스러워요. 모든 숫자가 Kilo-Byte이에요. JVM는 메모리 풀 몇 게 있고 메모리 풀이 최저 (xxMN), 최고(xxMX), 이제(xxC) 사용한 메모리 이에요.

그럼 보통 도커 컨테이너 안에 진짜 근 메모리 폴 만들었어요! 네 컴퓨터에 (16GB, Linux 4.10, Docker 17.04) OGCMX (old generation heap) 메모리 풀이 2.5GByte이에요. 그데 트리톤 컨테이너에 40MByte이에요. 그래서 2.5GByte 메모리 풀 있으면 128MByte 이상 메모리 사용하고 죽을 거예요. 지금은 도커가 그냥 너무 많이 메모리 사용하면 프로세스를 죽여요. 근데 JVM이 도커-호스트 메모리 크기 보여서 근 메모리 풀 만들어요.

근데 트리톤 컨테이너 거짓말 좀 잘해서 JVM이 컨테이너의 메모리 크기를 보여서 메모리 풀 충분히 작아요. 그럼 이 문제를 해결할 수 있어요. 그냥 최고 메모리 크기를 ‘-Xmx’로 설정해요. 그리고 난 JVM이 메모리 없으면 메모리 덤프 하고 빨리 죽기도 추천해요:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m"

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -jar /app/useless-demo-service.jar

이렇게 하면 보통 도커 컨테이너도 잘 작동해요.

스레드 아주 많아요

그럼 이제는 메모리 잘 설정했어요. 근데 아마 다른 것 도 더 잘 설정할 수 있을까요? 그래서 컨테이너 작동하는 동안 jstack로 어느 스레드가 있는 걸 봐요!

##
# Local docker run
##
# Run second version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x000056473bc03800 nid=0x42 waiting on condition [0x00007fefe4931000]
"Attach Listener" #22 daemon prio=9 os_prio=0 tid=0x000056473bf72800 nid=0x41 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #21 prio=5 os_prio=0 tid=0x000056473bcd1800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-0@27b0949f-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x000056473bf75000 nid=0x24 runnable [0x00007fefe4b33000]
"qtp791452441-17" #17 prio=5 os_prio=0 tid=0x000056473bc2d800 nid=0x22 runnable [0x00007fefe8160000]
"qtp791452441-16" #16 prio=5 os_prio=0 tid=0x000056473bc2b000 nid=0x21 runnable [0x00007fefe8261000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000056473bc29000 nid=0x20 waiting for monitor entry [0x00007fefe8363000]
"qtp791452441-14-lowPrioritySelector" #14 prio=1 os_prio=0 tid=0x000056473bc8c000 nid=0x1f waiting for monitor entry [0x00007fefe8464000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x000056473bc8a000 nid=0x1e waiting for monitor entry [0x00007fefe8565000]
"qtp791452441-12-lowPrioritySelector" #12 prio=1 os_prio=0 tid=0x000056473bc86800 nid=0x1d waiting for monitor entry [0x00007fefe8666000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x000056473bd8a800 nid=0x1c runnable [0x00007fefe8766000]
"qtp791452441-10" #10 prio=5 os_prio=0 tid=0x000056473bd84800 nid=0x1b runnable [0x00007fefe8867000]
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000056473bc63800 nid=0x19 runnable [0x0000000000000000]
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000056473bc14000 nid=0x18 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x000056473bbf9000 nid=0x17 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x000056473bbea000 nid=0x16 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x000056473bbe7000 nid=0x15 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x000056473bbe4800 nid=0x14 runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x000056473bbb5000 nid=0x13 in Object.wait() [0x00007fefe92cb000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x000056473bbb2000 nid=0x12 in Object.wait() [0x00007fefe93cc000]
"VM Thread" os_prio=0 tid=0x000056473bba8000 nid=0x11 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000056473b9f0000 nid=0x9 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000056473b9f2000 nid=0xa runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000056473b9f3800 nid=0xb runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000056473b9f5800 nid=0xc runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x000056473b9f7000 nid=0xd runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x000056473b9f9000 nid=0xe runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x000056473b9fa800 nid=0xf runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000056473b9fc800 nid=0x10 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x000056473bc59800 nid=0x1a waiting on condition

##
# Joyent Triton run
##
# Run second version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:2
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"qtp791452441-44" #44 prio=5 os_prio=0 tid=0x000000000215e000 nid=0x12de4 runnable [0x00007fffe51f7000]
"qtp791452441-43" #43 prio=5 os_prio=0 tid=0x0000000000262000 nid=0x12aa4 waiting for monitor entry [0x00007fffe52fa000]
"qtp791452441-42" #42 prio=5 os_prio=0 tid=0x000000000068a000 nid=0x11808 waiting on condition [0x00007fffe7c7a000]
"Scheduler-1896277646" #41 prio=5 os_prio=0 tid=0x0000000000145000 nid=0x1176f waiting on condition [0x00007fffe4e34000]
"Attach Listener" #39 daemon prio=9 os_prio=0 tid=0x0000000000717800 nid=0x116e2 waiting on condition [0x0000000000000000]
"DestroyJavaVM" #33 prio=5 os_prio=0 tid=0x0000000000716000 nid=0x10796 waiting on condition [0x0000000000000000]
"qtp791452441-32-acceptor-3@6dbc83d8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #32 prio=3 os_prio=0 tid=0x0000000000714800 nid=0x1082a runnable [0x00007fffe5cff000]
"qtp791452441-31-acceptor-2@1ac08473-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #31 prio=3 os_prio=0 tid=0x0000000000713000 nid=0x10829 waiting for monitor entry [0x00007fffe6000000]
"qtp791452441-30-acceptor-1@34228b2c-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #30 prio=3 os_prio=0 tid=0x000000000070d800 nid=0x10828 waiting for monitor entry [0x00007fffe67fe000]
"qtp791452441-29-acceptor-0@467e71fc-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #29 prio=3 os_prio=0 tid=0x000000000070b000 nid=0x10827 waiting for monitor entry [0x00007fffe6f72000]
"qtp791452441-28-lowPrioritySelector" #28 prio=1 os_prio=0 tid=0x0000000000695800 nid=0x10818 runnable [0x00007fffe72b3000]
"qtp791452441-27" #27 prio=5 os_prio=0 tid=0x000000000068d800 nid=0x10816 waiting on condition [0x00007fffe75f6000]
"qtp791452441-26-lowPrioritySelector" #26 prio=1 os_prio=0 tid=0x000000000068b800 nid=0x10815 runnable [0x00007fffe7937000]
"qtp791452441-24-lowPrioritySelector" #24 prio=1 os_prio=0 tid=0x0000000000687800 nid=0x10812 waiting for monitor entry [0x00007fffe7fbc000]
"qtp791452441-23" #23 prio=5 os_prio=0 tid=0x0000000000685800 nid=0x10810 waiting for monitor entry [0x00007fffe82fe000]
"qtp791452441-22-lowPrioritySelector" #22 prio=1 os_prio=0 tid=0x0000000000683800 nid=0x1080f waiting for monitor entry [0x00007fffe85ff000]
"qtp791452441-21" #21 prio=5 os_prio=0 tid=0x000000000067d000 nid=0x1080e runnable [0x00007fffe8b1d000]
"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00000000001ee800 nid=0x107c5 runnable [0x0000000000000000]
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00000000001e1800 nid=0x107c3 waiting on condition [0x0000000000000000]
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00000000001d7000 nid=0x107c1 waiting on condition [0x0000000000000000]
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00000000001d4000 nid=0x107bf waiting on condition [0x0000000000000000]
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00000000001b7800 nid=0x107bd waiting on condition [0x0000000000000000]
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00000000001ad800 nid=0x107bc waiting on condition [0x0000000000000000]
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00000000001a2000 nid=0x107bb waiting on condition [0x0000000000000000]
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x0000000000197800 nid=0x107b9 waiting on condition [0x0000000000000000]
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x0000000000172000 nid=0x107b8 waiting on condition [0x0000000000000000]
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x0000000000186800 nid=0x107b7 waiting on condition [0x0000000000000000]
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x000000000017c000 nid=0x107b6 waiting on condition [0x0000000000000000]
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x0000000000165800 nid=0x107b5 waiting on condition [0x0000000000000000]
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x000000000010f000 nid=0x107b3 waiting on condition [0x0000000000000000]
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x0000000000104800 nid=0x107b2 waiting on condition [0x0000000000000000]
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00000000000f5000 nid=0x107b1 waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x107b0 waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x107af runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x107a3 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x107a1 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x1079d runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00000000001f3800 nid=0x107c6 waiting on condition

어머! 30게 이상 스레드 있어요. 내 컴퓨터의 보통 도커 컨테이너 GC task thread 8게하고 CompilerThread 4개 있어요. 트리톤 컨테이너가 15개 있어요. JVM는 많은 CPU 코어 봐서 스레드 많이 만들어요. 근데 우리 컨테이너가 0.065 CPU 시간만 사용해도 돼요! 이 많은 스레드 돕지 않아요.

펭귄과 고래 위험 조심해주세요
Figure 3. 트리톤 재미 없는 놈 야!!

그리고 우리는 이것도 설정할 수 있어요:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m" \
    CPUS=1

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \
    -jar /app/useless-demo-service.jar

스레드 아직 많아요

acceptor, lowPrioritySelector 스레드가 아직 많아요. 트리톤 컨테이너 안에 acceptor 스레드 4개, lowPrioritySelector 스레드 4개 있어요:

##
# Local docker run
##
# Run third version of our container
sudo docker run -d -m 128m --cpus 0.065 --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
sudo docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"Attach Listener" #20 daemon prio=9 os_prio=0 tid=0x0000564e7cd17000 nid=0x37 waiting on condition [0x0000000000000000]
"qtp791452441-19" #19 prio=5 os_prio=0 tid=0x0000564e7cd15000 nid=0x36 waiting on condition [0x00007fdcc863f000]
"qtp791452441-17-acceptor-0@77fe3be4-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x0000564e7cc25000 nid=0x34 runnable [0x00007fdcc8841000]
"DestroyJavaVM" #16 prio=5 os_prio=0 tid=0x0000564e7c922800 nid=0x8 waiting on condition [0x0000000000000000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x0000564e7ce14000 nid=0x31 waiting for monitor entry [0x00007fdccbd6e000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x0000564e7ce12000 nid=0x30 runnable [0x00007fdccbe6e000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000564e7ce0f800 nid=0x2f waiting for monitor entry [0x00007fdccbf70000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000564e7ce0e000 nid=0x2e runnable [0x00007fdccc070000]
"qtp791452441-11-lowPrioritySelector" #11 prio=1 os_prio=0 tid=0x0000564e7ce0c000 nid=0x2d waiting for monitor entry [0x00007fdccc172000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000564e7ce0a000 nid=0x2b waiting for monitor entry [0x00007fdccc273000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000564e7ce07800 nid=0x2a runnable [0x00007fdccc373000]
"qtp791452441-8" #8 prio=5 os_prio=0 tid=0x0000564e7ce01000 nid=0x29 runnable [0x00007fdccc474000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000564e7c8c2800 nid=0xf runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000564e7c8b7800 nid=0xe waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x0000564e7c8ac800 nid=0xd waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x0000564e7c8aa800 nid=0xc runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000564e7c87d800 nid=0xb in Object.wait() [0x00007fdccccd4000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000564e7c873000 nid=0xa in Object.wait() [0x00007fdcccdd5000]
"VM Thread" os_prio=0 tid=0x0000564e7c869000 nid=0x9 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000564e7c8c5800 nid=0x10 waiting on condition


##
# Joyent Triton run
##
# Run third version of our container
eval "$(triton env)"
docker run -d -m 128m --name=jvm-on-small-container -p 8080:8080 gamlerhart/blog-jvm-on-small-containers:3
# List the threads running
docker exec jvm-on-small-container sh -c 'jstack `pgrep java`' | grep os_prio
"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x000000000028c000 nid=0x732a waiting on condition [0x0000000000000000]
"qtp791452441-22" #22 prio=5 os_prio=0 tid=0x00000000006eb000 nid=0x7178 waiting on condition [0x00007fffe81fe000]
"qtp791452441-21-acceptor-3@161ab1a8-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #21 prio=3 os_prio=0 tid=0x00000000006ea000 nid=0x7177 waiting for monitor entry [0x00007fffe84ff000]
"DestroyJavaVM" #20 prio=5 os_prio=0 tid=0x00000000001df000 nid=0x7109 waiting on condition [0x0000000000000000]
"qtp791452441-19-acceptor-2@70b4bcd9-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #19 prio=3 os_prio=0 tid=0x00000000006e7000 nid=0x7175 waiting for monitor entry [0x00007fffe8800000]
"qtp791452441-18-acceptor-1@57033676-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #18 prio=3 os_prio=0 tid=0x0000000000747800 nid=0x7174 waiting for monitor entry [0x00007fffe97f6000]
"qtp791452441-17-acceptor-0@67e0dd84-ServerConnector@512ddf17{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #17 prio=3 os_prio=0 tid=0x00000000006e4800 nid=0x7173 runnable [0x00007fffe8ffe000]
"qtp791452441-15-lowPrioritySelector" #15 prio=1 os_prio=0 tid=0x000000000066b000 nid=0x7159 waiting for monitor entry [0x00007fffe9af7000]
"qtp791452441-14" #14 prio=5 os_prio=0 tid=0x000000000066c800 nid=0x7157 runnable [0x00007fffe9df7000]
"qtp791452441-13-lowPrioritySelector" #13 prio=1 os_prio=0 tid=0x0000000000668800 nid=0x7156 waiting for monitor entry [0x00007fffea0f9000]
"qtp791452441-12" #12 prio=5 os_prio=0 tid=0x0000000000666800 nid=0x7155 runnable [0x00007fffea3f9000]
"qtp791452441-11" #11 prio=5 os_prio=0 tid=0x0000000000664800 nid=0x7154 runnable [0x00007fffea6fa000]
"qtp791452441-10-lowPrioritySelector" #10 prio=1 os_prio=0 tid=0x0000000000662800 nid=0x7153 waiting for monitor entry [0x00007fffea9fc000]
"qtp791452441-9" #9 prio=5 os_prio=0 tid=0x0000000000660800 nid=0x7152 runnable [0x00007fffeacfc000]
"qtp791452441-8-lowPrioritySelector" #8 prio=1 os_prio=0 tid=0x0000000000659800 nid=0x7151 waiting for monitor entry [0x00007fffeaffe000]
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x0000000000130800 nid=0x7120 runnable [0x0000000000000000]
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x0000000000104000 nid=0x711e waiting on condition [0x0000000000000000]
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00000000000f2800 nid=0x711d waiting on condition [0x0000000000000000]
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00000000000f0800 nid=0x711c runnable [0x0000000000000000]
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00000000000c1000 nid=0x7115 in Object.wait() [0x00007ffffd5fe000]
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00000000000be000 nid=0x7113 in Object.wait() [0x00007ffffe9fe000]
"VM Thread" os_prio=0 tid=0x00000000000b4000 nid=0x710f runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000000000128800 nid=0x7121 waiting on condition

그럼 웹 서버는 이 스레드를 만들어서 사용한 웹 서버 설정해야 돼요. 이 예는 임베디드 Jetty 사용하고 그냥 Java-Properties로 설정할 수 있어졌어요:

DemoService.scala
// Version 1 did just start Jetty
// val server = new Server(8080)
// It creates many acceptors, selectors and threads on big machines, even when running in small container
// So, let's explicitly configure it.
val minThreads = System.getProperty("jetty.min-threads", "8").toInt
val maxThreads = System.getProperty("jetty.max-threads", "200").toInt
var threadPool = new QueuedThreadPool(minThreads, maxThreads)

val server = new Server(threadPool)
val acceptorCount = System.getProperty("jetty.acceptor-threads", "-1").toInt
val selectorCount = System.getProperty("jetty.selector-threads", "-1").toInt
server.setConnectors(Array(new ServerConnector(server, acceptorCount, selectorCount)))

server.setHandler(new OurHandling)
server.start()

그리고 도커 파일 도 바꿨어요:

DOCKERFILE
FROM openjdk:8u121-jdk-alpine

# Assume already compiled for now
COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar

# Defaults for a tiny container
ENV MEMORY="64m" \
    CPUS=1\
    ACCEPTORS=1 \
    SELECTORS=1\
    JETTY_MIN_THREADS=4 \
    JETTY_MAX_THREADS=20

EXPOSE 8080

CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \
    -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \
    -Djetty.acceptor-threads=$ACCEPTORS \
    -Djetty.selector-threads=$SELECTORS \
    -Djetty.min-threads=$JETTY_MIN_THREADS \
    -Djetty.max-threads=$JETTY_MAX_THREADS \
    -jar /app/useless-demo-service.jar

스레드 많기 아직 조심하세요!

우리 웹앱 예 이제 괜찮게 작동해요. 근데 스레드 개수 잘 보세요. 다른 Java/프로그래밍 언어 라이브러리도 아마 스레드 풀 만들고 스레드 너무 많이 만들어요. 근데 이 라이브러리들이 도 설정할 수 있어요.

Update 2017년6월21일

먼저, JDK 프로그래머들이 도커 지원 개선 가요.. JDK 8u131부터, JVM는 CPU를 한도 해요. 그리고 `-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap’ 사용하면 메모리도 한도 해요.

그리고 Matt Rasband는 좋은 블로그 글 썼어요. 이 글이 어떡해 JVM의 메모리 사이즈 대략 추산하고 메모리의 명령 행 옵션 자동으로 설정하기 설명해요.

작은 JVM 컨테이너 잘 사용하세요 =).

Tags: 한국어 Containers Java