November 16, 2025

Learned/Discovered this week: bpftrace, Windows in Docker, Testcontainers & Docker upgrade, Debugging-Tips

A short list of things I’ve learned or discovered this week.

bpftrace: DTrace for Linux

I discovered that bpftrace exists for Linux. It is similar to DTrace, where you instrument your system to debug it. That is great news (to me), because DTrace never made really to mainstream Linux. (Even Windows has it). bpftrace compiles things to eBPF programs to do the tracing, so it works on must Linux flavors.

Linux having a tracing tool
Figure 1. Linux having a tracing tool

Found via Brendan Gregg. Its one of the best resources for performance debugging.

Windows and MacOS VMS via Docker

I discovered Windows in Docker, a way to run Windows as a Docker container. This is for testing, and the main benefit is that it integrates into the Docker ecosystem. That Windows VM can be part of a Docker compose setup. Plus you probably have Docker already installed and can avoid an extra tool.

However, unlike Vagrant, it doesn’t have well-prepared images or automatic WinRM integration. Any automation will take more work.

Windows on Docker
Figure 2. Windows on Docker

The same approach even works for MacOS in Docker. Yes, it even runs on AMD64. While technically impressive, it is very rough and barely usable. You need to do tons of manual steps to set it up. And everytime MacOS shows an animation, everything grinds to a halt for many seconds, at least on my AMD cpus.

MacOS running on AMD64
Figure 3. Yep, MacOS running on AMD64

Debugged: Suddenly Failing Tests

Large parts of the test suite at work suddenly started to fail. Without any code change that could influence the tests. The errors where basically that it couldn’t do the test setup, with a very long stack traces (aka Spring panicking).

After a while I found this line in the logs:

o.t.d.DockerClientProviderStrategy       : Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
build 14-Nov-2025 06:03:49          UnixSocketClientProviderStrategy: failed with exception BadRequestException (Status 400: {"message":"client version 1.32 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version"}
build 14-Nov-2025 06:03:49    )

This was the smoking gun: We had updated the OS on the build machines and that included a new Docker version. It refused to accept TestContainers request, because it uses an older client version id.

The solutions are:

  • Upgrade to TestContainers 2.x. Note that some Maven identifiers changed. From org.testcontainers:{some-db} to org.testcontainers:testconatainers-{some-db}

  • Or override some property to present to be a newer client version. See here.

Debugging: Old Tricks are still useful

I had some mysteriously hanging system this week. Some old and classic tricks quickly helped me debugging it:

  1. Keep a tracing-id in logs: I think that is pretty standard these days. Basically a request generates an id. Then that id is used for all log lines related to that request. Further, you then use that id across process/service boundary, so you can see the logs for a request across systems.

  2. Name threads with the tracing ID. This immediately gives you a clue on what thread is working on:

String threadName = Thread.currentThread().getName();
Thread.currentThread().setName(threadName + " working on:" + traceId);
try {
    // Run the request processing
} finally {
    Thread.currentThread().setName(threadName);
}
  1. Getting stack dumps, eg jstack {pid}. Then, with the named threads I easily can see where my request is stuck.

jstack {pid} | grep -A 200 {my-stuck-request-id)
"http-nio-8080-exec-3 working on:{the-stuck-request-id}" #20 [20] daemon prio=5 os_prio=0 cpu=226.71ms elapsed=4797.05s tid=0x00007fc289062040 nid=20 runnable  [0x00007fc28d0a6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.Net.connect0(java.base@21.0.8/Native Method)
at sun.nio.ch.Net.connect(java.base@21.0.8/Net.java:589)
at sun.nio.ch.Net.connect(java.base@21.0.8/Net.java:578)
// Snipp
at com.some.library.SomeHttpClient.execute(SomeHttpClient.java:94)
// Oh, our code trying to connect to things and timing out
at our.code.SomeSystem.executeRequest(SomeSystem.java:42)
at our.code.SomeSystem.doSomethingUseful(SomeSystem.java:100)
// Snip
at java.lang.invoke.LambdaForm$DMH/0x00007fc230283000.invokeVirtual(java.base@21.0.8/LambdaForm$DMH)
// Snip
Tags: Debugging Unix Development