Mystery Jetty Threads Due To Failed Startup
Yesterday I had an app with an embedded Jetty Server which didn’t shut down properly. The main thread exited and the
app had called .stop() on the Jetty server. However, the app kept running.
// App setup
// Then the app runs until the user stops it:
System.out.println("Press any key to stop");
while (System.in.available() <= 0) {
Thread.sleep(1000);
}
jetty.stop();
System.out.println("Stopped");
// App doesn't quit here. It keeps running.
An Java app which keeps running unexpectedly does indicates that threads are didn’t get stopped and keep the app alive. So I got a thread dump to see what is still running:
"WebSocketClient@592617454-23@1661" daemon prio=5 tid=0x17 nid=NA runnable "WebSocketClient@592617454-24@1662" daemon prio=5 tid=0x18 nid=NA runnable "WebSocketClient@592617454-25@1663" daemon prio=5 tid=0x19 nid=NA runnable "WebSocketClient@592617454-26@1664" daemon prio=5 tid=0x1a nid=NA runnable "HttpClient@619713e5-scheduler-1@2527" prio=5 tid=0x31 nid=NA waiting "qtp377478451-15@1239" prio=5 tid=0xf nid=NA waiting "qtp377478451-16@1240" prio=5 tid=0x10 nid=NA waiting "qtp377478451-17@1242" prio=5 tid=0x11 nid=NA waiting "qtp377478451-18@1243" prio=5 tid=0x12 nid=NA waiting "qtp377478451-19@1245" prio=5 tid=0x13 nid=NA waiting "qtp377478451-20@1247" prio=5 tid=0x14 nid=NA waiting "qtp377478451-21@1248" prio=5 tid=0x15 nid=NA waiting "qtp377478451-22@1249" prio=5 tid=0x16 nid=NA waiting "WebSocketClient@592617454-27@1665" daemon prio=5 tid=0x1b nid=NA waiting "WebSocketClient@592617454-28@1666" daemon prio=5 tid=0x1c nid=NA waiting "WebSocketClient@592617454-29@1667" daemon prio=5 tid=0x1d nid=NA waiting "WebSocketClient@592617454-30@1668" daemon prio=5 tid=0x1e nid=NA waiting "Common-Cleaner@1943" daemon prio=8 tid=0xa nid=NA waiting "Reference Handler@2584" daemon prio=10 tid=0x2 nid=NA runnable "Finalizer@2585" daemon prio=8 tid=0x3 nid=NA waiting "Signal Dispatcher@2586" daemon prio=9 tid=0x4 nid=NA runnable "DestroyJavaVM@2583" prio=5 tid=0x32 nid=NA runnable⏎
Aha, I saw the WebSocketClient threads and one Servlet is using a WebSocket client. That servlet might not
stop the WebSocketClient and keeps the rest of Jetty alive. I double-checked that my servlet stops the
WebSocketClient. My servlet did stop the client on destroy() and that code was called during the app shutdown.
public class ExampleServlet extends HttpServlet {
private WebSocketClient client;
// other stuff
@Override
public void destroy() {
super.destroy();
try {
client.stop();
} catch (Exception e) {
e.printStackTrace();
}
}
}So I just removed the WebSocketClient for a minute, just in case. Without the WebSocketClient there where still hanging Jetty threads:
"qtp377478451-15@1239" prio=5 tid=0xf nid=NA waiting "qtp377478451-16@1243" prio=5 tid=0x10 nid=NA waiting "qtp377478451-17@1244" prio=5 tid=0x11 nid=NA waiting "qtp377478451-18@1245" prio=5 tid=0x12 nid=NA waiting "qtp377478451-19@1246" prio=5 tid=0x13 nid=NA waiting "qtp377478451-20@1247" prio=5 tid=0x14 nid=NA waiting "qtp377478451-21@1248" prio=5 tid=0x15 nid=NA waiting "qtp377478451-22@1252" prio=5 tid=0x16 nid=NA waiting "Common-Cleaner@1516" daemon prio=8 tid=0xa nid=NA waiting "Reference Handler@1513" daemon prio=10 tid=0x2 nid=NA runnable "Finalizer@1514" daemon prio=8 tid=0x3 nid=NA waiting "Signal Dispatcher@1515" daemon prio=9 tid=0x4 nid=NA runnable "DestroyJavaVM@1512" prio=5 tid=0x1f nid=NA runnable
Ok, that seemed odd that Jetty doesn’t stop its threads. So I did a heap dump and inspected the JettyServer’s state.
And I found an instance with _state=-1.
Next, I looked at what _state=-1 means. A quick look at the source and it means [STATE_FAILED],
see source.
// Inheritance chain: Server -> HandlerWrapper -> AbstractHandlerContainer -> AbstractHandler -> ContainerLifeCycle -> AbstractLifeCycle
public abstract class AbstractLifeCycle implements LifeCycle
{
// Other stuff
private static final int STATE_FAILED = -1;
private static final int STATE_STOPPED = 0;
private static final int STATE_STARTING = 1;
private static final int STATE_STARTED = 2;
private static final int STATE_STOPPING = 3;
private volatile int _state = STATE_STOPPED;
// Other stuff
private void setFailed(Throwable th)
{
_state = STATE_FAILED;
if (LOG.isDebugEnabled())
LOG.warn(FAILED + " " + this + ": " + th, th);
for (Listener listener : _listeners)
{
listener.lifeCycleFailure(this, th);
}
}
// Other stuff
}Alright, the Server instance failed. That was a surprise because I didn’t saw any exception. I added a breakpoint to the place where the state is set to ` STATE_FAILE. Right when the app started the breakpoint fired. During the startup, Jetty failed but still worked? Well not really. The app starts Jetty first on port 80, which fails on Linux. On failure it starts Jetty again with a high port number. However, when that first failed instance already started the thread pool. And Jetty doesn’t stop those when it fails to bind the ports.
try {
// Try to bind low port
jetty = setupJetty(80);
} catch (Exception e) {
// If that fails fallback to high port range and start Jetty with a high port
// However, if Jetty fails to bind the port the thread pool already runs and doesn't stop by itself.
jetty = setupJetty(8080);
}To fix the issue we need to stop Jetty if it failed to start. That fixed my issue and the app quit without issues after this fix.
private static Server setupJetty(int port) throws Exception {
var jetty = new Server(port);
// Embedded handlers setup etc.
try{
jetty.start();
} catch (Exception e){
// The fix: Stop Jetty when it fails to start, to ensure the thread pools etc are stopped
jetty.stop();
throw e;
}
return jetty;
}