How To Terminate Threads in Java

How to shut down multi-threaded applications with Spring Boot

Everything has an end (photo: enobo)

I had this epiphany about how a thread is interrupted in Java recently. I’ve never read up on it and just got some piecemeal information from stackoverflow et al. when looking to how to deal with the dreaded InterruptedException being thrown out of a Thread.sleep().

How Java Threads Terminate

In Java, a thread terminates when the top-level method, i.e., the one that implements Runnable.run() or Callable.call() naturally exits.

There is a Thread.stop(), but it just kills the thread and may leave your application in an inconsistent state. In other words, you only want to call it when you want to kill your application anyway, and this can be done easier with just calling System.exit().

There are cases when you need to terminate a thread sooner than it comes to a natural end. One example is when your thread is designed to run forever because it processes some sort of events, polls the database or watches files. For example:

public void run() {
    while(true) {
        Event e = getEvent();

        processEvent(e);
    }
}

Another case is long-running threads where you simply cannot afford to wait until they end. On the desktop, a user may have clicked on a “quit” or “cancel” button and expects the operation to end in a reasonable time. On the server-side, operating systems grant processes only a limited time to shut down gracefully before they force-kill them.

Interrupting Threads

The Java-native way to stop threads it the Thread.interrupt() method. Note that the JVM does not in any way automatically interrupt threads. You have to implement the Thread.interrupt() call yourself, depending on your use case. You typically do this in a shutdown hook. Spring Boot comes with a default shutdown hook that looks for methods annotated with @PreDestroy, where you can put the interruption logic.

By itself, Thread.interrupt() does not do much because Java has no idea how to force a thread to terminate. It is the responsibility of the developer to check with the static method Thread.interrupted() if the current thread has been interrupted. If this method returns true, you should terminate the thread as quickly as possible and only execute the minimum necessary clean-up code.

For the infinite loop example, it would work like this:

public void run() {
    while(true) {
        if (Thread.interrupted()) {
            break;
        }

        Event e = getEvent();

        processEvent(e);
    }

    doSomeCleanUp();
}

Thread.interrupted() does reset the thread’s state, means it is no longer in interrupted state. Hence, code like below would not work:

public void run() {
    if (!Thread.interrupted()) {
        processEventPart1(e);
    }

    if (!Thread.interrupted()) {
        processEventPart2(e);
    }
    doSomeCleanUp();
}

If the first check returns true, the second one would return false and processEventPart2() is executed. That is why it is generally recommended to interrupt the current thread again after calling Thread.interrupted():

if (!Thread.interrupted()) {
 	processEventPart1(e);
    Thread.currentThread().interrupt();
 }

This ensures that subsequent checks still see an interrupted thread.

The problem with Thread.interrupt()

Thread.interrupt() interrupts not only your code where you expect it, but everything else that uses Thread.interrupted() or methods like Thread.sleep() or Thread.wait(). In my tests, both the Oracle JDBC and the MongoDB drivers reacted angrily to interruption with a barrage of “something terrible happened” types of exceptions. No wonder because from a database driver’s perspective interruptions are akin to yanking the network cable out of the computer.

Worse, since it is random where Thread.interrupt() hits the interrupted thread, you may or may not see the exceptions. I saw them regularly when using databases in data centers across half the country. If you test with local databases, chances are that everything works perfectly most of the time. For this reason, Oracle explicitly says, “Do not use the Thread.interrupt method” with their JDBC driver.

Bottom line, use Thread.interrupt() only if you have full control over all parts of the code, which is very rare nowadays.

Better Do Your Own Thing

When you can’t use Thread.interrupt(), you need to implement your own logic. Here’s an example for stopping an endless loop:

public class Process extends Runnable {
    private AtomicBoolean active = new AtomicBoolean(true);

    public void run() {
        while (active.get()) {
            doProcessing();
        }
    }

    public void stop() {
        active.set(false);
    }
}

AtomicBoolean is a JDK class that implements a thread-safe boolean variable. The stop() method simply sets it to false, which exists the loop the next time while is executed. You lose the ability to stop individual threads, but you don’t need that for a graceful shutdown.

Stop Sleeping and Waiting

Without Thread.interrupt() sleep() and wait() will continue to block until the specified timeout. There are three ways to deal with that:

• If sleeping or waiting times are short enough, you can just wait them out. For example, if you have 30 seconds to shut down your process, a sleep for a few seconds does not matter much.
• If your sleep times are much longer, then scheduling may be the better solution. For example, if you want to hit an API every five minutes, you can schedule the thread to run every five minutes instead of sleeping.
• You can divide your long sleeps into a loop of smaller sleeps, with checking your thread stop flag in between.

Graceful shutdown in Spring Boot

One of the many advantages in using Spring Boot is that its components are configured to gracefully shut down. However, if you start your own threads, you have to take care of them during shutdown as well. A good way is using ThreadPoolTaskExceutor, even if you don’t need the thread pooling functionality. Just make sure to size the pool correctly to avoid threads queuing up or wasting resources.

Spring Boot calls shutdown() on ThreadPooolTaskExceutor when the application terminates. By default, it then interrupts the threads. If you don’t want that for the reasons listed above, then you need to set setWaitForTasksToCompleteOnShutdown() to true. Another useful feature is setting a delay after the termination with setAwaitTerminationSeconds() or setAwaitTerminationMillis(). This causes Spring Boot to suspend its shutdown sequence until all threads have terminated or the specified timeout occurs. Without this delay, certain resources like database pools may be shutting down while a thread that uses them is still running.

If you use a custom thread termination method like the one in the previous example, you can subclass ThreadPooolTaskExceutor and override shutdown() in your Spring Boot configuration like this:

@Bean
public AsyncTaskExecutor asyncTaskExecutor(Process process) {
    ThreadPoolTaskExecutor te = new ThreadPoolTaskExecutor() {
        @Override
        public void shutdown() {
            process.stop();
            super.shutdown();
        }
    };
    te.setCorePoolSize(NUMBER_OF_THREADS);
    te.setMaxPoolSize(NUMBER_OF_THREADS);
    te.setWaitForTasksToCompleteOnShutdown(true);
    te.setAwaitTerminationSeconds(SHUTDOWN_DELAY_SECONDS);
    return te;
}

One more thing

This is not Java-specific, but something that is easy to overlook and will break a graceful shutdown when not done correctly

If you use UNIX shell scripts to start your JVM, make sure to use exec like this

exec java -jar myapp.jar

This should be the last line in a shell script because everything after exec will be ignored anyway.

If the JVM is started without exec, it runs as a child process of the shell and will not receive the SIGTERM shutdown signal, but hard-killed when the shell terminates. With exec the JVM replaces the shell and receives OS signals directly. Note, if you use multiple shell scripts that are calling each other, you need to use exec in each of them.

Conclusion

Shutting down a multithreaded Spring Boot application gracefully is more of an art than it should be. What makes it so tricky is that you really need to pay attention what is going on during shutdown. By default, threads are hard-terminated together with the JVM and don’t have an opportunity to log any errors. So it is easy to do it wrong without ever noticing it.