June 13, 2021

Tests Not Running: JUnit 5, Maven and Spring Boot Magic

I recently worked on an existing Spring Boot based codebase. Everything looked fine and kept adding new features and improving the codebase. I wrote my tests and run the test suite in my IDE. My changes got into production and everything looked fine.

A few weeks later I took a look at the Continues Integration Servers logs of that project and noticed that the tests are not running. That is no good. My local setup is not a reliable environment, for example, I can forget to check in a file into version control. And I usually run a reduced set of tests and rely on the CI system to run the whole tests suite.

Tests Not Running
Figure 1. Maven Avoid Running The Tests

Investigation

As said, the test ran fine in my IDE. Also, the tests are ordinary JUnit tests. Some tests had Spring 'magic' annotations. To ensure it’s not the Spring magic breaking the tests, I removed all test and only added a basic test.

Most Basic Test
package info.gamlor.blog;

import org.junit.Test;

import static junit.framework.TestCase.assertEquals;

public class BlogApplicationTest {
	@Test
	public void calculate1Plus1Gives2() {
		assertEquals(1+1,2);
	}
}

However, the Maven build still didn’t run any test:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Another common issue is that Maven test runner, called surefire, is outdated and can’t deal with a newer test library version. I did double-check and update that plugin. No tests ran.

Test classes need to follow a certain naming pattern to be picked up by Maven. I changed the configuration from the default to ensure it matches my class names. No tests ran.

I noticed that JUnit was a transitive dependency. So I explicitly included a Junit 4 version. No tests ran.

Then after some time, I started to question if the test ever ran for this project. I checked old CI system logs. Turns out, in older builds the tests ran. Something changed which broke the test running. I shortly skimmed the git logs, but I couldn’t see any obvious change related to the test setup.

It was git bisect time to find the commit which broke the tests. After a few bisect steps, I found the commit which broke the tests. It was the commit upgrading Spring Boot from 2.3 to 2.4.

Why Did The Spring Boot Update Broke The Tests?

That still left me puzzled. The general update worked fine. There was no compilation error or other issues. I continued to fiddle around without much success.

After a while without any progress, I decided to look at a reference 'example' project to see how the tests are set up there. I found the Spring Boot project generator and generated an example Spring Boot 2.4 project.

I ran the mvn test command on that example, and the tests did run:

[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

I checked the location of the tests etc. All seemed to be set up the way my project was. Then I looked at the example test:

Spring Boot Example
package info.gamlor.blog;

import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

@SpringBootTest
class BlogApplicationTests {

	@Test
	void contextLoads() {
	}

}

Looked the same at first sight, right? Except for the import: The Test class is org.junit.jupiter.api.Test instead of org.junit.Test. I am mostly busy with Scala/Typescript/Clojure these days, so I had to check what the difference is.

Junit 4, 5 and Maven

Turns out JUnit 5 is a complete rework and not an incremental change. With that the whole package structure changed. Spring Boot 2.4 and it’s Maven configuration upgraded to JUnit 5. Therefore, the Maven test runner searched for JUnit 5 tests and just skipped our Junit 4 tests.

JUnit 5 brings support for running JUnit 4 as well. When you add the junit-vintage-engine dependency then the JUnit 4 and 5 tests run together. This allows you to upgrade the test suite bit by bit to JUnit 5. However, I decided to update the test source code to use JUnit 5 instead.

Run Junit 4 and 5 together with junit-vintage-engine
<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>{version}</version>
    <scope>test</scope>
</dependency>

One last question left is why the upgrade to Spring 2.4 worked without any compilation error. For that, I searched the Maven dependency tree for Junit. It compiled because the test container dependency includes JUnit 4. I tried to exclude it, but it is a hard dependency.

$ mvn dependency:tree | grep -B 10 junit
...
[INFO] +- org.testcontainers:testcontainers:jar:1.15.2:test
[INFO] |  +- junit:junit:jar:4.13.2:test

Summary

Spring Boot switched to Junit 5 with version 2.4. When updating Spring Boot in an existing project, existing JUnit 4 tests get ignored by Maven. Because Junit 4 was still included as a transitive dependency there was no compilation error during the upgrade. You can run Junit 5 and 4 side by side by including junit-vintage-engine in your project.

Tags: Testing Java Development