Maven Surefire: multiple executions and debugging

Surefire is the default plugin to execute tests in a Maven build. It is well integrated with JUnit and TestNG which fulfill most of the needs but if you desire you can use another “provider” to use another test framework.

Of course the same support exists in IDE and you can debug in IDE your tests. However there are still few cases you will desire to debug using Maven command line:

• Your continuous integration fails but not the test in the IDE so you need to reproduce it as close as you can

• Surefire got a serious amount of configuration which would take time to reproduce in an IDE runner

• Surefire doesn’t exactly use the same classloading as IDE most of the time so you want to reproduce this particularity

• Probably much more ;)

When you use the default configuration you either rely on the implicit configuration or you explicitely configure it:

<plugin>
 <artifactId>maven-surefire-plugin</artifactId>
 <configuration>
   <reuseForks>true</reuseForks>
   <forkCount>1</forkCount>
   <!-- any config you need -->
 </configuration>
</plugin>

In such a case it is easy to run a single test using:

mvn test -Dtest=MyTest

And to debug it you can use:

mvn test -Dtest=MyTest -Dmaven.surefire.debug

Then it will run the JVM in (remote) debug mode and you can connect from your IDE using a remote debug configuration. Default uses port 5005 but you can configure it with the same property (maven.surefire.debug).

Until there all is smooth but suppose now you use multiple execution because you grouped tests? How to select the right execution?

Grouping tests

There are two main ways to group tests in execution in surefire:

• Using include/exclude configuration

• Using an actual group of the provider

The first option is easy but not that great to maintain so let’s use the second one. Since JUnit is likely still more used than TestNG let’s use that as well.

First you need to define a “class”. Personally I use a marker interface:

package com.github.rmannibucau.surefire.executions;

public interface Group1{
}

Of course in practise don’t hesitate to use a name representing something. In my case a common example is “Embedded” opposed to “Arquillian”.

TIP: if you have two groups you need a single marker interface since surefire allows you to exclude a group.

Then with JUnit you need to use @org.junit.experimental.categories.Category annotation to mark the test group. For instance for an ApplicationComposer test it could look like:

@Category(Group1.class)
@Classes @Default
@EnableServices("jaxrs")
@RunWith(ApplicationComposer.class)
public class MyEETest{
   @RandomPort("http")
   private URL base;

   @Test
   public void doTest() {
       // ...
   }
}

In the same module you can define an Arquillian test without any marker for instance.

Concurrent testing framework pitfall

To test EE applications it is very convenient to use ApplicationComposer for all the backend (until JAX-RS) but sometimes desired to use TomEEEmbedded or Arquillian JUnit runner for the frontend to benefit from a real HTTP server (aka Tomcat).

Note: using EE test stacks as example but it is the same with any frameworks (runner/rules in JUnit world) you would use.

It doesn’t look an issue to mix this kind of tests but it can be a nightmare to run them in a suite (in the build) since both don’t have the same lifecycle. Typically in my previous example ApplicationComposer will run the container per class or method depending how it is used and the other runner (let’s say Arquillian) is better used when ran once per suite (a single container is started/stopped). The issue is if an ApplicationComposer test runs in between two Arquillian tests then it will breaks the assumption of the Arquillian container.

To avoid this kind of issue it is better to split both executions and ensure tests are executed in a suite they are compatible with. Concretely for previous example we’ll have an execution for embedded tests (ApplicationComposer) and one for Arquillian.

Configure multiple executions

Since surefire is set up by default you can use the default execution to execute one of the suite but why privileging one? Personally I prefer to skip the default execution and redefine one execution per suite:

<plugin>
 <artifactId>maven-surefire-plugin</artifactId>
 <executions>
   <execution>
     <id>applicationcomposer</id>
     <goals>
       <goal>test</goal>
     </goals>
     <configuration>
       <groups>com.github.rmannibucau.surefire.executions.Group1</groups>
       <skip>${maven.test.skip}</skip>
       <systemPropertyVariables>
         <openejb.jul.forceReload>true</openejb.jul.forceReload>
       </systemPropertyVariables>
     </configuration>
   </execution>
   <execution>
     <id>arquillian</id>
     <goals>
       <goal>test</goal>
     </goals>
     <configuration>
       <excludedGroups>com.github.rmannibucau.surefire.executions.Group1</excludedGroups>
       <skip>${maven.test.skip}</skip>
       <systemPropertyVariables>
         <java.io.tmpdir>${project.build.directory}</java.io.tmpdir>
       </systemPropertyVariables>
     </configuration>
   </execution>
 </executions>
 <configuration>
   <skip>true</skip>
   <failIfNoTests>false</failIfNoTests>
   <reuseForks>true</reuseForks>
   <forkCount>1</forkCount>
 </configuration>
</plugin>

Defining multiple executions is built-in in Maven pom.xml syntax so nothing complicated. The nice thing with Maven syntax there is you can make any configuration specific (systemPropertyVariables in previous sample) but also share some configuration (reuseForks).

The group selection is done using excludedGroups or groups tags. Finally the shared configuration enforces skip to skip the default execution but both suite define this value to ${maven.test.skip} to be active by default and skipped if this property is set.

How to debug a particular test then?

But then you execute surefire 3 times and if you run previous debugging command you will maybe not execute in the right context your test (likely if your test is not in the first executed suite).

This is indeed bad but since Maven 3.3.1 you can select the execution you want to use and therefore select the right suite. For that you need to execute surefire:test goal instead of the default test one and qualify it with the execution name using @ delimiter:

mvn surefire:test@arquillian -Dtest=MyTest -Dmaven.surefire.debug

Conclusion

Even if setting multiple suite requires some work it also allows to share a test suite configuration between developers through the pom (and therefore avoid a big README explaining how to set it up in any runner). It also allows to execute suite one by one and potentially exclude a single one if your project lifecycle requires it (let say you have “backend” and “frontend” tests, if you rewrite the “frontend” part you can exclude these tests and add a new “frontend-ng” group).

Since Maven now allows to work smoothly with this kind of setup you shouldn’t exclude any test framework because it would break other tests: this is no more true!