Attached Maven tests with Cucumber

Sharing test code

It's common practice to build some kind of extensible, customizable application that can be deployed with a variety of customizations for different clients. When the base set of functionality is covered by integration and end-to-end tests, it is desirable to run these common tests on the different customized versions of the product.

This was the challenge we faced with my colleagues on a recent project, where we developed a Java based web application that was intended to be deployed with different configurations and customized look and feel. The base version of the application was functional on it's own, implemented in a Maven war project, had some sensible defaults and the core functionality. The client specific versions, also Maven war projects, provided custom design, configuration and additional functionality. The base project already had quite a lot of Cucumber feature specification that drove end-to-end tests to verify the application through the browser via the WebDriver API. We intended to find an elegant way to run this test suite on the client specific projects, and extend it to cover the additional functionality.

The Really Extensible Calculator

For the sake of simplicity, here is a reduced similar problem that lacks the complexity of a web application, but hopefully able to demonstrate the need for shared integration tests. The sample project is available on Github.

Suppose we have Calculator that can be configured to support many different operations. It has two public methods:

• performOperation, that takes a string as the name of the operation to be performed and two integers as it's operands
• getResult, that returns the result of the last operation

The Calculator reads a configuration file at startup to determine which operations to support, and what implementation do the operations have.

public class Calculator {
	private Map<String, Operation> operations = new HashMap<>();
	private int result = 0;
	
	public Calculator() {
		...
		// Reads the config file and stores Operation
		// instances in the 'operations' map
		...
	}
	
	public void performOperation(String name, int arg1, int arg2) {
		if (operations.containsKey(name)) {
			result = operations.get(name).perform(arg1, arg2);
		} else {
			throw new RuntimeException("No such operation.");
		}
	}
	
	public int getResult() {
		return result;
	}
}

Implementations of the Operation interface must provide the perform method, for example, our basic feature set might contain an addition:

public class Addition implements Operation {
	public int perform(int arg1, int arg2) {
		return arg1 + arg2;
	}
}

To register an Operation in the Calculator, one must declare it in the config.properties:

add=hu.advancedweb.Addition

To extend this system, in the case of Maven, we simply need to declare the Calculator as a dependency, provide additional Operations and our custom configuration file.

Integration tests for the base functionality

The Calculator project is documented and tested by Cucumber feature specifications and some glue code. The feature files are run with the JUnit Cucumber runner, as defined below.

@RunWith(Cucumber.class)
@CucumberOptions(format={"pretty"}, features = {"classpath:feature"})
public class FeatureTest {
}

The feature directory contains the Cucumber feature files in all projects. It's referenced via the classpath, not with direct file path. It is necessary to ensure that the test runner takes all feature files into account, even if they are in a dependency in some jar file.

The specification for the base Addition feature is the following:

Feature: Addition

  Scenario Outline: Add two numbers
    Given a Calculator
    When I perform addition with <num1> and <num2>
    Then I should get <result>

    Examples: 
      | num1 | num2 | result |
      | 1    | 2    | 3      |
      | 5    | 5    | 10     |

To make this specification living, we have to provide the appropriate glue code. The first and the last sentence in this scenario is pretty general, not specific to the addition feature. In order to make the glue code more extensible we implement the general step definitions in a separate class.

public class CommonSteps {
	public static Calculator calculator;
	
	@Given("^a Calculator$")
	public void a_Calculator() throws Throwable {
		calculator = new Calculator();
	}
	
	@Then("^I should get (\\d+)$")
	public void I_should_get(int arg1) throws Throwable {
		int result = calculator.getResult();
		assertThat(result, equalTo(arg1));
	}
}

And the glue code for the addition operation:

public class AdditionSteps {
	
	@When("^I perform addition with (\\d+) and (\\d+)$")
	public void I_perform_addition_with_and(int arg1, int arg2) throws Throwable {
	    CommonSteps.calculator.performOperation("add", arg1, arg2);
	}
	
}

Sharing test code and feature files

At this point we can easily test and extend our project with the built-in facilities of Maven. For example, we might create a new project that simply depend on the core module, and extend it with a Multiplication operation. To do that, we just have to provide one implementation class, and a modified configuration.

public class Multiplication implements Operation {
	public int perform(int arg1, int arg2) {
		return arg1 * arg2;
	}
}

To run the test suite in the extended project, it must depend on the required test dependencies. One way to do that with Maven is to provide a common parent project that declares the necessary test dependencies.

There are a few things to be done to run the integration test suite on the new project. First, the core module have to export it's test code and feature files. We can create a test-jar artifact with the maven-jar-plugin.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-jar-plugin</artifactId>
	<version>2.4</version>
	<executions>
		<execution>
			<goals>
				<goal>test-jar</goal>
			</goals>
		</execution>
	</executions>
</plugin>

The exported test-jar artifact contains the compiled tests for the project and the test resources of the project.

The second step is to use these tests in the new project. There are two things to it. In it's test scope the extended project has to depend on the test-jar artifact of the core project (as well as to the main artifact on it's compile scope of course).

<dependencies>
	<dependency>
		<groupId>hu.advancedweb</groupId>
		<artifactId>cucumber-example</artifactId>
		<version>0.0.1-SNAPSHOT</version>
	</dependency>
	<dependency>
		<groupId>hu.advancedweb</groupId>
		<artifactId>cucumber-example</artifactId>
		<version>0.0.1-SNAPSHOT</version>
		<type>test-jar</type>
		<scope>test</scope>
	</dependency>
</dependencies>

Then we have to tell Maven to run these included tests. From the maven-surefire-plugin version 2.15 there is a way to scan dependencies for tests to run. (Note: it works the same way with the maven-failsafe-plugin.)

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.18</version>
			<configuration>
				<dependenciesToScan>
					<dependency>hu.advancedweb:cucumber-example</dependency>
				</dependenciesToScan>
			</configuration>
		</plugin>
	</plugins>
</build>

This way the core module's test classes and feature files are part of the test suite of the extended project.

We might extend this test suite easily, we only have to provide the feature file and the step definitions for the new functionality:

Feature: Multiplication

  Scenario Outline: Multiply numbers
    Given a Calculator
    When I perform multiplication with <num1> and <num2>
    Then I should get <result>

    Examples: 
      | num1 | num2 | result |
      | 1    | 2    | 2      |
      | 5    | 5    | 25     |

public class MultiplicationSteps {
	@When("^I perform multiplication with (\\d+) and (\\d+)$")
	public void I_perform_addition_with_and(int arg1, int arg2) throws Throwable {
	    CommonSteps.calculator.performOperation("multiply", arg1, arg2);
	}
}

If we build the extended project, all the tests for the core module and the new functionality should run.

Conclusion

I think attached tests are the most suitable to share integration tests, but in some cases it might be beneficial for sharing tests with smaller scope, such as unit tests to catch some weird bugs. For example if someone creates a class with a FQN that exist in a dependency might violate some of it's invariants. In this case these tests demonstrates that their subject works fine in integration with it's current environment.