Scott: detailed failure reports and hassle free assertions for Java tests

Today I am excited to present Scott. The goal of this project is to provide detailed failure messages for tests written in Java, without the use of complex assertion libraries to aid developers in rapid development, troubleshooting and debugging of tests.

Motivation

Lately I tried the Spock Framework, and found it very pleasant to write tests with it. The main thing that got me was I had to worry much less about expressing assertions than before, and still have meaningful failure messages. (Spock has a lot of other features, but this is the main reason I think it's awesome. For example, many commenters mentioned it's also great for mocking, but I try to avoid mocking as much as I can, it's just not my cup of tea.)

But the thing is that I need to test Java, not Groovy applications. The Java/Groovy integration is really smooth, but it's still a different language, framework, editor-plugin to use and job to incorporate to the build flow. Not to mention that the projects I'd like to use it already has a lot of tests with different tools, so introducing another testing framework would make the stack more complex, and the tests more diverse.

So I really wanted a small, non-intrusive tool that help achieve a similar effect without the need to use another framework or language.

The way of Scott

So it works as follows.

• write tests in plain Java as you would do normally
• run them with JUnit or something as you would do normally
• include Scott to the build flow, that automatically creates the detailed failure messages for failing tests

So there is no need to use a new API or change anything in the tests, the only task to do is to include Scott to the pom.xml.

The error message contains the source code of the test case with some runtime information printed on it as comments. For example, the printed source has extra comment for every line that has a variable assignment that contains the new value for the variable. (Much like Chrome's debugger.) If a call to a method changes a local variable, the new value is printed too.

Detecting such changes are done simply by checking the string representations of the objects referenced by variables that are attached to the detailed report.

Using Scott, even simple assertions produce meaningful failure messages and a lot of extra information to reduce the need to debug a test.

Example

To demonstrate Scott, let's see a simple test case that exercises java.util.List a bit.

@Test
public void test_2() throws Throwable {
	Integer[] array = new Integer[] { 1, 4, 2, 3 };
	List<Integer> list = Arrays.asList(array);

	Collections.sort(list);

	assertArrayEquals(array, new Integer[] { 1, 4, 2, 3 });
}

This test fails. It says

test_2(hu.advancedweb.example.ListTest): arrays first differed at element [1]; expected:<2> but was:<4>

Let's add Scott to the mix and see what happens.

test_2(hu.advancedweb.example.ListTest) FAILED!
  26|   @Test
  27|   public void test_2() throws Throwable {
  28|           Integer[] array = new Integer[] { 1, 4, 2, 3 }; //array=[1, 4, 2, 3]
  29|           List<Integer> list = Arrays.asList(array); //list=[1, 4, 2, 3]
  30|
  31|           Collections.sort(list); //array=[1, 2, 3, 4] //list=[1, 2, 3, 4]
  32|
  33|           assertArrayEquals(array, new Integer[] { 1, 4, 2, 3 });
  34|   }

The List created by Arrays.aslist is backed by the array passed as an argument, so sorting the List affects the array too. Note for the curious: add an element to the list before sorting it, and see what happens.

How it works

Data about variables has to be collected at runtime. To achieve this Scott instruments the bytecode of the test methods on the fly during class loading with a Java Agent, and manipulates it with with ASM.

The instrumentation happens really fast, many other tool use them in the industry, for example the JaCoCo Java Code Coverage Library.

Scott inserts code to the test methods that has no effect but to record the interesting stuff happens at runtime (line number, variable name, new value, etc.).

These events are saved in a store object, and queried by a JUnit RunListener. Before every test it clears the event store, and after a failing test it constructs the report based on the runtime information and the source code of the test if it's available. This is usually the case when running unit tests.

Plans for the future

I think there are many features that would be great to implement in the future:

• Inspired by Spock, it would be really cool to show parameters passed to assert statements. Let the developer express assertions like this: assertTrue(myList.contains(1) && myList.size() > 5), and provide meaningful messages by dissecting the arguments.
• Currently it supports Maven and JUnit only, and tests must reside in the test directory to be discovered. In the future it would be great to support other tools as well.
• The reporting is fully text based. IDE support would be really nice.

Scott needs Your help!

Scott loves contributions, just visit the contribution guide for some notes on how to get started. If you are looking for issues that can get you started the development, check out the Issues marked with the help-wanted tag.

You can find Scott in this Github repository.