JUnit and Cucumber test reports based on source code and behavior

Scott provides detailed failure messages for Java tests and does so without the use of complex assertion libraries. It can greatly increase productivity, because it allows the developers to express the tests more naturally without losing the much-needed information that tells what went wrong in a scenario.

Consider the following test with its dead simple assertion:

As you can see, all information is presented on the source code of the test method as comments. All important data is captured by this report even without specialized assertions.

Access to this data is really beneficial if you catch a broken test on the CI server and you don't have anything else but the failure report to reproduce the issue.

This is not bad, but the most recent version of Scott takes it one step further.

Cucumber meets Scott

Today I'd like to write about a new feature that can help to trace issues with end-to-end tests. For a baseline, this is how Cucumber reports an error:

Just dropping Scott into the mix, the very same test produces a more informative report:

Scott for Cucumber tracks whole scenarios, and in case of a failure, it prints the details of every step involved. This is true even if the same glue method is involved in multiple steps. Because E2E tests are slower than unit and API tests, usually it's not feasible in a large system to run all of them locally after each change. Moreover, they are usually more prone to timing related issues, resulting in flaky tests.

This means that it's much more likely to see a failing E2E test in the CI environment than a failing unit test. Scott's report can provide valuable information to make it easier to reproduce and fix these issues based on the error messages they produce.

The above example uses Cucumber's HTML reporter to present the enhanced reports, but the other renderers are supported as well, so the information is accessible in the Jenkins build or the command line.

Try it

I hope you enjoyed this post, and give Scott a try. This example project contains the necessary configuration to use Scott with Cucumber. See it's pom.xml for the project setup and the FeatureTest class for the @CucumberOptions. For more information on about how to use Scott, please the usage guide.

In the future, I plan to extend Scott's capabilities with support for more testing tools and frameworks. If you have suggestions, please open an issue.