Detecting errors in the browser with Selenium

Recently I've been writing Selenium tests for a large web application. The tests cover the most important features, and the suite grows nicely, but some flaky tests causing much pain and making the progress slower.

While investigating the problematic scenarios I've found that in many cases errors arise in the browser when a test randomly fails. Moreover, many passing tests also produce Javascript errors occasionally. In the latter cases the bug not surface in the test, but might cause problems later on.

These random errors are best eliminated, so I decided to add checks to detect client side log messages.

Detecting browser log messages

I considered two ways of reading client side log entries:

• Using the Selenium's built in mechanism via DesiredCapabilities to query log messages.
• Injecting a Javascript agent after every page load to the application under test that tracks errors and log messages.

There are many other ways of doing this, like opening the developer console and taking a screenshot of it or using a custom browser plug-in, but this time I've tried to keep things simple. I also did not want to modify the application's error handling for the sake of the tests.

Using DesiredCapabilities

Reading the logs via Selenium API is reliable and simple. All it takes is to instantiate the WebDriver with the proper settings.

LoggingPreferences logs = new LoggingPreferences();
logs.enable(LogType.BROWSER, Level.SEVERE);

DesiredCapabilities desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setCapability(CapabilityType.LOGGING_PREFS, logs);

WebDriver driver = new FirefoxDriver(desiredCapabilities);

Then check the log messages:

LogEntries logEntries = driver.manage().logs().get(LogType.BROWSER);

for (LogEntry logEntry : logEntries) {
	// Get log message, timestamp and level.
}

Unfortunately, the only important information I was able to get through this API is the log message, not the line numbers or the stack traces. On the plus side, all error messages - including the ones during page load - can be acquired.

Testing small or extremely deterministic applications this might be enough, as simply reproducing the steps manually will bring forth the bug with all the relevant technical data.

Using a Javascript agent

Of course, errors contain much more information than just a message. One way to work around the problem with the previous method is to capture the exceptions in the browser. This can be a bit brittle and cumbersome, because it needs an agent to track errors and log messages and provide a way to query them. If the agent is subject to some kind of abuse, it might sabotage the log recording or even break the tests. Because I didn't want to modify the system under test, I inject the agent to the browser after every page load. This means the agent is unable to provide information about errors that occurred before the page is completely loaded.

The agent can be injected with the executeScript method provided by the JavascriptExecutor interface.

The Javascript code of the agent can be something like this:

(function() {
	if (window.logEvents) return;

	function captureUnhandled(errorMsg, url, lineNumber, column, errorObj) {
		var logMessage = new Date() + ' unhandled - ' + errorMsg + ', ' + errorObj.stack;
		window.logEvents.push(logMessage);
	}

	function capture(level) {
		return function() {
			var args = Array.prototype.slice.call(arguments, 0);
			var logMessage = new Date() + ' ' + level + ' - ';
			for(var i=0; i<args.length; i++) {
				if (args[i] instanceof Error) {
					logMessage += args[i].message + ', ' + args[i].stack;
				} else {
					logMessage += args[i];
				}
			}
			window.logEvents.push(logMessage);
		}
	}
	console = console || {};
	console.warn = capture('warn');
	console.error = capture('error');
	window.onerror = captureUnhandled;
	window.logEvents = [];
}());

As you can see, the agent keeps track of not only the errors and log messages, but also preserves the associated stack traces. For practical reasons the agent above can be injected many times to the same page without causing any problems. This is handy, because it eliminates the need to keep track whether the script is already injected or not.

The captured events can be checked easily with the following code:

	List<String> logEvents = (List<String>) ((JavascriptExecutor)driver).executeScript("return window.logEvents;");
	
	for (String logEvent : logEvents) {
		// The logEvent String contains all collected information.
	}

Making assertions and reports from client side logs

Because each method has some drawbacks, I use them both. Right now I just store the client side log information in the Cucumber test report to aid manual inspection of the failing tests, but it could be used to assert that there are no exceptions in the Javascript application during the tests.

This approach has some limitations though, as it can work properly for testing one page applications only. When a test scenario performs an action that leads to another webpage, all previously collected data simply vanish with the fresh page load.