Recently I’ve been writing Selenium tests for a large web application. The tests cover the most important features, and the suite grows nicely, but some flaky tests causing much pain and making the progress slower.
These random errors are best eliminated, so I decided to add checks to detect client side log messages.
Detecting browser log messages
I considered two ways of reading client side log entries:
- Using the Selenium’s built in mechanism via DesiredCapabilities to query log messages.
There are many other ways of doing this, like opening the developer console and taking a screenshot of it or using a custom browser plug-in, but this time I’ve tried to keep things simple. I also did not want to modify the application’s error handling for the sake of the tests.
Reading the logs via Selenium API is reliable and simple. All it takes is to instantiate the WebDriver with the proper settings.
Then check the log messages:
Unfortunately, the only important information I was able to get through this API is the log message, not the line numbers or the stack traces. On the plus side, all error messages - including the ones during page load - can be acquired.
Testing small or extremely deterministic applications this might be enough, as simply reproducing the steps manually will bring forth the bug with all the relevant technical data.
Of course, errors contain much more information than just a message. One way to work around the problem with the previous method is to capture the exceptions in the browser. This can be a bit brittle and cumbersome, because it needs an agent to track errors and log messages and provide a way to query them. If the agent is subject to some kind of abuse, it might sabotage the log recording or even break the tests. Because I didn’t want to modify the system under test, I inject the agent to the browser after every page load. This means the agent is unable to provide information about errors that occurred before the page is completely loaded.
As you can see, the agent keeps track of not only the errors and log messages, but also preserves the associated stack traces. For practical reasons the agent above can be injected many times to the same page without causing any problems. This is handy, because it eliminates the need to keep track whether the script is already injected or not.
The captured events can be checked easily with the following code:
Making assertions and reports from client side logs
This approach has some limitations though, as it can work properly for testing one page applications only. When a test scenario performs an action that leads to another webpage, all previously collected data simply vanish with the fresh page load.