Testing Bash scripts with the Bats testing framework
First impressions using Bats
There are many testing frameworks for Bash, however, some of them are not actively maintained or are used only by a small group of people. Which one should we choose? Or should we just build our custom implementation?
Recently I started working on a project which is written in Bash and a bit more complex than the typical scripts I'm used to. The project already had some test scripts to exercise its behavior. As the test suite grew, common functionality---such as assertions---was extracted to make the tests easier to maintain.
It served us great in the beginning, but it quickly led to building our custom framework that lacked documentation and some basic features. This is why we decided to switch to an open-source testing framework.
Enter Bats
The first framework I've investigated was Bats. It is a well-established contender in the scene of Bash testing frameworks as it's been around since 2011 and it has a solid user base. The original repository was discontinued in 2016, but it was forked to the bats-core organization to ensure the project's maintenance and to collect useful third-party libraries.
Look and feel of the tests
Here's how a test case looks like:
@test "hello.sh should great the user" {
run src/hello.sh John
assert_output "Hello John"
}
The custom @test annotation makes test cases easier to read because one can use a proper sentence to describe
their intention instead of having to encode it to a function name. This syntax is special:
Bats transforms the test cases to valid Bash code before executing them.
Bats provides the run command to wrap command execution and capture its result code and output. It's convenient
because most of the assertions (like assert_output in the example) can work with these values without having
to pass them as an explicit argument.
Test cases work in "strict mode": they will pass if all commands in the test case finish with a zero exit code, otherwise it will be marked as a failure. To illustrate this, the previous snippet could be written without using Bats assertions:
@test "hello.sh should great the user" {
result=$(src/hello.sh John)
[ "$result" = "Hello John" ]
}
Bats itself embraces this pattern and recommends Bash conditions to express assertions. By default it does not come with any built-in assertions, they are defined in the bats-core/bats-assert extension.
Both approaches have their benefits. Generally, I prefer to use dedicated assertions because they make the tests easier to read, but occasionally it's quite nice that basically anything can be easily expressed with conditions.
Another thing to consider is error reporting. With the dedicated assertions the report contains the full context, including the expected and the actual values:
✗ hello.sh should great the user
(from function `assert_output' in file test/../lib/bats-assert/src/assert_output.bash, line 186,
in test file test/hello_test.bats, line 8)
'assert_output "Hello John"' failed
-- output differs --
expected: Hello John
actual : Hello Jane
This is not true for simple conditions:
✗ hello.sh should great the user
(in test file test/hello_test.bats, line 8)
`[ "$result" = "Hello John" ]' failed
Finally, when it comes to defining custom assertions, with conditionals it's possible to express almost anything. One thing to keep in mind is that tests will fail on the first non-zero exit code, so calls to helper commands have to be defined accordingly.
It's also possible to define custom assertions similar to the ones provided in bats-assert. The
bats-support library provides common functions for error reporting and
output formatting, and with load shared test code can be imported to the test cases
(doc).
Test file discovery
Tests can be executed simply by pointing Bats to the directory where the test files reside:
bats <test_dir_path>
It can also consider subdirectories with the --recursive flag. With --filter one can specify a regex, and only tests
with a matching name will be executed.
Bats only considers files with the .bats extension. This is not a huge problem, but editors have to be adapted a bit
to offer Bash syntax highlight and the usual features for these files.
Test isolation
It's an important feature of a test framework to protect tests from each other by preventing state leak. Bats achieves this by executing each test case in its own process.
With this, state, Bash options and mocks defined in tests are not visible to other test cases.
@test "first test" {
global="test"
[ "$global" = "test" ]
}
@test "second test" {
[ "$global" = "" ]
}
Regarding custom Bash options, Bats has no problem with sourcing scripts that set custom Bash options. Even if options
like -e or -u are set, it does not cause problems for the test framework. This is important when it comes to
unit testing individual functions as it ensures that they work
similarly in a unit test environment as they would when the script is executed normally.
Because a non-zero exit code results in a failed test, I thought that all unit tested functions are inherently run with
the errexit mode set. Luckily this is not the case. The run construct provides a sandbox to the function under test.
Usually this means that whatever Bash options you use, they will be applied to the unit tested functions without Bats
overriding anything.
One exception to this is the -e option, which seems to be always unset no matter how hard I try to pass it to the function
under test. However, I've had some surprises before
how -e works when testing functions, so I'm not blaming
Bats for this quirk.
The bad parts
One area where Bats could improve is reporting. Test cases are not separated by test files, which can make the output messy for larger projects involving multiple test files:
✓ this is the first test from the first test suite
✓ this is the second test from the first test suite
✓ this is the first test from the second test suite
✓ this is the second test from the second test suite
Also, while it's possible to define before/after hooks to run code around each test scenario, Bats does not support beforeAll/afterAll construct where global setup and teardown logic could be added.
Finally, although the bats-core project has good and up-to-date documentation, the third-party libraries, including bats-assert
seem to lag behind. In many cases, they link to old, obsolete repositories and documents that were not updated in the last couple of years.
Summary
So far I'm satisfied with Bats. It's a mature and feature-rich testing framework. Its huge user base and extensibility make it a very compelling choice, and none of its shortcomings was a dealbreaker for me.
