Testing

Catch Test Framework

A powerful tool to facilitate writing test for our software is the Catch Test Framework. Before you start reading this page, it is strongly recommended that you read the tutorial on the Catch Github repo, as well as the Reference.
Please note that the version of Catch used in our codebase is Catch 1.x.

Using Catch in Skyward-Boardcore

Test source file

Tests that use catch must be located under the src/test/catch1 directory in skyward-boardcore.
Source names must start with test and words-must-be-separated-by-dashes. Example: test-temperature-sensor.cpp.
Once you have created your source file, the first thing is to add the following line of code:

#ifdef  STANDALONE_CATCH1_TEST
#include  "catch1-tests-entry.cpp"
#endif

#include  <catch.hpp>

The first 3 lines are needed in the case you want to run the test by itself, defining its own entrypoint. More on this in the next section. Once you've added this 4 lines of code, you are ready to write your tests. Take a look at the examples in the Catch Github repo linked above, and at the examples in src/tests/examples in skyward-boardcore and the next sections to know how to proceed.

Entrypoint configuration

To run the tests, you first need to define the entrypoint in sbs.conf. You will find two types of entrypoints for a test:

Standalone test

This is a test that will run alone. Useful when you are writing the test and don't want to wait for all other tests to run together with the one you are writing.
The entrypoint for the is defined as follows:

[example-test-factorial] #name of the entrypoint
Type: test
BoardId: stm32f407vg_stm32f4discovery #The board you want to run the test on
BinName: example-test-factorial #Name of the binary, must be the same as the name of the entrypoint
Include: #Required sources, if any
Defines: -DSTANDALONE_CATCH1_TEST
Main: examples/example-test-factorial #Relative path of the .cpp file, without extension

Note that, in order to run the test by itself, we need to define STANDALONE_CATCH1_TEST.

Grouped tests

Before committing changes, especially to the master branch, we want to be sure that no problems are present in the code. We could run all the tests one by one, but compiling, flashing and executing them one by one would be a nightmare as the number of tests increase. Fortunately, the Catch framework allows for tests defined in different source files to run together with little to no effort required by the developer.
In fact, in order to do so, we just to need to compile the test source files together with the catch1-tests-entry.cpp entrypoint source.
The configuration in sbs.conf will look like this:

#Add here all the source files of the tests you want to run
[example-tests]
Type: srcfiles
Files: src/tests/examples/example-test-factorial
	   src/tests/examples/example-test-fsm

# Run all example tests togheter
[example-catch1-tests]
Type: test
BoardId: stm32f407vg_stm32f4discovery
BinName: example-catch1-tests
#Add all required sources + tests sources
Include: %shared %example-tests 
Defines:
Main: catch1-tests-entry

When you run example-catch1-tests, both the tests in example-test-fsm.cpp and example-test-factorial.cpp will run.

Test execution configuration

Catch provides some command line options to configure the test output and decide which tests to run. The options are described in the command line reference page on the Catch Github repo. In boardcore, these command line options are passed via a define in sbs.conf: CATCH1_CL_OPTIONS. Usage: -DCATCH1_CL_OPTIONS="\"[tag1][tag2] -a -b -c\"" Example:

[example-catch1-tests]
Type: test
BoardId: stm32f407vg_stm32f4discovery
BinName: example-catch1-tests
#Add all required sources + tests sources
Include: %shared %example-tests
Defines: -DCATCH1_CL_OPTIONS="\"[homeone] -s\""
Main: catch1-tests-entry

Patterns & Best Practices

Test Setup & Teardown

In many cases you might have to setup objects before executing tests, and then tear them down (for example cleaning up dynamically allocated memory) after the test has finished executing.
There are many ways to achieve this, some good and some bad. Here are a few examples:

BAD: Setup at the beginning of the test case, teardown at the end

This might, at a first glance, look like a very simple and clean way of achieving what we are trying to do:

TEST_CASE("This will leak memory")
{
	// Setup: 
	char* array = new char[20]();

	// Test case
	SECTION("Example section")
	{
		REQUIRE(array[0] == 5); //This will fail, array is zero-initialized
	}
	
	// Teardown
	delete[] array;
}

Where's the catch? If a REQUIRE macro fails, the whole test case is interrupted, the line delete[] array; will not be executed, and memory will be leaked.

GOOD: RAII (Resource Acquisition Is Initialization)

This is a pattern that you should use in all the code base, and not just for testing: any object should acquire resources in its constructor, and release them in its destructor. The destructor is automatically called when the object goes out of scope (of course you can't use new to create the object, or the whole purpose of this point is defeated, and we get back at the previous example).

// Wrapper for the array
class TestArrayWrapper
{
public:
	TestArrayWrapper()
	{
		array = new char[20];
	}
	~TestArrayWrapper()
	{
		delete[] array;
	}
	char* array;
};

TEST_CASE("This will not leak memory")
{
	//Setup
	TestArrayWrapper arr;

	SECTION("Example section")
	{
		REQUIRE(arr.array[0] == 5); //This will fail
	}
	//Teardown
	//You don't have to do anything:
	//arr, and consequently arr.array, will be automatically destroyed here
}

In most cases you won't need to write a wrapper: most classes (especially in the standard library) already use RAII (For example, in this case, you could have used the std::array container)

GOOD: Test Fixtures

An approach much similar to the previous are test fixtures: before proceeding, read their reference page on the Catch Github.

class ArrayTestFixture
{
public:
	ArrayTestFixture()
	{
		array = new char[20];
	}
	~ArrayTestFixture()
	{
		delete[] array;
	}
protected:
	char* array;
};

// This created a new class derived from ArrayTestFixture, with this method 
// as a member.  Therefore this method has access to protected 
// ArrayTestFixture members
TEST_CASE_METHOD(ArrayTestFixture, "This will not leak memory")
{
	// Setup
	// Nothing to do, we are a member of ArrayTestFixture and we 
	// have access to its other members

	SECTION("Example section")
	{
		REQUIRE(array[0] == 5); //This will fail
	}
	// Teardown
	// You don't have to do anything:
	// The fixture class will be detroyed after this method terminates
}