|
|
# Testing
|
|
|
## Catch Test Framework
|
|
|
A powerful tool to facilitate writing test for our software is the [Catch Test Framework](https://github.com/catchorg/Catch2/tree/Catch1.x). In this page I will provide basic examples on how to use it in our codebase, but it is strongly recommended that you read the [tutorial on the Catch Github repo](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/tutorial.md), as well as the [Reference](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/Readme.md).
|
|
|
Please note that the version of Catch used in our codebase is Catch1.x.
|
|
|
A powerful tool to facilitate writing test for our software is the [Catch Test Framework](https://github.com/catchorg/Catch2/tree/Catch1.x). Before you start reading this page, it is strongly recommended that you read the [tutorial on the Catch Github repo](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/tutorial.md), as well as the [Reference](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/Readme.md).
|
|
|
Please note that the version of Catch used in our codebase is Catch 1.x.
|
|
|
|
|
|
## Testing strategy (WIP)
|
|
|
|
|
|
## Writing tests
|
|
|
## Using Catch in Skyward-Boardcore
|
|
|
### Test source file
|
|
|
Tests that use catch must be located under the src/test/catch1 directory in skyward-boardcore.
|
|
|
Source names must start with `test` and words-must-be-separated-by-dashes. Example: `test-temperature-sensor.cpp`.
|
|
|
Once you have created your source file, the first thing is to add the following line of code:
|
|
|
```c
|
|
|
```cpp
|
|
|
#ifdef STANDALONE_CATCH1_TEST
|
|
|
#include "catch1-tests-entry.cpp"
|
|
|
#endif
|
... | ... | @@ -18,43 +18,7 @@ Once you have created your source file, the first thing is to add the following |
|
|
#include <catch.hpp>
|
|
|
```
|
|
|
The first 3 lines are needed in the case you want to run the test by itself, defining its own entrypoint. More on this in the next section.
|
|
|
Once you've added this 4 lines of code, you are ready to write your tests. Take a look at the examples in the Catch Github repo linked above, and at the examples in `src/tests/examples` in skyward-boardcore to know how to proceed.
|
|
|
|
|
|
### Order of execution
|
|
|
One thing that may not be clear is: how are test cases and sections executed by the Catch framework?
|
|
|
When you write a *TEST_CASE* with multiple sections inside it, the whole *TEST_CASE* "function" is executed **once for each section**. This means that all the code written outside sections is repeated for each section. Take a look at this example:
|
|
|
```cpp
|
|
|
TEST_CASE("Order of execution", "")
|
|
|
{
|
|
|
printf("Setup\n");
|
|
|
SECTION("S1")
|
|
|
{
|
|
|
printf("Executing the first section\n");
|
|
|
REQUIRE(true);
|
|
|
}
|
|
|
SECTION("S2")
|
|
|
{
|
|
|
printf("Executing the second section\n");
|
|
|
REQUIRE(true);
|
|
|
}
|
|
|
|
|
|
printf("Teardown\n");
|
|
|
}
|
|
|
```
|
|
|
The output will be:
|
|
|
```
|
|
|
Setup
|
|
|
Executing the first section
|
|
|
Teardown
|
|
|
Setup
|
|
|
Executing the second section
|
|
|
Teardown
|
|
|
```
|
|
|
|
|
|
This has a few consequences:
|
|
|
- You can create and *setup* objects writing the setup code at the beginning of the test case. For example, you can create a state machine and bring it to a particular state you want to test.
|
|
|
- When a section starts executing, it will find the object exactly in the same state as you set it up. Changes to the object in previous sections are not reflected in other sections, because the object is effectively destroyed and recreated for each section.
|
|
|
- *Teardown* code can be written at the end of the test case. You can clear resources here, for example deleting a dinamically allocated object, in order not to leak memory each time a section is executed.
|
|
|
Once you've added this 4 lines of code, you are ready to write your tests. Take a look at the examples in the Catch Github repo linked above, and at the examples in `src/tests/examples` in skyward-boardcore and the next sections to know how to proceed.
|
|
|
|
|
|
### Entrypoint configuration
|
|
|
To run the tests, you first need to define the entrypoint in `sbs.conf`. You will find two types of entrypoints for a test:
|
... | ... | @@ -64,14 +28,11 @@ The entrypoint for the is defined as follows: |
|
|
```tcl
|
|
|
[example-test-factorial] #name of the entrypoint
|
|
|
Type: test
|
|
|
#The board you want to run the test on
|
|
|
BoardId: stm32f429zi_stm32f4discovery
|
|
|
#Name of the binary, must be the same as the name of the entrypoint
|
|
|
BinName: example-test-factorial
|
|
|
BoardId: stm32f429zi_stm32f4discovery #The board you want to run the test on
|
|
|
BinName: example-test-factorial #Name of the binary, must be the same as the name of the entrypoint
|
|
|
Include: #Required sources, if any
|
|
|
Defines: -DSTANDALONE_CATCH1_TEST
|
|
|
#Relative path of the .cpp file, without extension
|
|
|
Main: examples/example-test-factorial
|
|
|
Main: examples/example-test-factorial #Relative path of the .cpp file, without extension
|
|
|
```
|
|
|
Note that, in order to run the test by itself, we need to define `STANDALONE_CATCH1_TEST`.
|
|
|
#### Grouped tests
|
... | ... | @@ -92,7 +53,7 @@ Type: test |
|
|
BoardId: stm32f429zi_stm32f4discovery
|
|
|
BinName: example-catch1-tests
|
|
|
#Add all required sources + tests sources
|
|
|
Include: %shared %example-tests
|
|
|
Include: %shared %example-tests
|
|
|
Defines:
|
|
|
Main: catch1-tests-entry
|
|
|
```
|
... | ... | @@ -110,7 +71,104 @@ Type: test |
|
|
BoardId: stm32f429zi_stm32f4discovery
|
|
|
BinName: example-catch1-tests
|
|
|
#Add all required sources + tests sources
|
|
|
Include: %shared %example-tests
|
|
|
Include: %shared %example-tests
|
|
|
Defines: -DCATCH1_CL_OPTIONS="\"[homeone] -s\""
|
|
|
Main: catch1-tests-entry
|
|
|
```
|
|
|
## Patterns & Best Practices
|
|
|
### Test Setup & Teardown
|
|
|
In many cases you might have to setup objects before executing tests, and then tear them down (for example cleaning up dynamically allocated memory) after the test has finished executing.
|
|
|
There are many ways to achieve this, some good and some bad. Here are a few examples:
|
|
|
|
|
|
#### BAD: Setup at the beginning of the test case, teardown at the end
|
|
|
This might, at a first glance, look like a very simple and clean way of achieving what we are trying to do:
|
|
|
```cpp
|
|
|
TEST_CASE("This will leak memory")
|
|
|
{
|
|
|
// Setup:
|
|
|
char* array = new char[20]();
|
|
|
|
|
|
// Test case
|
|
|
SECTION("Example section")
|
|
|
{
|
|
|
REQUIRE(array[0] == 5); //This will fail, array is zero-initialized
|
|
|
}
|
|
|
|
|
|
// Teardown
|
|
|
delete[] array;
|
|
|
}
|
|
|
```
|
|
|
Where's the catch? If a *REQUIRE* macro fails, the whole test case is interrupted, the line `delete[] array;` will not be executed, and memory will be leaked.
|
|
|
|
|
|
#### GOOD: RAII (Resource Acquisition Is Initialization)
|
|
|
This is a pattern that you should use in all the code base, and not just for testing: any object should acquire resources in its constructor, and release them in its destructor. The destructor is automatically called when the object goes out of scope (of course you can't use *new* to create the object, or the whole purpose of this point is defeated, and we get back at the previous example).
|
|
|
|
|
|
```cpp
|
|
|
// Wrapper for the array
|
|
|
class TestArrayWrapper
|
|
|
{
|
|
|
public:
|
|
|
TestArrayWrapper()
|
|
|
{
|
|
|
array = new char[20];
|
|
|
}
|
|
|
~TestArrayWrapper()
|
|
|
{
|
|
|
delete[] array;
|
|
|
}
|
|
|
char* array;
|
|
|
};
|
|
|
|
|
|
TEST_CASE("This will not leak memory")
|
|
|
{
|
|
|
//Setup
|
|
|
TestArrayWrapper arr;
|
|
|
|
|
|
SECTION("Example section")
|
|
|
{
|
|
|
REQUIRE(arr.array[0] == 5); //This will fail
|
|
|
}
|
|
|
//Teardown
|
|
|
//You don't have to do anything:
|
|
|
//arr, and consequently arr.array, will be automatically destroyed here
|
|
|
}
|
|
|
```
|
|
|
In most cases you won't need to write a wrapper: most classes (especially in the standard library) already use RAII (For example, in this case, you could have used the [std::array](https://en.cppreference.com/w/cpp/container/array) container)
|
|
|
|
|
|
#### GOOD: Test Fixtures
|
|
|
An approach much similar to the previous are test fixtures: before proceeding, read their reference page on the [Catch Github](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/test-fixtures.md).
|
|
|
|
|
|
```cpp
|
|
|
class ArrayTestFixture
|
|
|
{
|
|
|
public:
|
|
|
ArrayTestFixture()
|
|
|
{
|
|
|
array = new char[20];
|
|
|
}
|
|
|
~ArrayTestFixture()
|
|
|
{
|
|
|
delete[] array;
|
|
|
}
|
|
|
protected:
|
|
|
char* array;
|
|
|
};
|
|
|
|
|
|
// This created a new class derived from ArrayTestFixture, with this method
|
|
|
// as a member. Therefore this method has access to protected
|
|
|
// ArrayTestFixture members
|
|
|
TEST_CASE_METHOD(ArrayTestFixture, "This will not leak memory")
|
|
|
{
|
|
|
// Setup
|
|
|
// Nothing to do, we are a member of ArrayTestFixture and we
|
|
|
// have access to its other members
|
|
|
|
|
|
SECTION("Example section")
|
|
|
{
|
|
|
REQUIRE(array[0] == 5); //This will fail
|
|
|
}
|
|
|
// Teardown
|
|
|
// You don't have to do anything:
|
|
|
// The fixture class will be detroyed after this method terminates
|
|
|
}
|
|
|
``` |
|
|
\ No newline at end of file |