Decoding Binary Logs

Once we have logged all our data in a .log binary file, we need to translate it in a human-readable format. This is usually the CSV (Comam Separated Values) format.

To do so, a C++ program called LogDecoder is used. The main source is located in skyward-boardcore/src/shared/logger/decoder/logdecoder.cpp. There is a little setup required before being able to use LogDecoder to decode our logs:

1. Add decoding functions to each logged struct

Every data type (usually structs) logged in the binary file must have two functions needed to convert binary data to CSV:

1. static void header(std::ostream& os): writes the header of the CSV file (first row containing column names) on the output stream.
2. void print(std::ostream& os): writes a row of the CSV file, filled with the data currently stored in the instance of the struct.

For example:

struct GyroscopeData
{
    uint64_t timestamp;
    float gyro_x;
    float gyro_y;
    float gyro_z;

    uint8_t flag;

    static std::string header()
    {
        // Just returns the name of the columns of the csv
        return "timestamp,gyro_x,gyro_y,gyro_z,flag\n";
    }

    void print(std::ostream& os)    
    {
        // Don't forget commas between the values!
        os << timestamp << "," << gyro_x << "," << gyro_y << "," 
        << gyro_z << "," << (int)flag << "\n";
    }
};

Important things to keep in mind:

• Don't forget commas between columns
• Don't forget to add a newline at the end of each row (including the header)
• Char-type variables like int8_t and uint8_t need to be casted to int before printing in the output stream, in order to avoid printing special characters that could cause problems in some text editors or data processing tools. (e.g. if flag == 0 it would print the first character in the ASCII table: '\0')

2. Register logged data types

The deserializer needs to know each and every data type that is being logged in order to be able decode it.
To do this, we need to create/modify the LogTypes.h file and add a new line for each struct in the registerTypes(...) function. An example can be found here.
The line to be added it's the same for each data type, changing only the name:

    ds.registerType<MyDataType>(print<MyDataType>, MyDataType::header());

Replace MyDataType with the name of the structs that are being logged.

Expanding on the first example, our function would look like this:

void registerTypes(Deserializer& ds)
{
    // The logger automatically logs its stats, so this first line is always needed
    ds.registerType<LogStats>(print<LogStats>, LogStats::header());

    ds.registerType<GyroscopeData>(print<GyroscopeData>,
                                         GyroscopeData::header());
}

Important things to keep in mind:

• Remember to register the LogStats struct, as the logger automatically logs its statistics once every second
• Remember to include the headers containing the definitions of your data types

3. Compiling LogDecoder

The deserializer is compiled on your pc using a standard GCC toolchain, not the miosix one. The only caveat is that it needs to be compiled on a Linux machine, as Linux has the same byte order as Miosix on ARM processors for data stored in memory. In Windows the byte order is different, and the deconding procedure will fail.

To compile LogDecoder, just run make in its directory.

4. Decoding logs

To decode just one log in the directory in which the binary log file is placed:

logdecoder path/to/logXX.dat

or

logdecoder -a

to decode all the logs in the current directory (it will decode only logs with file name in the form logXX.dat, with XX from 0 to 99)

If the decoding process finishes with no errors, one .csv file for each data type logged will be created.
If there are problems, the log may be corrupted or, most probably, you forgot to register some data types in step 2.