Overview

A model-based safety testing approach was successfully applied to the module testing of software for motor current control.

Using this method it was possible to test the motor control loop software in a more effective and efficient way than would be feasible using conventional unit testing approaches. A large number of tests could be run on complete control loops over multiple execution cycles, providing the level of verification appropriate for software in safety-related applications.

Objective

The need was to carry out effective, efficient and repeatable testing of embedded motor controller software, to a depth suitable for being able to use the motor in safety-related applications. The code had already been written, and was stable. Our aims were to provide a sound basis to support the argument that the software was suitable for use in safety-related contexts, and to provide a basis for continuous integration testing.

Factors to be considered

Verifying that complex current control algorithms are correctly implemented solely by running on the final target hardware is not an effective approach. It is difficult to test the software across its full operational envelope on a hardware test system. It is therefore necessary to also unit/module test the software to demonstrate that the specified algorithms are correctly implemented, as a small error in the implementation of these algorithms can have a significant effect on the performance of the motor, and in the worst-case could result in outcomes with potential safety consequences such as sudden loss of torque.

To unit or module test software requires knowledge of the expected outputs for each test case. For complex real-time control loop software in which the current output depends on previous outputs as well as current inputs, calculating the expected outputs for all test cases can be a very laborious task. To achieve the level of confidence that we need for using the software in safety-related applications we need to run a large number of tests. We would like to test behaviour across multiple executions of the loop, so we need to know what the expected outputs across time are, for a set of inputs that may also be changing across time.

Approach

A testing method was designed that exercised the software at the level of complete control loops. This was considered to be more effective than testing only at the level of individual software functions, due to the high level of interdependency between the functions. Also, by testing the complete integrated control loop, the number of tests that have to run and maintained. The internal design of the loop software can be re-factored, for example to optimize execution speed, without the tests needing to change.

It was decided to test using multiple executions of the control loops. The control algorithms include integral terms that are maintained between loop executions. Testing by executing the loop multiple times allows us to verify the correct behaviour of these integrators.

It was decided to use model-based safety testing with an executable mathematical model to generate the expected results of the tests. This reduced the amount of manual calculation needed, and by basing our models on models developed earlier by the control engineers, allowed us to verify the software against those models.

The model-based safety testing process

Build executable mathematical models

The first step was to creating the executable mathematical models that would be our “oracle” for generating expected results for our tests.

The open-source modeling tool Scicos was selected. This provides a graphical notation suitable for describing control algorithms. An example of a proportional-integral algorithm with integrator “wind-up” protection and output limiting is shown below.

The models were created from existing Simulink models of the control system. These were then refined to provide a precise alignment with the expected output of the software. This was necessary firstly because the Simulink models are analogue whereas parts of the software implementation are implemented in fixed-point to reduce execution time, and secondly because some features of the implemented control loops, that are necessary in a real system, such as anti-windup and over-current protection were not present in the Simulink models.

Build the test harness

The test approach required several things over and above run-of-the-mill unit testing:

1. Test a set of interrelated software functions together, while “mocking” others
2. Run the software functions under test multiple times and check outputs on each iteration
3. Easily pull the output from the execution of mathematical models into the test harness as expected results

To be able to do these things, giving maximum control over test execution (and saving costs on testing tools) a custom test harness was written in ‘C’ (which was the language of the software under test). Testing was driven from data structures that were created from the mathematical model execution output. These data specified how many times the control loop was to be exercised, and the expected outputs to each of the “mock” functions on each iteration. GNU gcov was used for coverage checks, with test failures reported for low coverage. The harness also calculated cyclomatic complexity and other metrics for the code under test with test failures reported for out-of-range metrics. Test output was in XML format that could be read in by Jenkins.

Design the tests

The test design was based on the requirements of the international standard for functional safety IEC 61508. The primary test requirement was complete black-box testing. In addition, boundary-value analysis and equivalence-class analysis were carried out and additional test cases added from these.

Execute the test cases in the models

Once the test cases were specified, input data sets to the mathematical models were created for them, and the models executed to generate expected outputs for each test case in text files.

Build the test programs

The output text files from the execution of the mathematical models were converted into ‘C’ data structures. Then the test harness, code under test, and these data structures were combined into test programs that covered the complete set of test cases.

The test programs were kept in the same Git repository as the software under test.

Run the tests

Jenkins was used for nightly building and execution of the test programs. Any test failures were automatically reported for corrective action. This was part of a complete nightly build of the complete product software.

The results

It was possible, with model-based safety testing using executable mathematical models and a custom testing harness, to test the motor control loop software in a more effective and efficient way than would be feasible using conventional unit testing approaches. A large number of tests could be run on complete control loops over multiple execution cycles, providing the level of verification appropriate for software in safety-related applications.

Most of the test code created was generic and re-usable. The need to write and maintain a large number of individual custom test scripts was avoided.

The testing was “black box” only. Access to internal variables was not needed. This will reduce the maintenance overhead on this testing when changes are made to the implementation.

The tests were easily incorporated into a Jenkins based continuous integration system.

There was no need to purchase any expensive software to be able to achieve this.