How Validas performs tool classification & qualification shown based on the example of a compiler
Tool classification and qualification is a two-stage process. To determine if the tool needs to be qualified, it is first classified. The process described below is applicable for proprietary and open-source compilers alike.
To classify a tool, Validas analyzes the context in which the tool is used. Most compilers are not stand-alone tools but work as part of a tool chain. They produce an output (for example *.o) for other tools and can consume inputs (for example *.c) from tools like Code Generators. The following image shows an example tool chain:
The tool chain is highly customer-specific and can also contain other tools, for example code checkers, diagnostic tools or a build system. Knowing the design of the tool chain helps to understand the tool usage and makes it easier to derive countermeasures.
Validas analyzes the configuration and the Use Case of the tool. The Use Case describes how the tool is used and which features of the tool are selected. In this example, the compiler (GCC) is used from command line and the call command looks as follows:
For each selected feature of the tool, the potential malfunctions need to be determined as well as their impact on functional safety. For “-fdata-sections” a malfunction could be that the compiler fails to put each data item in its own section. It could be argued that this property of the generated object file has no impact on safety. But in this example, it has an impact and therefore this malfunction is critical.
After identifying all potential tool malfunctions, Validas starts defining appropriate countermeasures for each critical, potential malfunction. For the described malfunction of “fdata-sections”, a valid mitigation could be code and binary review to check if each data item was put in a separate section. Mitigations like these are very time-consuming, difficult to implement and therefore rarely found in compiler Safety Manuals. When all necessary countermeasures for the tool are derived, Validas adds an appropriate Anomalous Operating Condition Handling and a Known Bug Analysis as well.
Based on the collected information, the compiler can be classified. If critical and potential malfunctions cannot be covered by an appropriate mitigation, the tool needs to be qualified. Most compilers are qualified by requirements-based testing.
During qualification, Validas analyzes all critical and potential malfunctions of the compiler and creates requirements-based test suites for them. One core feature of a compiler is transforming source code to object code according to a selected language standard (in this example C99). To test the possible malfunctions for this feature, there are already commercial test suites available. One of them SuperTest from the company SolidSands, which Validas used already in several previous compiler qualification projects.
Covering compiler optimizations is challenging and critical in many projects. Most Use Cases contain bundled optimization options like “-O2”. The best approach to qualify these features would be to develop test suites covering the potential malfunctions from every option contained in “-O2”. However, this approach is sometimes not possible due to high costs or because not all the bundled options are known. This is the case for many proprietary compilers designed for small targets and high optimization. To cover this, Validas proposes different solutions to argue for sufficient test coverage.
After test analysis, Validas gathers the collected tool data and the information about the created artifacts in a Tool Chain Analyzer-model. This model and the created artifacts are then bundled in a Qualification Kit (QKit) with a project-specific Test Automation Unit. The QKit performs test execution, automated test analysis and generates the necessary reports.
By using the Validas Qualification Kit you can adapt or extend your qualification and quickly requalify your tools. With this, you are well equipped for your safety assessment and are capable of supporting dynamic software development.