Tool Qualification Symposium 2014 - Abstracts

Munich, 9-10 April 2014

See below the abstracts of accepted presentations. For further details see also the call for presentations.

Keynote Speech: The Economics of Tool Qualification

Author: Matteo Bordin (Adacore)

Abstract:

This talk focuses on the economic analysis of preparating off-the-shelf tool qualification kit. It provides the point-of-view of a tool provider working with major industrial clients in several industrial domains. Our goal is to efficiently produce tool qualification kits that can be easily adapted to different clients and updated upon new tool releases.

The analysis is based on the experience accumulated during the qualification several tools, among others:

The presentation will outline AdaCore productisation process starting from the planning process, the inception of tool operation requirements, the tool development and its verification.

Particular emphasis will be given to the qualification of tools based on open-source technologies and to the long-term management and evolution of legacy versions of qualified tools.
This presentation is intended to provide a pragmatic and direct viewpoint on the tool qualication problem.

Back to top

Invited Talk: TPT - Qualification Kit

Author: Reny Grönberg (PikeTec GmbH)

Abstract:

TPT is a model-based tool environment for the test embedded systems, in particular for requirements based testing control and regulation systems. TPT supports all important areas of the test process. In detail, these are the areas of test management, test case modeling, performing tests, test assessment and test documentation.

ISO 26262 "Functional Safety - Road vehicles" is a new international standard that applies worldwide for all automobile manufacturers (OEMs) and their suppliers. If TPT is used in the automotive sector, a qualification of the tool is required in accordance with ISO 26262-8. The qualification is done in two steps. First an analysis and classification of the tool must be performed. Then, if necessary, the required qualification measures will be derived and performed.

A tool can only be qualified by embedding it into a development process, which also means integrating the tool into a specific safety lifecycle. This implies that the user must perform a tool-qualification in a practical project environment. The tool manufacturer/producer has to show that the tool is qualifiable.

To simplify the qualification-process and to reuse knowledge, a preliminary tool qualification can be reasonable, if the application of the tool can be derived from a standard configuration.

The use of TPT can be derived from a standard configuration. To show this and to minimize the tool user's effort, Piketec offers a TPT-"Qualification-Kit". The Qualification-Kit includes reference use cases, as well as a validation suite, which can be executed in the customer's environment. Thereby it can be ensured that TPT has the desired reliability and quality in the customer specific run-time environment.

Back to top

Invited Talk: Experiences of Using Formal Methods for Tool Qualification

Authors: Arnaud Dieumegard / Andres Toom / Marc Pantel (IRIT/ENSEEIHT)

Abstract:

As embedded systems are getting more and more complex, tools play a key role in their development. This leads to a growing cost in tools development and qualification. Managing such development activities is a challenge that can no longer be efficiently tackled using classical approaches like test or peer review due to their lack of exhaustivity and efficiency. Tool development based on formal methods allow providing both a higher confidence and the required data for the qualification of the tool. While they requires a strong knowledge of the underlying concepts, it is possible to ease their access with model driven engineering technologies. This talk will relate some applications of formal methods for the development of tools for safety critical embedded systems software development and the possible management of certification and qualification requirements.

Back to top

Leveraging Tool Pre-Qualification in the context of DO-178C and DO-330

Author: Dr. Udo Brockmeyer (BTC Embedded Systems AG)

Abstract:

The development of safety critical software requires implementation of rigorous processes with strong emphasis on risk analysis and safety requirements. Such process frameworks are described in standards like ISO 26262 for the automotive domain, or IEC 61508 for electronic systems, or DO- 178B/C for Aerospace systems. When developing software according to these safety standards, for each tool used to develop and test such safety critical software, a use case based risk assessment must be performed. Depending on this assessment, a tool might need to be qualified to get confidence that the tool does not add additional risk into a development process. This presentation shows how end users can leverage from tool pre-qualification performed by a tool vendor. In this concrete case it is done for the model based testing tool IBM® Rational® Rhapsody® TestConductor Add-On in the context of DO-178C and DO-330 resp.

Back to top

Variant Handling in Tool Chain Analysis

Authors: Andrea Osterloh (Validas AG) / Tina Heimer (Carmeq GmbH)

Abstract:

Usually tool classification is either done for each tool or for each project. While the first approach results in fixed safety guidelines that are hard to satisfy exactly in concrete projects, the later approach causes considerably effort for each project.
At Carmeq there are several project teams with similar but different tool chains that require tool classification. Also the existing tool chains are extended if new tools are integrated. Reducing the effort for tool classification requires to maintain one model that contains several tool chains and allows each project to define its specific tool chain by reusing some elements and specializing other elements through variants.
Every team can define its own tool chain within this single model by using a different variant of the tool chain. The Validas Tool Chain Analyzer supports these variants and allows the user to model tool chains with variants and different tool versions in one model.
For each tool there are different safety guidelines restricting the usage of the tool depending on the given process / variant. Using the safety manual generator the guidelines can be easily generated for each variant tool chain from the model.
In the talk we present the variant and versioning concept of the Tool Chain Analyzer and how it has been applied to represent the different tool chains at Carmeq in one model. Using this flexibility the extra effort in using qualified tool chains for safety related projects remains considerably small.

Back to top

Automotive SPICE and TÜV SÜD "Fit-for-purpose" Certificate - A Case Study

Authors: Nicole Pappler / Dr. Julian Wolf / Bernd Spanfelner / Doris Wild (TÜV SÜD)

Abstract:

One of the qualification methods for increasing the confidence in using a software tool proposed by the ISO 26262 standard is the evaluation of its development process. Therefore, the process evaluation is one part of the TÜV SÜD "Fit-for-purpose" certificate. Additionally, customers of software tool vendors often ask for a certain Automotive SPICE level for the development process of a software tool.
Our talk discusses the new TÜV SÜD approach of combining an Automotive SPICE assessment with the TÜV SÜD "Fit-for-purpose" certificate for software tools. The focus is on a case study performed in March 2013. This case study highlights the relation between the common Automotive SPICE assessment and the "Fit-for Purpose" certification procedure. Furthermore, we discuss the pros and cons for the tool vendor and describe a possible added value.

Back to top

Unified Formalism for Tool Classification

Authors: Jan Philipps / Dr. Oscar Slotosch (Validas AG)

Abstract:

In the talk we present an unified formalism for tool qualification that relates the approaches to tool classification and qualification of different safety standards, such as ISO 26262, IEC 61508, EN 51028, DO-178C and DO-330.
All these standards have first a determination of the qualification need ("classification") and, depending on the classification, a qualification of the tools. However there are some differences, e.g. ISO 26262 does not differentiate between different kinds of tools (constructive, analytic, verification), but classifies according to the likelihood of detecting potential tool errors.
This potential error detection analysis is part of tool qualification in other standards and only required for tools with qualification need.
In the talk we present a unified model for tool classification that combines the approaches. The advantage of this model is that one model and one qualification kit can satisfy the requirements of many standards. The result is that there is a reuse of the required documents, templates, test cases, test automation units, etc. for different standards and tools. Furthermore we show how this model has been implemented in the Validas Tool Chain Analyzer which allows us to automatically determine the tool classification according to different standards and create qualification kits that satisfy requirements of different standards.

Back to top

Qualification of the BMW Code Generator

Authors: Thomas Wengler / Nils Köhler (BMW AG)

Abstract:

Due to the usage of model based-development for safety critical systems the qualification of the development tool chain has become necessary. The tool chain consists of the modeling tool, the code generator and the compiler. Using a qualified tool chain avoids errors introduced by the tool chain.
Furthermore the usage of the qualified tool chains allows to review the model instead of the code. In addition the effort for analyzing the PIL tests is reduced, since there is no need to consider potential tool errors any more during review. The applied qualification method for the tool chain has been developed together with TÜV Nord and other partners starting in 2005. Since then BMW AG has qualified several tool chains for different modeling tools, code generators and compilers successfully.
During the validations several relevant errors in all tools (code generators, compilers) have been detected. To ensure that those errors do not have impact modeling guidelines and review checklists have been created. In order to reduce the manual effort for those reviews, especially for the numerical instabilities an analysis tool has been created that observes the models during simulation and generates hints for critical parts based on the simulation values. This tool allows to steer the reviews to those findings and reduce the review effort.

Back to top

Model-based Tool Classification

Author: Dr. David Seider (Validas AG)

Abstract:

Safety standards as the ISO 26262, IEC 61508, EN 50128, or DO-330 require to classify the used development tools according to their potential impact on the safety of the product. Tools that have been classified as critical have to be qualified.
In this talk we give an introduction to the classification process of the ISO 26262 and other safety standards. Furthermore, we present a model-based classification approach, where a formal model of the tool chain is specified that captures all information which is required to classify the tools. The model contains - amongst others - elements for tools, artifacts, use cases, potential errors, and mitigations (checks, restrictions) and is implemented in the Tool Chain Analyzer tool by Validas. It will be shown, how this approach allows for an automatic determination of the use-case specific confidence level and supports derivation of the necessary safety guidelines. All in all, this rigorous approach helps to optimize development tool chains and make them compliant to safety standards.

Back to top

SafeTI™ Compiler Qualification Kit latest developments and application results

Authors: Greg Miller / Tom Suchyta (Texas Instruments)

Abstract:

In July 2013, Texas Instruments released our first SafeTI™ Compiler Qualification Kit for Hercules ARM Safety Critical MCUs. The kit is part of TI's SafeTI™ design packages where the SafeTI™ Compiler Qualification Kit assists customers in their efforts to qualify their use of the TI ARM C/C++ Compiler to functional safety standards. Application of the kit has been assessed by TÜV Nord to comply with both IEC 61508 and ISO 26262. The Qualification Kit has already been successfully used by two customers for qualifying TI's ARM Compiler for safety applications with a third customer engagement already under way.
This talk will cover the latest developments for Texas Instrument's SafeTI™ Compiler Qualification Kit as well as results of customer's applications of the Kit.

Back to top

Toolchain qualification - A pragmatic approach

Author: Daniel Owens (ARM)

Abstract:

Functional safety standards are vague by design, particularly with regard to software development tools. Complying with the letter of the standard is necessary, but is it sufficient? At ARM, we advocate due diligence inspection of all available data rather than targeting the minimum requirements for tools outlined in the safety standard. The ARM Compiler Qualification Kit exports proprietary information about the toolchain that would otherwise be difficult, if not impossible for the manufacturer to produce. In this talk, I will describe the rationale behind the ARM Compiler Qualification Kit and the types of information that major OEMs and Tier 1's are finding invaluable for deeming their toolchain fit for purpose.

Back to top

Tool Qualification of AUTOSAR Tools? Methods for Qualifying Configurable Software

Authors: Dr. Rafael Zalman / Tobias Wenzel / Ashok Abbi (Infineon), Dr. Oscar Slotosch / Dr. Martin Wildmoser (Validas AG)

Abstract:

We have analyzed different usage scenarios of AUTOSAR tools (including a use case were a tool was qualified) and discuss the pros & cons. Since AUTOSAR is an automotive configuration approach the ISO 26262, part 6-C was considered.
The usage scenarios are a) use a qualified generator, b) tests all possible configurations with sufficient coverage and c) check/review the results of the generator against the generator's input. We see there two specializations b1) the tool user tests this (which requires very deep AUTOSAR knowledge) and b2) the tool user sends his configuration to the tool provider that executes the tests. We will compare the effort with the efforts required for tool qualification in the presentation and present the approach of the performed qualification of the AUTOSAR code generator.

Back to top

Configuration File Tools: Development and qualification of a tool chain for Helicopter Configuration

Authors: JM. Cadet (ATOS) / Frederic Pothon (ACG-Solutions)

Abstract:

Multiple factors influence the final configuration of the aircrafts such as helicopter that are heavily customized for each operator. These adaptations are efficiently performed through configuration files, called Parameter Data Item Files (PDI) in DO-178C/ED-12C. These files may include up to 200 000 parameters. They cannot efficiently and safely produce without a tool chain in which a great confidence can be obtained.
Atos have developed a complete tool chain, from the interface allowing an operator to select the applicable configuration to the binary PDI instance. This presentation will discuss about the stakes from Aircraft manufacturers, the architecture of the tool chain and its impact on qualification levels, and about the tool qualification processes applied in compliance with DO-330/ED-215 to qualify the tool in the final context. Focus and feedback on complementary activities performed by the tool developer and tool user will be highlighted.

Back to top

Tool Qualification Kit Development Process on the Example of Excel

Authors: Gopikrishna Chandrasekaran / Shanmugananth Murugan / Azarudeen (Bosch GmbH) / Dr. Martin Wildmoser / Robert Reitmeier (Validas AG) / Dr. Jürgen Klarmann (ETAS)

Abstract:

In the talk we present how a tool qualification kit for Excel has been developed. The focus is the development process that can be applied to any other tool as well. Since the Q-kits shall be flexible and extensible, we are using a model-based approach. In the talk we present the complete process, including the determination of tool features, potential tool errors, mitigations, cost analysis (optionally), test creation, test automation and documentation. All steps have corresponding quality assurance steps. We also present an infrastructure that allows to cooperate with tool providers and finally creates a model-based qualification kit. The Kit includes a qualification support tool to generate the necessary documents (tool classification report, tool qualification plan and tool safety manual) and test plans for a specific configuration. We demonstrate how it has been possible to create a qualification kit for selected features of Excel within 2 months.

Back to top

IBM Rational Tool Qualification Kits

Author: Karla Ducharme (IBM Rational)

Abstract:

IBM has customers in many industries governed by safety standards that require some kind of tool qualification. These industries include automotive, medical devices, pharmaceuticals, aerospace, rail, nuclear and other similar industries.
Specific to automotive, the recent introduction of functional safety standards that require tool qualification activities in addition to organizational and product development changes has increased the cost and effort to develop automotive products.
To help lower the time and costs for the tool qualification activities, IBM Rational is providing tool qualification artifacts for customers to use as inputs and artifacts as they qualify their specific tool usage. These artifacts are "generic" in the sense they have been created outside the context of a specific use/project. For every project, the customer is responsible for their own tool qualification and needs to analyze their tool usage and use these artifacts as appropriate.
To help lower the time and costs for the tool qualification activities, IBM Rational is providing tool qualification artifacts fo The types of artifacts IBM is creating include tool validation test suites, safety manuals and workflow guides, third party certification of tool development processes according to an international standards, third party tool certification according to specific standards, and templates. This session will describe the specific artifacts for each tool and updates made in 2013 and share examples of how customers have used these artifacts to support their tool qualification efforts.

Back to top

Model-based Tool Qualification

Author: Dr. Martin Wildmoser (Validas AG)

Abstract:

In some projects the tools used for SW development need to be qualified to satisfy the safety standards ISO 26262, IEC 61508, EN 50128 or DO-330. Tool validation by testing is one of the main qualification methods.
If not done manually it requires to use a test automation unit (TAU) and appropriate test cases that provide sufficient evidence for the tool being compliant with it usage dependent specification.
In this talk we introduce into the tool qualification requirements from the safety standards and refine them to requirements for test cases and the TAU. Furthermore we present a qualification model that captures all information that is required to qualify the tool for its specific use cases. For these use cases a tailored tool qualification plan including a list of tests to be executed can be computed automatically from the model.
The model contains elements for Tools, Features, Errors, Tests, Checks and Restrictions and is implemented in the Tool Chain Analyzer tool from Validas.
Also the work flow and the artifacts for developing and reviewing the test cases may be included in this model with guidance provided to the user by a qualification support tool. The model becomes the base for creating a qualification kit, that may be extended or tailored to a project that requires it.

Back to top

Qualification of a Tool Chain for FPGA Development

Authors: Dr. Giulio Corradi (Xilinx) / Sylvia Waldhausen (TÜV SÜD Rail)

Abstract:

The IEC61508 Edition 2 considers FPGA implementations either for the systematic development and component requirements. To enable the systematic development of FPGA technologies Xilinx partnered with TÜV SÜD to qualify its tool chain according to IEC61508 and ISO26262. The presentation will go through the main elements of the qualified tool chain, its verification process, the isolation design flow, design preservation and hardware co-simulation explaining how the elements are mapped into the proper T1,T2, T3 category of the IEC61508 to enable up to SIL3 designs.
The second part of the presentation will show the selected approach for the tool qualification in accordance with IEC 61508 and ISO 26262. The ISE tool suite is an integrated development environment for FPGA programming, it is formed by co-operating development tools and analysis tools.
As basis of the tool certification, Xilinx performed an exhaustive analysis of possible errors (H&R analysis) of the tools that are part of the FPGA programming tool chain.
Measures and recommendations for safety related development can be directly related to the analysis results. This yields a recommended development workflow and recommendations for the implementation of safety-related applications. (Customer Safety Documentation provided by Xilinx).
The above results were complemented by a process evaluation and an audit of the Verification / Validation activities in order to accomplish tool certification.

Back to top