Automotive industry needs to progress on several fronts together which include functional safety, cybersecurity, ECU communication and overall innovation in automotive software development. If there is one aspect that’s equally important in achieving growth in all these fronts, it is software V&V (verification and validation) . Software V&V (verification and validation is also known as software qualification test or functional test.
Rigorous and iterative software validation ensures early identification of bugs and their resolution. But wait! Is identifying the bugs at an early stage enough?….
How do we make sure that we do not leave anything to chance in our pursuit of making innovative automotive software? ….
The key is to follow the best-practices identified by the automotive experts and that’s where ASPICE takes the lead.
ASPICE stands for Automotive Software Process Improvement and Capability Determination. As is clear from its name, ASPICE has two dimensions to itself:
improvement of the software development process, and
– assessment of capability of software developers by the OEMs.
Unlike ISO 26262 which provides concrete set of guidelines for development of an automotive system including test methods, ASPICE is more of a set of best practices. However, both follow the V-model of software development and can be implemented together as well.
Coming back to ASPICE, it is important to understand that ASPICE focusses on software processes. These processes are defined in the Process Reference Model (PRM) of ASPICE. In conjunction with indicators from Process Assessment Model (PAM), these processes are assessed. In this blog, we take a look at on one such process- Software Qualification.
“Process Reference Model is a set of all the processes that needs to be applied during automotive software development.
Process Assessment Model is a way of evaluating the organization’s capability to run mature automotive software development processes. It maps process performance to capability levels with the help of certain indicators.”
Various Tenets of ASPICE Compliant Software Qualification
Different processes defined in the process reference model are divided into categories and groups. This categorization is based on the unique functional objectives of these processes. For instance, software qualification process comes under Software Engineering Process Group.
Software qualification test is performed against the software requirements. The purpose is to make sure that the integrated software fulfills all the requirements enlisted during the software requirement analysis. Software qualification is a common step in v-model development. So what’s different in ASPICE?
The answer to this question lies in the base practices to be followed during the process, process outcomes and the work products derived as the output.
“Base practices are a set of typical activities performed in a process. For instance, the first base practice for software qualification is creating a test strategy. Implementing the base practices for each process is the first step towards ASPICE compliance for an organization.”
Let’s discuss each process outcomes in the software qualification test process and the base practices associated with it.
- Test Strategy: The first outcome of a successful implementation of this process is development of test strategy. This strategy must have following attributes:
- It must be consistent with the project plan as well as the release plan*
- Strategy must include a regression test strategy in order to re-test the integrated software in an instance of code change.
In line with this process outcome, the base practice is to create such a test strategy with all the prescribed attributes.
*Release plan comprises of functionality to be included in each release, associated elements like software/hardware and mapping of requirement to each release.
- Specification for Software Qualification Test: The test specification is a crucial document as it provides evidence for compliance of software with the requirements. It is developed based on verification criteria according to the test strategy and includes the specification for development of test cases as well. When the tests are executed, the test cases are selected from this specification.
- Test Case Selection: Based on the test strategy and the release plan, the test cases are chosen from the specification. The test engineers select the test cases in a way to ensure sufficient coverage as per the strategy and release plan.
- Test Execution: The integrated software is tested using the selected test cases. As the test cases have been created as per the software requirements, the software is validated against them.
- Ensure Bi-direction Traceability & Consistency: Bi-direction traceability is to build a software that is bug-free and easy to maintain. When you can trace back a piece of code to the requirement and vice-versa, it gets easy to make changes, add/remove functionality and so on. One of the process outcomes is to ensure that bi-directional traceability and the resulting consistency is maintained.
- Test Result Summarization & Communication: At the end of it, the test result is summarized and communicated to all stakeholders who are affected by it. They can be the customers who might have to look into some of the requirements or the developers who need to make bug fixes. Every stakeholder needs to look at the test report summary and understand the consequences. At every iteration, regression testing will also be performed and made a part of test result.
Necessary Details on Developing ASPICE Suggested Test Strategy
ASPICE mentions in its process assessment model (PAM) that a test strategy must be developed which is in sync with the project and release plan. This strategy can be seen as a gospel of sorts for the testing team. From testing approach and automation to deliverables, this strategy gives a clear-cut guideline for the testing team to follow.
Let’s look at the various contents of this strategy:
- Risk Management: A lot of hindrances come in the way of successful completion of the deliverables. Every such hindrance is considered a risk item. Any issue or condition that adversely impacts V&V goals including requirement coverage, test coverage or effectiveness of tests is a risk item. Risk items can be an incorrect test environment, or an incorrect test objective and various others. During the risk management exercise, all such risk items are identified and reported to the project manager. A collective decision is then taken on these items.
- Test and Verification Approach: Test and verification approach defines the testing processes involved, types of testing, automation tools if required and a few more details depending on the test environment. Software qualification test or software validation comes under black box testing approach.
Under this approach strategies like equivalence class partition and boundary value analysis are implemented. In the context of automotive software testing, conditions such as under/over voltage and current, timed event testing and CAN parameters may be tested. Whether automation tools would be used during the course of software validation testing is also considered when test approach is defined. Being clear about the approach is a tested way of complying with functional requirements and ensuring freedom from redundancy.
Apart from these important test strategies, few aspects like V&V schedule, configuration management and test review are integral part of test strategy.
Validation schedule is all about prioritizing important task which is done together by project and test managers.
For configuration management, usually a configuration management tool is used for maintaining all the test artefacts which would later be used as ASPICE compliance evidence. Configuration management tool is also used for test review log. Test review comprises both test case review and test coverage review. There is a traceability matrix that need to be updated by the test team and must be peer-reviewed for coverage review.
Conclusion
ASPICE is not mandated across the globe. But the OEMs understand its prowess and often assess the capabilities of their suppliers based on ASPICE levels. Software qualification is one process that matters the most while assessing a supplier’s capability. While higher levels of ASPICE are still aspirational and often restricted to very large OEMs, however, Level 2&3 are steadily being adopted by medium and small OEMs as well.
