1 Introduction

Conditions in the automotive electronics industry have changed dramatically over the past decade. At first, only a few ECUs were used in cars, but now some luxury cars have more than 60 ECUs installed. The added electronics improve safety, comfort and save energy. Today, more innovations rely on Electronic technology, and many functions are increasingly software-dependent.

The growth in complexity makes comprehensive and efficient testing more important than ever. The widespread use of a large number of electronic components has led to a dramatic increase in the number of potential error sources. Testing can detect and correct errors very early and minimize costs, and is essential at all stages of ECU development. Some system flaws are only exposed when the components are integrated and run under real-world and real-time conditions. This makes testing a cross-departmental and cross-vendor discipline.

The large number of electronic failures that occurred early on illustrate what can go wrong without considering the above facts and ignoring system testing. The later a problem is discovered in the development process, the more severe the impact on cost growth. Product recalls due to bug fixes in extreme cases illustrate this more clearly. While members of the automotive industry have learned these lessons and are now placing great emphasis on testing, further efficiencies can be achieved by leveraging existing resources. Although the testing cost occupies a considerable project budget, the correct function of the ECU is guaranteed. Therefore, it is very important to achieve the highest test quality and test depth using clear concepts such as using modern methods and tools to replace inappropriate automated testing steps.

2 Analysis, simulation and testing tools

The ECU network is the backbone of automotive electronics. Here, the residual bus simulation method establishes an important basis for conducting ECU testing. Most ECUs will not function meaningfully without a preliminary simulation of the ECU environment. For example, many ECUs can only function properly if they provide network management functions.

Different analysis functions, simulation components and test sequences rely on models integrated in the tool in the form of databases. They may be communication matrices in DBC format for CAN, FIBEX files for FlexRay, XML function catalogs for MOST or LDF files for LIN. Likewise, the CDD and ODX description files can be used to describe the diagnostic functions of the ECU. In addition to the basic information of the system, the test description file also contains the symbolic names of signals, messages, and diagnostic services. This simplifies the work of testers and test developers and creates an abstraction layer between tests and communication descriptions.

Any simple PC workstation running a Windows operating system can run CANoe. Using the real-time configuration system enables the creation of a more powerful test station with high real-time performance. The real-time configuration system consists of two parts (Figure 2): a dedicated PC running a real-time operating system (Windows CE) for performing residual bus simulations and actual testing; and a separate PC for the graphical user interface and evaluate. In this setup, the system can also be used as a test execution environment for component HIL testing.

[Figure 2: CANoe Real-Time running on two computers provides higher real-time performance]

3 Integration of test and development

Today’s development models require testing at multiple stages of development (Figure 3). Typically, individual testing is an independent, discrete activity, performed by specialized people using specialized tools, languages, and methods on properly equipped, dedicated workstations. Here, creating tests is usually an independent task, separate from other development activities.

[Figure 3: Testing is integral to all stages of development]

This segmented approach to work results from assigning the many different tasks in the development process to dedicated workgroups. However, if the requirements for task segmentation are too strict, then there is a high probability that the numerous connection points that exist between different development and testing tasks will not be optimally exploited. For example, only a well-coordinated component test and system test can avoid developing a large number of redundant test cases with the same content. When using compatible tools, the test cases that have been developed can be used as a basis for development in other different areas. Avoiding redundant development frees up resources that, for example, can be devoted to validation of existing test cases and their advanced development. Comprehensive test management provides a solid foundation for collaboration, and sharing the same test cases optimizes the breadth and depth of testing. Coordination also helps identify and fill testing gaps.

In addition to connecting the different testing phases, development and testing activities must also be linked and adapted to each other. Testing should be understood as an integral part of development that needs to be widely supported using appropriate methods and tools. This must be guaranteed procedurally and organizationally. Here, it is important that the testing is carried out together with the development and not only during the formal validation phase of the requirements. Ideally, testing can be performed directly at the ECU developer’s workplace with existing resources.

For this purpose, CANoe provides a runtime environment for executing tests and can be used in parallel with residual bus simulation and analysis functions. The flow is very easy to set up, especially if the developer is already using CANoe for residual bus simulation and bus communication analysis.

CANoe’s test components can be tested manually, semi-automatically and fully automatically. Developers can start with simple tests and then extend and refine them. Often, the creation of complex tests is the task of the validation department to build their tests on top of the developer’s tests.

An important basis for such testing is residual bus emulation. Ideally, such simulations are not built manually, but are automatically generated and parameterized from the system’s description database. The actual work is done by so-called modeling DLLs (such as interaction layers, network management, etc.), which are provided with the tool or by Vector as an OEM-specific modeling package. Residual bus simulation provides signals to analog nodes that can be taken directly from the test script, or stimulated or added manually.

Testing that accompanies development often omits formal execution and documentation compared to the planning, execution, and documentation activities that are systematically performed during the testing phase. However, these tests make a real contribution to improving overall quality, as they allow developers to deliver a more stable product for subsequent testing phases.

4 Maturity assessment and error analysis

During development, in order to assess the maturity of the ECU, a thorough evaluation of all performed tests is required. Consideration must be given to the quality of individual test results in terms of reliability and relevance, and more importantly, the use of appropriate tests to comprehensively cover the required characteristics. Therefore, the results of tests that are not very formally performed can also be helpful for maturity analysis. The prerequisite is that (in addition to documenting the testing process) a compatible configuration management is used. It is also necessary from the perspective of completing a quality-oriented, structured development process.

Whenever and wherever (test lab or development site) a test is executed with CANoe, a test record is generated, which is created without user or test case developer intervention. This eliminates the need for extra work when doing tests that accompany development. XML for test records is an open format that can be used by other tools for more in-depth processing of the results. For example, a test management system can be used to evaluate test records during a maturity analysis.

The essence of this work is the mapping of test results to test cases and test cases to requirements. This can be easily achieved using a unique identifier (such as a requirement ID) that test case developers refer to in individual test cases. CANoe automatically copies this identifier into the test record so that all test cases, test results and requirements can be unambiguously associated (Figure 4).

[Figure 4: Test cases and test results are explicitly referenced by ID]

Analyzing why the error actually occurred is at least as important as documenting and evaluating the test results. Most testing tools do not provide any such functionality or only provide imperfect functionality. An important reason is that error analysis is often treated as a completely separate task for the developer. First, they face the problem of understanding the bugs detected in the tests and tracking down their causes. In particular, when test labs report bugs, developers often cannot even use the systems used in the tests.

Therefore, it is mandatory to accurately record the test steps on the test bench and record every interaction with the DUT, especially the bus communication. During the analysis phase, CANoe allows to replay and analyze any required records (logging records). This makes it possible to use and benefit from the same test system at the development site as on the test bench, so developers can independently re-execute error-generating test cases. In many cases, relevant test cases can be executed even when simplification is necessary (eg to avoid selecting non-existing measurement hardware).

5 Signal abstraction and diagnostics

Abstraction is an important method commonly used when dealing with complexity in software development and system design. It can also be used to handle tests. The increased functionality in the ECU not only increases the complexity of the system, but also requires more extensive and complex testing. Choosing the right abstraction layer to compose a test not only affects the effort required to create the test case, which in turn affects the cost, but also the quality of the test case. Just like all other software components, test cases themselves may have bugs and should be checked before use. On the other hand are necessary maintenance tasks, such as adapting to changes in requirements and revising test cases.

Signal layer abstraction is a common method for testing ECU functionality, which is why actual applications in ECUs are usually based on signal abstraction (Figure 5). For example, in a CAN system, the ECU interaction layer provides the signal abstraction. This is exactly how CANoe uses the interaction layer; it parameterizes itself from the information contained in the network description, which is also used to create the ECU software. This ensures that the ECU and the test environment use the same abstraction layer for optimal coordination with each other.

At the same time, the signal abstraction also represents – at least at the protocol level – residual bus emulation. For example, it ensures that periodic signals are actually sent periodically. During the tests, the ECUs were run in a realistic environment regarding bus communication. Also, when the system communication matrix is ​​modified, it is often possible to continue using test cases that remain unchanged. For the same application, abstraction enables the reuse of test cases in similar projects.

In ECU testing, it is not just the signal interface that is important. Testing of many ECU functions only makes sense with deeper access to the ECU. Such access is provided by diagnostic and calibration interfaces, which are accessed through the ECU’s existing bus interface. There is no point in using simple message sequences to access these interfaces, because there is a prescribed protocol underneath the communication process. It is more convenient and reliable to use appropriate diagnostic and calibration abstractions.

In CANoe, a description file from the CANdela tool or an ODX description file is responsible for the parameterization of the diagnostic access layer. If there is no description file for the actual diagnostic capabilities of the ECU, the general description for KWP2000 and UDS provided by CANoe can be used. Both ECU-specific general description files or diagnostic description files facilitate easy access to the diagnostic services defined there. It is possible to obtain the same abstractions and advantages as the signal abstraction described earlier.

Via the test scripts in CANoe it is possible to access the internal variables of the ECU using the measurement and calibration protocols CCP and XCP. The measurement and calibration tool CANape handles these protocols and the required ECU description files (A2L).

CANoe controls CANape via COM interface. This achieves the same goals as the signal and diagnostic abstractions described earlier.

6 Efficient test generation

A careful study of test cases shows that many test cases can be generated from only a few looping patterns. This is even more evident in gateway ECUs: most test cases are used to check the routing of signals and messages. Ultimately, the only reason to use a large number of test cases is the large number of possible input and output data. But the same type of patterns exist in other types of ECUs as well. Abstractly speaking, this means that many functions are tested in such a way that the ECU is first brought into a special state with the appropriate stimulus, and then the entered state is checked. The cyclic pattern for this test case is: set the signal (excitation), wait for the maximum allowable response time, then check the signal in the new ECU state. Based on experience with test patterns, the user may identify several additional runtime patterns from which many test cases are generated.

These patterns represent opportunities to further optimize test case generation. In addition to providing typical test case programming, CANoe also provides users with a way to define test cases based on test patterns. Because the test steps are known and permanently integrated into the provided schema, there is no need to program schema content (Figure 6). Test case generation is reduced to defining the target behavior, including any required supplementary data, such as allowable build times.

Clearly, the provided test schemas sit on top of the signal abstraction described earlier, allowing symbolic access to signals, messages, etc. with the help of an associated database. It is also possible to use diagnostic services or I/O signals. In short: the entire CANoe test infrastructure can be shared with test patterns. If there are requirements beyond these capabilities, test case programming is still an option.

[Figure 6: Simplifies test development by abstracting the actual execution of test cases using patterns]

Automatic generation of test cases is another way to efficiently create tests with a suitable source of information. The generated test content is necessarily limited by the level of description and depth of source. However, generation provides valuable support, mainly when covering formally defined ECU basic characteristics through tests. Generating tests requires fairly low effort, which results in tests that can be run faster and potentially detect bad trends earlier.

[Figure 7: Tests can be created from completely different sources using generators]

Vector’s toolchain takes advantage of this test generator approach. Description files such as the DBC database or CANdela definitions are used as sources for the generator (Figure 7). The generator uses these files to generate test cases, which are then executed by CANoe. Because test scripts may use the entire tool infrastructure, test generators are usually designed to be very simple. For example, with a small amount of work, the generator can create appropriate test cases from a user-defined gateway description (eg database form or Excel spreadsheet). With the aforementioned test mode, only a simple conversion is required from the customer-specific data to the test mode format. Users can create such generators directly. Vector provides further support in the form of project services.

7 Summary

The only way for automotive OEMs and suppliers to respond to the growing demand for ECU testing is to efficiently create tests and automate their execution. The test tool presented in this paper provides a proven solution for implementing test tasks using signal abstraction, integrated diagnostics, calibration and I/O interfaces, test pattern concepts and test case generators. CANoe is a high-performance runtime environment for testing ECUs and networks. Use the tool to create and execute tests at an early stage with minimal effort at the developer’s workplace. CANoe’s open interface facilitates the seamless integration of comprehensive test strategies and tool-supported test management. While some users still see it as a vision, proper integration of CANoe may determine maturity today. Vector is continuously developing CANoe for use in these areas and provides users with modern and efficient test bench support.