Leading through testing: Why “automated” does not happen automatically

Test automation is basically a process of using specialized software to perform tests on a system, without human intervention. This is particularly important in the automotive industry, as it involves testing and ensuring the function and quality of vehicle components and systems. Typical test objects are, for example, communication between control units, tangible customer functions in the car such as various displays, air-conditioning systems, driving dynamics functions, or software for components in the vehicle, to name just a few.

Especially the standardization of processes is of high relevance for the further development and, if necessary, the introduction of test automation. Only through this standardization is it possible to have tests performed automatically, which on the one hand leads to a considerable efficiency and also quality increase of the test processes, but on the other hand also saves time and thus costs: Once test steps have been defined, it is possible to perform them again at any time in exactly the same way. This has another advantage, namely the high potential scalability of the test situation for test engineers and their departments, thus for the entire company. For example, a very large number of tests can be automated, sometimes in parallel, they can be virtualized – more on virtual reality, Metaverse and co. later – and the resulting data can be massively bundled and centralized.

Above all, test automation opens up another invaluable potential in addition to its actual field of application: Engineers and developers can concentrate on complex, value-adding tasks instead of performing tests manually.

However, the evolution from manual to test automation does not happen by itself or overnight: For companies, it always starts with defining their own processes so that they can be standardized and automated. Companies like Cognizant Mobility help with such steps, but the impetus must always come from within the company. To show the advantages for decision support is the declared task of this article, which inevitably now also goes into medias res to go deeper into the matter.

From wrench to software: test automation in the change of time!

As long as there has been industrial vehicle manufacturing, there has been testing for all it’s worth, and optimizing and ideally automating processes is an essential trait of humanity per se. Even the wooden wedge on the gas pedal in the workshop is a form of automation that is now less artisanal and more industrialized and ideally digitized.

Even today, a vehicle door has to withstand being opened and closed 10,000 times by a robot to test whether the door handle can withstand this long-term stress. An automatic tailgate must close correctly thousands of times and reliably seal the passenger compartment. Above all, the software that enables and controls countless processes in the vehicle needs to be tested accordingly, in terms of both quality and quantity. Like hardware tests, these software tests can be automated, often with specific variants of automated testing such as xRPA. Information on this can be found here, for example, but we will go into this later.

Hardware tests are also test processes that can be partially automated, and the testing of materials used in vehicles is also essential, both for materials for the bodywork, for example, and for windshields onto which a head-up display is to be projected: Is it in the right place, easy to read no matter what the light, can the customer adjust the size, and does it reflect the correct information? How materials can be found by means of A.I., even long before testing, can be read in this extremely readable article about the startup “Exomatter”.

Nowadays, it is mainly specific suppliers, but also large OEMs, that are active in the field of test automation of software-based processes. The number and volume of software used in vehicles has grown tremendously over the past two decades, with great momentum in recent years, bringing with it massive requirements. After all, software must be tested not only for function, but also for safety and communication, as well as for interfunctionality and user-friendliness, and must comply with various regulations(such as UNECE R155).

Companies such as Cognizant Mobility are mainly concerned with system and customer functional testing in this context. An example of this is the communication between two ECUs (Electric Control Unit): The communicated signals are transported via BUS and checked whether the content and address data are correct and whether the communication is secured. This wants to be tested, as do customer functions that can be experienced (as opposed to ECU-relevant functions), such as the electric opening of doors, windows or tailgates, or the switching of driving modes (e.g. from “Eco” to “Sport”), as well as climate functions, for example: Are the signals sent correctly via the ECUs, with the correct parameters, and does this work as often as desired – even in special cases?

Testing thus covers an enormous range of points to be verified during vehicle development, and automation of the test processes is particularly useful for processes that have to be repeated constantly or carried out in large numbers. The effort involved can vary: In the case of software for a liftgate module, it is often only tested whether the liftgate opens and the display in the vehicle is correct for this (it goes without saying that extensive function and material tests have already been carried out as part of vehicle development).

However, if the update is a central gateway or software with a large and new scope, the testing must be much more extensive, since many different components and functions in the vehicle interact with it. So everything that has been changed or is affected must be tested. The definition of the test scope is therefore also a relevant aspect of test automation. Once this definition is in place and the scope of the test scenario has been determined, a process can be derived from this and each new software generation can be tested to the same extent. For example, the basic functionalities can be defined, which are tested immediately with each update, and in addition, further processes are determined and automated, which are newly added.

What is relevant here, of course, as attentive readers have surely already noticed, is that these processes, in order to be automated, should ideally always run in the same way. But what if testing scenarios need to be adaptable? When rare edge cases need to be tested or completely new routines occur? Then xRPA can be a solution.

xRPA and test automation: From highways and data highways

Whoever says “xRPA” must first say “RPA” and summarizes casually: Classic Robotic Process Automation plays a decisive role in test automation. It is a technology that enables the automation of business processes (even outside of traditional automotive testing) by mimicking human actions within software applications. In the context of automated testing, RPA can also be used to automate test cases that would otherwise have to be executed manually. This increases the efficiency of the testing process and allows more tests to be performed in less time.

xRPA (Extended Robot Process Automation), on the other hand, is a subarea of test automation and offers two elementary advantages over it. The primary task (and possibility) of xRPA is, on the one hand, to connect individual test systems that usually do not (cannot) necessarily communicate with each other and to transfer specifications from one system to the other. Thus, test tasks can be chained, connected in series or parallel, which also creates space to alternate tests by means of switching on modern A.I. systems.

Another advantage is that – as is the case with some OEMs in the development phase – test cases can be developed and parameterized accordingly. For this purpose, a test framework is developed in which tests can be repeated several times, always with slight modifications. There are other options for this, but the fact that xRPA can provide not only automation but also alternation should be mentioned here.

Why interfunctionality and communication between different systems and devices is extremely important in the Internet of Things and Industry 4.0, and why efficient testing of complex system landscapes can mean not only cost savings but also rapid time-to-market, as well as the existential advantages for companies associated with this, probably need no further explanation.

Test automation at the limit: Because automotive can (and must) also be fun.

Also in the experience portfolio of suppliers such as Cognizant Mobility is testing at the limits, which takes place on the test bench or in the vehicle itself rather than on the test bench. For this purpose, the vehicle is moved in the limit range, for example under high engine speeds or at high speed: usually a testing area that can only be automated to a limited extent and, what is more, is not suitable for every test engineer. It is often less a matter of hard facts than of feeling when a vehicle is to be tested in maximum drift on the test track to see whether it will drift at all with or without DST, or at what setting level. Here, the feeling of the highly trained test drivers and customer expectations play a major role in the test scenario: A private user certainly applies different standards to even a very sporty vehicle than a professional race driver. Incidentally, the test drivers sometimes cover enormous distances with test vehicles – entire fleets collect data worldwide that needs to be transmitted and evaluated. So this is also a clear use case for automated and also virtual testing, we’ll talk about that in a moment.

Of course, a wide variety of test scenarios can also be automated in the limit ranges. A car on the test bench with the gas pedal fully depressed can be tested, for example, to see whether all control units receive the correct signal when the brake is suddenly applied: Are the signals sent correctly and also executed time-critically?

Testing this, also and especially in extreme situations from drift to hazard braking, so that the vehicle of the relaxed vacationer on the highway receives a warning signal and the emergency brake assistant kicks in, is also an explicit task of automated testing and thus creates safety and trust.

Advantages of test automation: And we thought it was obvious

Some of the most compelling benefits of automated testing and especially Extended Robot Process Automation are now well known to us: They are efficient, can be performed in large numbers and quickly, are scalable, of high quality, can even be alternated in some cases, and give valuable skilled workers the opportunity to perform other and more complex tasks in the company. This almost – pun intended – automatically leads to faster overall development for companies, which can make earlier time to market possible. Costs can be saved and efficiency can be noticeably increased. Also, test data based on automated testing is more robust and reliable, thus increasing the quality of the final product.

Ideally, partners who are commissioned with the automated testing of various systems and functions know the entire testing process and can support it. Part of the philosophy of Cognizant Mobility’s testing department, for example, is to know exactly the status quo: Determining the scope of the test scenario as well as the creation of test cases and the administration of the entire test management are familiar to Cognizant Mobility as a partner for decades (including the BMW Group), as is the entire process from software delivery and its launch to reporting for the customer. The best tests are of no use if the data supplied cannot be bundled and evaluated efficiently. Before testing is after testing! In this way, customized individual solutions can be developed, especially in the area of xRPA, even in the context of large legacy or enterprise solutions.

Anyone familiar with the project landscape also knows anyway that there are no one-fits-all solutions, not even for supposedly uniform standard processes: The path from “let’s just do it this way, it works” to effective test automation is always also an individual path, in which tools are developed to determine productivity and the degree of scaling, to communicate with systems, in short: to really deliver to customers exactly what is needed without unnecessary effort. Being able to draw on a pool of experience, solutions and also concrete software and testing solutions is an invaluable advantage for developers and suppliers (which, phrases aside, can usually be interpreted and appreciated very well and very concretely).

Test automation: The test tracks of the future

It remains to wonder where the journey is headed. Put in flowery terms, but no less true, and clear not only to insiders: the path of testing is moving away from the test track. Test benches, elaborately created test tracks that have to be painstakingly adjusted for the smallest changes, “feeling things” for the test driver: relics, often already today. Of course: Some test benches will continue to play central roles in safeguarding, as customer demands are too diverse to really digitize all of them completely, but: Virtualization cannot be stopped – we refer (as reading after this post, of course) to our interview with Stefan Wenz from Epic Games and his very interesting remarks about autonomous testing of driving functions via Unreal Engine..

To create a test track once virtually, to equip it with sensors and to test now no longer in real time, but simulated, millions of times, in every desired change (different road surface? ground temperature? rain? sun? inclination?), in parallel, supported by A.I., to test previously almost unthinkable rare edge cases despite all experience? This not only sounds good, it is already the truth today. Entire urban areas can be scanned and digitized using modern means such as A.I. and machine learning ,and then revitalized and populated as desired.

The almost playful approach that is possible here also allows end customer testing at very early stages. It is conceivable, for example, to provide a Digital Twin, a digital clone of the vehicle in an actual game, which players can drive – and thus test. Thousands of times, with successful games like Fortnite even millions of times (see, once again the reference, the above linked and insightful interview with Epic Games).

In keeping with this, new basic ideas are appearing on the digital market such as the Metaverse, in which the customer can also be virtualized. Entering the Digital Sibling as an avatar while the real user wears VR goggles or enters a corresponding partial model and explores the car virtually, almost as if it were real – except for the sense of touch? This is not a vision, it is reality, which is already widely used in development and in the area of private applications, keyword augmented reality from Pokemon Go to the virtual couch in the digitally captured living room. With the advent of more and more high-tech solutions in the areas of artificial intelligence, data science and automotive, scenarios that may, should, must be tested are almost bigger than life, almost immeasurably large.

Testing must therefore be automated, imperative, if only to be able to meet the endless demands of an increasingly virtualizing and digitizing world, the rapid development in the automotive market and Industry 4.0. It is only natural to network systems and functions (keyword “Connected Car”), to test and secure communication, and instead of elaborate test stations with underworked test engineers, to decentralize: remotely, from the home office, start the test computer and go through a few million test scenarios, process the data in Big Data pipelines and evaluate it professionally? It will never be a piece of cake, but it will also no longer entail months of effort, but will basically happen at the push of a button.

Of course, the all-clear: We still need test engineers. The systems themselves, as well as their test environments, still want to be developed, built and maintained, physically and virtually. And no sooner has one test scenario been created than the next one follows, because tasks are becoming increasingly complex. They are being automated, optimized and further developed, but the number of tasks is not decreasing; rather, the possible uses of the tasks are becoming broader. A movement from one to the other, from the wrench to the test stand, and now from the test stand to the remote test computer. A challenge for a whole industry, a whole world, which every generation experiences anew and which does not cease to be characterized by tremendous diversity and tension, even in the present day. So many new possibilities, so many scenarios, so many test cases that accompany the further path of test automation – this allows only one conclusion: It is in the very DNA of test automation to accompany and enable the change that ushers in the next generation of mobility. And the most exciting thing about it, at least from some perspectives, is the inevitable evolution of the testing landscape that follows, participating in the most current and essential developments.

Exciting.