WHITE PAPER

Virtual Test Division

Warren Ahner, CEO RightHook Inc.

How to Accelerate Development And Advance Testing Accuracy for ADAS Systems

Executive Summary

Today’s drivers face myriad distractions on the road. As vehicles evolve to provide drivers with a seamless connection between the road and their busy lives in the office or at home, the level of potential distractions and events, both inside and outside the vehicle, dramatically increases. Vehicle manufacturers and suppliers are packing vehicles with new technology designed to mitigate these distractions by integrating driver assistance systems into vehicles. Called Advanced Driver Assistance Systems (ADAS), ADAS provide invaluable assistance to drivers while driving, resulting in substantial increases in driver and road safety.

ADAS span a spectrum ranging from relatively simple assistance systems, such as anti-lock braking systems or parking assists, to far more complex systems, such as Traffic Jam Assist (TJA). The system’s functional scope ranges from relatively simple indicating functions, such as symbols on the Head-Up Display (HUD) or warning signals, to continuously acting automated functions i.e. adaptive cruise control or active lane keeping assistance, to finally much more complex emergency intervention systems, such as emergency braking or automatic avoidance systems. Some emergency systems are already legally mandated or will soon become mandatory for all new cars within the next few years. Ultimately, these highly complex driver assistance systems pave the way for autonomous driving. But until then, engineers are faced with complex development challenges in the design of increasingly complicated subsystems.

This article spotlights the challenges engineers are facing in the development of these complex, interconnected systems and illustrates the many benefits of using driving simulator technology to support and accelerate the entire development process, both for suppliers and OEMs.

The Automotive Development Challenge: Increasing Complexity

The functional scope of today’s vehicles is broader, and vastly more complex than before, given the ever-increasing interconnectedness of the vehicle’s subsystems. Until very recently, conventional cruise control consisted of a closed feedback loop between the vehicle speed sensor and the throttle. Many of today’s vehicles, however, feature new adaptive cruise control technology which automatically controls the vehicle’s acceleration and braking. By monitoring other vehicles and objects on the road, adaptive cruise control enables a safe and comfortable driving experience by helping the driver keep a steady vehicle speed at a given moment.

While the driver is enjoying a safe and comfortable driving experience, he or she is unaware that under the hood multiple interconnected systems are seamlessly interfacing with half a dozen sensors, Human Machine Interface (HMI) systems and driver monitoring systems; all allowing a safe and comfortable driving experience.

Systems that were recently considered next generation active safety are now essential features being demanded by younger, more tech-savvy customers. As the adoption of these systems increases and they become mainstream, so does the operational domain. The biggest challenge facing engineers and developers of level 2-3 autonomous driving systems is managing the driver’s expectations of what the system can do.

The way today’s ADAS functions are used by drivers presents another challenge. In the past these systems’ sole focus was on safety features (i.e.an emergency break feature). Today many ADAS features are aimed at proactively improving the driver’s comfort (i.e. lane-keeping).

These complex ADAS features and the corresponding benefits to drivers come at a heightened development cost, however. How do you adequately test these complex and interconnected subsystems, and the final vehicle, to ensure they’re safe enough and predictable enough? Especially when physical prototypes on which to conduct your tests are in short supply or even absent? Some of these systems will be used multiple times per car trip (such as lane-keeping) while others are hopefully never or rarely employed (emergency braking) Regardless of the number of times used, these systems must all be adequately tested.

In addition to managing the system’s complexity, automakers also need to manage diverse development timelines for the vehicle’s components; timelines that are often dictated by their suppliers. The ability to simulate all of these various systems concurrently while having a meaningful test case becomes increasingly hard to do using only conventional tools.

ADAS systems are a joint effort between OEM and supplier.

Most ADAS systems involve intellectual property from a number of technology partners, since they’re too complex to be developed solely by a single supplier. Adding to this complexity is the requirement that every component in a safety-certified ADAS system must be certified, from the underlying hardware to the operating system, middleware, algorithms and application code. The application code also needs to be certified to the appropriate automotive safety integrity level. This is a time-consuming and expensive process.

Testing and validation is arguably the most difficult and time-consuming aspect of ADAS development. And yet it is also the most critical aspect in terms of giving OEMs and Suppliers peace of mind and re-assuring them that their sub systems and final vehicle are safe and reliable. The ultimate goal is to achieve 100% accuracy with no false alarms and in all possible scenarios and conditions: traffic, weather, number of obstacles or pedestrians in the scenario, etc.

It’s critical that the development team’s test database include all possible test cases in order to test for all possible conditions and scenarios. Unfortunately, the reality is that this isn’t remotely possible. This is why suppliers spend years testing and validating their systems and conducting extensive field tests in different geographic and climate regions before bringing their components to market.

The costs associated with these years of testing and system validation are enormous. Vehicle and component manufacturers thus actively seek opportunities to reduce costs and accelerate development time, yet without compromising the accuracy associated with extensive testing. Doing so will result in getting a vehicle to market faster and ahead of the competition, and hence is critical for ensuring a competitive market advantage.

HIL, SIL, MIL and DIL

The vehicle manufacturer is ultimately responsible for not only integrating the various driver assistance subsystems delivered by suppliers, but also for testing and validating the safe operation of the subsystems within the full vehicle. The challenge facing the vehicle manufacturer is how to synchronize overall vehicle decision making when the subsystems’ suppliers have different decision-making and development timelines, ranging from timelines for software models to first hardware prototypes.

Engineers typically spend significant time on offline simulation efforts in order to develop the algorithms and ensure that all sensors are working properly under the myriad possible scenarios and conditions. This can result in hundreds of simulations being run on a daily basis. Increasingly, developers are turning to virtual simulation and driving simulators as a way to accelerate the development and the testing of components and full vehicles. Engineers begin using a driving simulator as soon as they obtain good results from the offline simulation.

Using a driving simulator facilitates the analysis process on a subset of situations, mainly two different types: one is a situation involving the highly dynamic behavior of the car, i.e. obstacle avoidance or an emergency braking event where the engineer needs to understand how the car behaves at the limit of its dynamic behavior. The other situation is one where the driver’s comfort is important. Defining and evaluating comfort is a much more subjective matter, since comfort levels vary to a person. This subjective situation involves the driver’s comfort level based on their reaction to the simulator’s driving algorithm. Let’s consider for example a lane-keeping algorithm intended to keep the car in one lane. If this algorithm were to continuously move and thus affect the car’s steering, the result could be the driver experiencing motion sickness. This would in turn make them feel unsafe being driven by a robot algorithm.

The ability to concurrently analyze multiple situations and scenarios is leading more and more OEMs to use driving simulators at the point where all subsystems are integrated and tested together, regardless of whether it’s via software in the loop (SIL), model in the loop (MIL) or hardware in the loop (HIL). As the OEM progresses in the development process, more and more subsystem software models can be replaced by hardware solutions – thus pushing the driving simulator system more and more closely to resemble reality.

The evolution of computing technology in the vehicle development process is leading to a significant shift among automotive companies, one where today’s automotive companies are evolving into both hardware and software companies. This has led to a huge uptake in the use of software in the loop systems. As this technology gets closer to being able to run on everyday computers, we will continue to see the adoption of SIL. HIL and MIL always have been and will continue to be part of the automotive development process “V”.

The importance of standardized test scenarios and regulations in ADAS development

Unfortunately, offline simulation doesn’t allow for testing the behavior of an autonomous algorithm e.g. when the driver is attempting to regain control of the vehicle after an event. The only way to truly evaluate how a human being will respond in a given situation is to actually use a human being in the desired situation, and then evaluate their reaction. Unfortunately, there are infinite possible test scenarios, which means there is no end to the number of potential tests. This neatly summarizes one of the biggest challenges (the billion-mile problem ) facing ADAS and autonomous vehicle engineers : the inability to actually predict every potential situation a driver could encounter when actually driving a car.

This illustrates the importance of agreeing on acceptable test boundaries in the absence of any global test standards. Far too many interest groups - national governments, state governments, insurance institutes, academic think tanks, various automotive engineering groups and industry consortiums - are competing for a sliver of influence on the standards, all with their own motivations and drivers. As an industry, we are a long way away from standardization.

The role of the driver in ADAS

Driving simulators are much more than just an integrated platform facilitating basic functionality testing of assistance subsystems. Driving simulators are sophisticated, powerful development tools that give engineers the ability to accurately test that one interface with the biggest uncertainty factor in the entire system – the driver themself. A driving simulator lets engineers take into account unforeseeable human reactions to specific events while simultaneously guaranteeing safe vehicle operation. It also allows engineers to assess and optimize the end user’s perceptions while being supported by the available ADAS functionality in the car.

As autonomous driving gets ever closer to reality, using a real driver in a driving simulator will become an ideal way to test the driver monitoring systems which will become increasingly used.

Today there are two types of driver monitoring systems, and they are definitely not created equally. The first system to come to market was active wheel feedback. This system sampled feedback from the steering wheel position sensor and sometimes a torque sensor (or part of the power steering system) to evaluate if the wheel was being held by a user. We’ve all seen YouTube videos showing how easily these systems can be defeated; a water bottle and some tape is all that is required.

The second is an eye tracking system, which by todays’ technology standards is the gold standard. This system uses infrared cameras to evaluate the attentiveness of the driver and provide feedback to the system when there is a problem. This system not only evaluates if a user is in the seat and awake, but it can alert the system when the user is looking away from the road and at their phone.

Studies on driving simulators have been done showing the devastating impact distracted driving can have while using ADAS systems. Simple glances at a phone to read notices “in 350 feet, turn left” or a text message asking “Text OK if you are going to be arriving withing 15 minutes for your 6:00 PM reservation for 2” can turn into a deadly crash. When drivers are distracted and taken out of the immediate driving experience, they tend to ease up on driving movements. This can result in them losing their grip on the steering wheel, and even sitting differently which can delay their reaction time to using the controls. This all makes driver monitoring even more imperative until we have a perfect driverless systems. Until we have a truly Level 4 system that can exist without a steering wheel and pedals, we will need the capability to increasingly better monitor driver’s reactions with clear and concise user interfaces.

How will the technology enter the market?

Currently the most advanced ADAS system a consumer can purchase is in a luxury vehicle, and at best that’s only a level 2/3 system. Currently there’s momentum behind this technology becoming acceptable into everyday mid-market cars. A few companies offer option packages with compelling lane keeping and adaptive cruise control systems for less than the price of selecting the fancy stereo upgrade. Eventually every car will have this hardware as future safety systems become mandated and are built on many of the same sensors. In the near future, ADAS will simply become a software option for purchase.

How can OEMs benefit from the use of driving simulators?

OEMs are in a difficult situation. On the one hand, they have to add more and more software engineering skills to their traditional mechanical engineering skill set. Yet on the other hand, they’re increasingly reliant on collaborating with suppliers on different driver assistance subsystems. The various components, sensors and algorithms all need to be integrated into a safe environment. The ultimate desired state is for these subsystems to end up in a car that meets or exceeds the customer’s expectations, and at a competitive price point. The way to accomplish this it by putting a driving simulator at the? front and in the? center in the development activities. This promotes and enables collaboration among the engineering functions and between the OEM and supplier, and facilitates testing at early development stages before physical prototypes are even available. Finally, driving simulators provide the ultimate environment for including the driver and assessing the driver’s perception of the entire system and its features.

It’s currently not feasible to consider all the potential real world scenarios in testing. That’s why driving simulators are the ideal tool for setting up edge cases and performing extensive testing even before any hardware is built. Changing environmental conditions (weather, light, obstacles, …) is simply a mouse click away; changing tires might take 30 seconds. While a driving simulator cannot address all real world test scenarios; it can certainly address far more than were traditionally possible. With this powerful development tool, automotive companies can develop their test strategy according to their intended use cases and upcoming regulations.

Driving simulators are the stepping stone to the final frontier: virtual sign off where all testing is done in the laboratory with a combination of offline and online simulation and without the use of a physical prototype. This is currently not reality, but thanks to the power of the driving simulator, this goal is within reach.