ODD: The Main Challenge of Full Self Driving and Autonomous Vehicle Testing

After $100B invested and no viable products in mainstream use, the autonomous vehicle (AV) sector has been a source of fascination for some, concern for others. 

But the main reason autonomous vehicles have stalled out has nothing to do with funding or resources. Rather, most AVs lack the engineering processes needed to bring them to mainstream utility. 

Waymo, SpaceX, Zoox, and other engineering firms have vehicles that look and feel great. But the rubber meets the road (literally) when it comes to the near-infinite scenarios that a vehicle could potentially encounter. 

In mass-market use cases, then the operational design domain (ODD) needed to test and validate the safety of autonomous vehicles is abnormally high. This is where most companies run into problems: they simply don’t have the resources to adequately test their vehicles. 

So is that it? Does this spell the end of the autonomous dream? Not necessarily.

Let’s take a look at why autonomous ODD has become so complicated, and what AV engineers should do to combat it. 

Why operational design domain (ODD) has become so critical to self driving car safety

The vast majority of engineering projects, including AVs, prioritize physical testing. This is exactly what it sounds like: engineers construct a prototype and subject it to a variety of scenarios at a physical testing site.

This test data is the primary decision making tool when it comes to further design iterations, refinements, marketability, software, and more. 

For traditional vehicles, where a human driver will always be present, this method is perfectly reasonable. When a vehicle encounters an unfamiliar scenario, the human driver can (and hopefully will) make the necessary changes and adjustments to maintain operation and safety.

However, for all the advancements in AI and machine learning, autonomous decision making is nowhere near as sophisticated as the human brain. If AVs are going to be prepared for any and all scenarios, they need to have referential data—and the more, the better. 

Beyond this, machine learning is very data hungry and synthetic data hasn’t matched real-world data quality. This has forced machine learning teams to mandate large percentages of data from the real world. 

The result: an ODD that is larger than any physical testing site can account for. 

And here’s the ultimate problem: autonomous vehicles have very little margin for error. The backlash from Uber’s famous 2018 autonomous vehicle accident—that resulted in a fatality - demonstrates this fact. 

Consumer demand and legal liabilities mean that autonomous vehicles have to get as close to “perfect” to be viable. Obviously, perfection is impossible. 

Instead, most engineers ask themselves: how close to zero can we get? For most, the answer is: “better than a human driver.”

But human drivers are exposed to countless situations over the course of their lives that enable them to react to novel scenarios. What’s more, human beings have the ability to adapt to new challenges in a way that AI and machine learning has yet to fully emulate.

As a result, testing and accounting for every scenario to the point where AVs can model human behavior has resulted in a massive ODD. We’re not exaggerating when we say that this requires billions of autonomous vehicle test cases to become as good or better than a human driver. 

So while most engineering firms have simply “copied and pasted” the physical testing model to self driving car tests, it just hasn’t been sufficient to develop a viable product.

Challenges in scaling physical prototyping for autonomous vehicle validation

While physical prototyping works for traditional vehicles, it’s not a workable solution for AVs. More specifically, physical testing is not scalable to the massive number of scenarios an AV may encounter throughout its lifetime.

Here are some of the reasons why AV companies fail when they try to scale physical prototype testing. 

High overhead costs

When looking at resources, both physical and labor, scaling physical prototyping requires serious investment of resources in each successive phase:

• Manufacturing hardware and assembling new prototypes
• Identifying test locations 
• Filing paperwork with governing bodies
• Creating & executing test plans safely

Each of these steps comes with its own set of costs. Also, keep in mind that these costs add up not for each individual testing site, but also for each individual prototype that must be newly constructed. 

Leaders in autonomous development (like Waymo) have even built mini-cities to fully control the operating conditions for autonomous testing. This allows them to avoid waiting for a particular scenario to occur in an actual city driving experience and enables the testing of any scenario at any time.

To account for the full extent of ODD requires more investment than most AV companies are able and willing to make. 

Physical limitations

In addition to cost, physical testing can only provide you a limited amount of useful data: 

• Only one prototype can be tested at a time—which slows the creation and collection of key data
• Making minor changes to prototypes requires major interventions late in the process
• Problems with the prototype only arise after you’ve invested significant resources in manufacturing and assembling that model

Despite millions of miles of test data, many leading AV manufacturers are years, perhaps decades, away from mainstream utility. That’s because for all the investment required for physical testing, the ROI - return on information - is too low. 

Errors are costly

Although this falls under the header of physical limitations and overhead costs, it’s important enough that we want to call it out on its own.

Simply put, with physical testing, errors are costly.

Even assuming adequate autonomous vehicle safety protocols, even small errors can result in minimal to significant damage to the vehicle. Not only does it cost money to repair the prototype, but there’s also the opportunity cost. If you’re spending time repairing a prototype, that’s time that it’s not out collecting test data. 

Then when you spot the problem and concoct a solution, you then have to manufacture a new prototype and re-test it. The cycle continues for as many iterations as necessary to fix the problem. With AVs, that’s a long and tedious road. 

Why most solutions to the ODD problem fall short in autonomous software testing

Many leading AV manufacturers are well aware of the ODD problem. There have been a number of solutions proposed, and some have achieved a limited measure of utility. 

However, it’s important to keep in mind that none of these solutions have actually solved the root problem: bringing AVs to mainstream utility. Here’s why each one of them has failed. 

Reducing ODD

Scaling real-world testing to meet a large ODD is cost prohibitive. Think about it. Physical testing sites have to be built, which require resources and labor to make happen. Given the number of potential scenarios a vehicle could encounter and the corresponding number of test cases that would need to be constructed - the ballooning costs are obvious.

As a result, many companies are responding by scaling back their autonomous ambitions, focusing on off-road use cases like farm vehicles, airport support trucks, and marine vessels. Unfortunately, this approach doesn’t actually solve the problem - it merely avoids it. 

Testing “cities”

On the other side of the spectrum, companies like Waymo have constructed massive testing “cities.” These are essentially diverse testing sites with enough various features to increase the scenarios a vehicle is exposed to. 

The most obvious problem: these are enormously expensive, and only the most well-backed companies have a hope and prayer of adopting this model. 

What’s more, a single testing city still limited AVs to the particular climate, terrain, and other geographic factors within the region. As a result, the AV’s inputs are still limited, just less so than traditional testing sites.  

Crowdsourced test data

One of the more innovative solutions is to stream data from vehicles purchased by consumers to assess performance and push algorithm updates. While effective at gathering data from diverse scenarios, it has the consequence of pushing testing costs onto consumers. 

Underuse of simulations

As we’ll discuss below, simulations are key to solving the ODD problem. Of course, simulations are nothing new to AV engineers. The problem is that, in most cases, engineers don’t use them to their fullest potential.

Major AV companies talk about the billions of simulated miles they’ve executed. While true, most of their processes involve simulations taking a second-seat to real-world testing. They’re never the source of truth used to make engineering and business decisions. 

Currently, most engineering processes reward physical testing. As a result, until a serious culture change happens in the engineering sector, prototype AVs will continue to be the more trusted and mandated tool.

Model-based design: an agile alternative to reducing ODD and improving the safety of self driving cars

Regardless of the specific approach, the fundamental problem is that companies are trying to tackle an engineering process problem with a business process solution - essentially apples and oranges. 

At the end of the day, all this does is kick the can down the road. If AVs are ever going to reach mainstream utility, then companies need to adopt a completely different approach. 

In the half-decade since industry experts originally identified this problem, none have succeeded with a business process solution. The most success we’ve seen has been with existent companies through service contracts with current customers. 

Bottom line: it’s time for an alternative solution.

Rather than reduce the ODD, like many companies have tried and failed at doing, businesses need to change their engineering processes to meet the expanding needs of ODD. This means moving away from physical testing to leveraging new developments in AI, machine learning, digital modeling, virtual simulations and most importantly, a new approach: model-based design. 

In model-based design, engineering focuses on developing virtual models of products, rather than fielded prototypes. Because these models can be created and changed at no cost, it’s easier and more agile to make adjustments based on test data. In fact, it is as simple as a click of a button. 

Scaling a model involves some of the same resource commitments as physical testing. However, it provides viable alternatives to the challenges we mentioned above:

• By investing in virtual models rather than physical testing, costs of manufacturing and assembling physical prototypes are eliminated
• Without the limitations of physical testing, virtual simulations can run countless more scenarios at a faster pace. This means thousands of hours of prototype operation can be accomplished in minutes
• Errors in simulated testing result in no physical damage, and changes to the model can happen with a click of the button
• Problems that arise can be solved and retested before a single physical component is manufactured

At the end of the day, the biggest benefit of model-based design is that it turns development of self-driving cars into an agile process. Like in software engineering, agile AV development and testing can result in faster design, testing, and shipping. 

Rather than reduce the ODD, model-based design enables companies to meet the demands of AV testing without spending exorbitant resources. 

Final thoughts: It’s time to solve the challenges in autonomous vehicle testing and validation

As we mentioned earlier, the ODD problem won’t go away until there’s a culture change among AV engineers. Autonomy is a fundamentally new technology, and will require new, innovative solutions if it’s ever going to reach mainstream utility.

But there’s more to the story. Because of cost prohibitions, only the largest and most funded engineering companies are going to have the resources required to scale traditional testing and validation processes. However, innovation happens on the margins. It’s the up and coming engineers that are going to break the mold and reshape how we think about autonomy, vehicles, and transportation in general. 

Model based design is a solution that can help empower engineering firms of all sizes to rise to the occasion. At Collimator, we're building a platform to solve the AV development challenges. Speak to an engineer today to find out more.