Verification and Validation

"Verification: Are we building the product right?

Validation: Are we building the right product?"

Barry Boehm

Not so long ago, most systems had defined usage cycles and a deterministic set of finite inputs and outputs. Human intervention was then earmarked for situations that fell outside these categories or for situations that were too ambiguous to program. For these systems, a few end-to-end tests were sufficient because most of the complexity was delegated to the operator. 

Today, the requirements for system verification and validation have significantly changed. Many systems are now operating in the real world without human intelligence to fall back on. For example, functionality such as cruise control and automatic lane keeping are now in every new vehicle. Technologies like these show significant promise and are already changing the world for the better. However, fatal accidents such as the one in 2018 involving a Tesla and a truck when the car’s perception system in autopilot mode failed to detect a white truck with a cloudy sky background showed that we still have a long way to go before widespread acceptance and adoption of autonomous vehicles becomes a reality. In fact, according to a 2019 AAA survey, a worrying 71% of people are afraid to drive in fully driverless vehicles. Therefore, the stakes for system level verification and validation have never been higher than they are today.

17B

kilometers of test driving are required to demonstrate a failure rate significantly better than humans - Rand Corporation

How has Verification and Validation changed for modern systems?

System verification and validation can be a complex task, especially in regulated environments. Systems such as autonomous vehicles are constantly exposed to non-linear inputs and quickly changing conditions including weather, people, bicycles, projectiles, etc. Engineers can't simulate all of this in thousands or even millions of runs. To minimize the chances of catastrophic failure and maximize chances of meeting the quality standards and expectations of all stakeholders - including the regulators, engineers need to have:

  • Orders of magnitude more system level test cases. As an example, as of 2020, Alphabet’s Waymo had simulated over 15 billion miles of driving and General Motors’s Cruise ran about 200,000 hours of compute jobs each day. Despite testing their systems over billions of miles, they both still fall into the Level 2 (out of 5) classification of self driving. This means they can autonomously manage speed and steering, but they require the driver to remain focused and take control at any time
  • On-going evaluation of system performance. It is no longer enough to vet systems through release. Engineers need continuous monitoring and assessment through the system's lifecycle. Today, a typical connected vehicle will generate over 25GB of data per hour. That data is fed back to train neural networks to continuously improve the vehicle’s performance

A shift is underway — are your Verification and Validation tools helping or hindering?

Collimator is the only tool that allows you to import big data directly via API, model test cases using Python or a graphical UI, validate system performance in real-time, and create reports to share with your team so you’re all on the same page - literally!

Unlock the full potential of your engineering team with powerful features

Design

Explore the vast number of possibilities, model to your desired complexity

Simulate

Move faster with speed and agility, get high fidelity insights earlier

Visualize

Analyze your results, increase confidence on your designs, iterate

Deploy

Automatically generate code and deploy code to your target hardware

Collaborate

Integrate your workflows and streamline
collaboration

Traditional Applications

Cannot quickly ingest or export the amounts of data required
Involves extra time, effort and money to run HPC simulation
Difficult to collaborate without being in the same room
Requires extra tools to share results or progress with your team
Traditional Applications UI
Collimator UI
Collimator Logo
Seamlessly ingest or export data by directly connecting to your database via API
Quickly test or simulate performance over millions of runs using HPC in the Cloud
Efficiently collaborate with one source of truth and role based access control
Easily create reports to share with system engineers, perception engineers and product teams

Traditional Applications

Traditional Applications
Cannot quickly ingest or export the amounts of data required
Involves extra time, effort and money to run HPC simulation
Difficult to collaborate without being in the same room
Requires extra tools to share results or progress with your team
Collimator Logo
Collimator UI
Seamlessly ingest or export data by directly connecting to your database via API
Quickly test or simulate performance over millions of runs using HPC in the Cloud
Efficiently collaborate with one source of truth and role based access control
Easily create reports to share with system engineers, perception engineers and product teams

Discover how the most innovative companies in the world are using Collimator today

See Collimator in action