What is model validation?

If you are already familiar with the concept of model validation you can skip this Section.

Imagine that you are developing a super product or process. At some point, you need to test test it. Then, based on the test outcome, you continue your development in a certain direction. Then, at some point you test your work-product again… and you keep in iterating such a process over and over until you get something good to deploy.

This is the way things are typically developed.

However, running tests in a target environment (i.e. the real-world environment where your product - or service - shall be ultimately deployed) has an associated cost both in terms of money and in terms of time. Very often such a cost also include personal stress.

A solution to alleviate such pain would be to run your tests in a virtual environment where you have a model of your target environment. If the tests run in the virtual environment show good performance of your work-product, then you should get the same good performance when you move to the target environment.

Well, you should because the previous statement is true if, and only if, your virtual environment adequately represents the target environment and if it behaves similarly.

Model validation is the process of evaluating how good is the model of your target environment, i.e. it checks the similarity between your virtual and target environments through some validation metrics.

A typical validation process consists in the following steps:

  1. Design a set of experiments to be carried out on the target environment (aka DoE). The set of experiments consists in defining the set of stimuli (input) to be given to the target environment,

  2. Run the experiments of point 1. on the target environment and log its response. The set of the input signals along with the response of your target environment is denoted as measurement dataset (or, in short, just dataset),

  3. Run exactly the same experiment defined in point 1. on your model and you log its response. We refer to the response of your model as simulation results,

  4. Evaluate how “close” are the simulation results of point 3. and the logged response of point 2. with respect to some validation metrics.

../_images/ModelValidation.svg

The model validation process. In this picture the validation method only returns a pass/fail value but in general it returns the evaluation of some model quality metrics.

If the results of step 4. are good, then you can safely keep in developing and test in the virtual environment. Most likely things will work in the target environment as well - but it is good practice to verify that every once in a while.

Keep in mind that “all models are wrong, but some are useful.” ;-).

However, it is worth noting that your model is valid only within the region covered by the dataset. If you plan to use your model with data outside the dataset coverage region, then you have no guarantees that things will work in target environment as they worked in the virtual environment.

Example Let’s show how steps 1-4 can be applied through a simple real-world example.

Assume that you are developing some cool autonomous driving algorithm that shall be deployed in a car, which represent your target environment.

Assume that you developed the model of a car where its input signals are:

  1. accelerator pedal position,

  2. steering wheel position and

  3. road profile,

whereas its output signals are:

  1. longitudinal speed and

  2. lateral speed.

Next, you want to validate your model.

Steps 1-4 are carried out in the following way:

  1. You choose a driving route with sufficiently road slope variation. You decide to take a ride on that path by adopting a nasty driving style with sudden accelerations and abrupt steering movements. Congrats! You just made a Design of Experiment (DoE).

  2. You take a ride with the target vehicle and you drive according to the DoE performed in the previous step. You log the input signals (i.e. the accelerator pedal position, the steering wheel position and the road profile time-series) along with the output signals (i.e. longitudinal and lateral speed time-series) of the vehicle while driving. Such logs represent your dataset. Note how input and output are separated.

  3. Feed your model with the input signals that you logged during the drive and log your model output corresponding to the longitudinal and lateral vehicle speed dynamics.

  4. Compare the longitudinal and lateral vehicle speed time-series logged during the actual drive with the simulated results with respect to some validation metrics.

You haven’ finished yet. In-fact, when you develop and validate a model, you should also consider the coverage region of the model along with the validation results.

If you logged data only in the accelerator pedal position range [0,40] %, the steering angle in the range [-2,2]° and the road profile was flat for all the time, then you have to ship such an information along with your model to a potential model user.

The cost saving when using models is clear, but there is no free lunch. In-fact, the challenge relies in the design of good models.

Nevertheless, although the design of good models is an art that cannot be completely automated, we can at least validate them automatically and here is where Dymoval comes into play. In-fact, Dymoval will not help you in developing any model at all, it will just tell you if your models are good or not after you developed them.

Note

A small caution on the nomenclature used: we will interchangeably use the expressions real-world system and target environment. This because what Control System engineers call system is often referred as environment by Software Engineers.