Design of Experiments (DoE)

When running experiments, it is important to stimulate the target environment in a way that we extract as much information as possible from it.

Good experiments shall stress the target environment as much as possible under different conditions to get a sufficiently informative measurement dataset.

Example

If we are developing the model of a car, we want to log sensors measurements while driving within a wide range of speeds, with different accelerations profiles, in different road and weather conditions and so on and so forth.

If we log sensors measurements only when we are driving on a flat road and in the range 0-10 km/h and by doing exactly the same maneuvers over and over, then it would be hard to disagree on that the collected measurements dataset is poorly informative.

Ideally, the best approach would be to stimulate the system at every frequency. Ideally, such a stimulus should be a white noise signal. In practice, a white noise signal can be mimicked through a Pseudo-Random-Binary-Sequence (PRBS), which is extremely easy to implement on digital platforms. Alternatively, you could use a chirp signal, or you could just manually drive your real system as randomly as possible. Regardless of the approach you choose, the goal is to hit as many corners as possible.

Next, let’s say that we have completed your experiments. To assess the quality of the experiments, we can simply check how similar the input signal was to white noise. This is done by analyzing the auto-correlation function of our input signal. Dymoval has the XCorrelation class that can be used for this purpose.

Assume that you have a signal u expressed as a \(N \times p\) array, where \(N\) is the number of samples and \(p\) is the dimension of the signal.

You can create and analyze the auto-correlation function of the signal u with the following:

from dymoval.correlation import XCorrelation

Ruu = XCorrelation("Ruu", u, u)
# Plot the auto-correlation function
Ruu.plot()
# Estimate whiteness
Ruu.whiteness()

The whiteness level of the signal u is computed according to different metrics, see compute_statistics().

Note

Sometimes it is not possible to stimulate our real-world system as randomly as possible. For example, if you are designing a space-shuttle I don’t think you will get any approval for driving it randomly in the sky only for the purpose of collecting measurements. In this case, you can simply ignore this evaluation metric, and if you are validating your model you can pass the argument ignore_input=True to validate_models() or to ValidationSession class constructor.

Over-sampled signals

If the signal under examination is over-sampled, then the whiteness results may be inaccurate. This occurs because consecutive measurements are naturally correlated when a sensor pick samples too quickly, leading to higher values in the auto-correlation function. In this case you are analyzing how measurements are correlated whereas we are interested in the overall signal auto-correlation.

To accommodate this issue, dymoval down-sample the auto-correlation function to the limit given by Shannon-Nyquist criteria. That is, dymoval compute the auto-correlation of signals by considering a lag time \(lag\_time = 1/(2B)\), where \(B\) is the signal bandwidth. This adjustment ensures that we are analyzing the auto-correlation of the actual signal rather than consecutive measurements. Dymoval handles this automatically, provided that the bandwidth and sampling period are supplied to the constructor of the XCorrelation class.

Note

In future releases we plan to further provide measures (Cramer-Rao Theorem? Fisher Matrix?) on the information level contained in a given measurements dataset in within its coverage region.

Coverage region

Other than having an informative measurement dataset in terms of signal variation in terms of similarity to white-noise, we also want to cover as much amplitude as possible during our experiments. Such an amplitude is referred as coverage region.

It is worth mentioning that size of the coverage region and whiteness within go hand in hand, as shown in the next example.

Example

With reference to the example above, imagine driving the car only at constant speeds ranging from 0 to 180 km/h on a flat road, without ever accelerating or braking. For instance, you make a first run driving (and logging) data at a constant speed of 10 km/h, without accelerating or braking, and staying on a flat road. Then, you perform a second run at a constant speed of 20 km/h under the same conditions as the previous run, and so on, until reaching the final run at 180 km/h.

Your measurements dataset will have a fairly large coverage region, but it will contain little information since all the runs were conducted at constant speeds without any acceleration or braking, and on a flat road.

Hence, at the end of this phase we should have both the coverage region and the information richness of the input signal.