Interval predictor model

In Regression analysis, an Interval Predictor Model (IPM) is an approach to regression where bounds on the function to be approximated are obtained. This differs from other techniques in Machine Learning, where usually one wishes to estimate point values or an entire probability distribution. Interval Predictor Models are sometimes referred to as a nonparametric regression technique, because a potentially infinite set of functions are contained by the IPM, and no specific distribution is implied for the regressed variables. As a consequence of the theory of Scenario Optimization, in many cases rigorous predictions can be made regarding the performance of the model at test time . Hence an Interval Predictor Model can be seen as a guaranteed bound on quantile regression.

Convex Interval Predictor Models

Typically the Interval Predictor Model is created by specifying a parametric function, which is usually chosen to be the product of a parameter vector and a basis. Usually the basis is made up of polynomial features or a radial basis is sometimes used. Then a convex set is assigned to the parameter vector, and the size of the convex set is minimized such that every possible data point can be predicted by one possible value of the parameters. Ellipsoidal parameters sets were used by Campi (2009), which yield a convex optimization program to train the IPM . Crespo (2016) proposed the use of a hyperrectangular parameter set . This results in a very convenient form for the bounds of the IPM, and hence the IPM can be trained with a linear optimization program:

$\operatorname {arg\,min} _{p}{\{\mathbb {E} _{x}({\bar {y}}_{p}(x)-{\underline {y}}_{p}(x)):{\bar {y}}_{p}(x^{(i)})>y^{(i)}>{\underline {y}}_{p}(x^{(i)}),i=1,...,N\}}$

where the training data examples are $y^{(i)}$ and $x^{(i)}$, and the Interval Predictor Model bounds ${\underline {y}}_{p}(x)$ and ${\overline {y}}_{p}(x)$ are parameterised by the parameter vector $p$. The reliability of such an IPM is obtained by noting that for a convex IPM the number of support constraints is less than the dimensionality of the trainable parameters, and hence the scenario approach can be applied.

Lacerda (2017) demonstrated that this approach can be extended to situations where the training data is interval valued rather than point valued .

Non-convex Interval Predictor Models

In Campi (2015) a non-convex theory of scenario optimization was proposed . This involves measuring the number of support constraints, $S$, for the Interval Predictor Model after training and hence making predictions about the reliability of the model. This enables non-convex IPMs to be created, such as a single layer neural network. Campi (2015) demonstrates that an algorithm with cost ${\mathcal {O}}(S)$ can determine the reliability of the model at test time in an a-priori fashion (i.e. without an evaluation on a validation set) . This is achieved by minimizing the max-error loss function given by

${\mathcal {L}}_{\text{max-error}}=\max _{i}|y^{(i)}-{\hat {y}}_{p}(x^{(i)})|$

where the Interval Predictor Model center line ${\hat {y}}_{p}(x)=({\overline {y}}_{p}(x)+{\underline {y}}_{p}(x))\times 1/2$. This results in an IPM which makes predictions with homoscedastic uncertainty.

Sadeghi (2019) demonstrates that the non-convex scenario approach from Campi (2015) can be extended to train deeper Neural Networks which predict intervals with hetreoscedastic uncertainty on datasets with Imprecision .

Applications

Initially, Scenario Optimization was applied to robust control problems . Crespo (2015) applied Interval Predictor Models to the design of space radiation shielding . In Patelli (2017), Faes (2019), and Crespo (2018), Interval Predictor models were applied to the Structural reliability analysis problem . Brandt (2017) applies Interval Predictor Models to fatigue damage estimation of offshore wind turbines jacket substructures .

Software Implementations

PyIPM provides an Open source Matlab and Python implementation of the work of Crespo (2015) . A similar implementation is available in the OpenCOSSAN software .