Statistical model validation
In statistics, model validation is the task of confirming that the outputs of a statistical model are acceptable with respect to the real data-generating process. In other words, model validation is the task of confirming that the outputs of a statistical model have enough fidelity to the outputs of the data-generating process that the objectives of the investigation can be achieved.
Overview
Model validation can be based on two types of data: data that was used in the construction of the model and data that was not used in the construction. Validation based on the first type usually involves analyzing the goodness of fit of the model or analyzing whether the residuals seem to be random (i.e. residual diagnostics). Validation based on the second type usually involves analyzing whether the model's predictive performance deteriorates non-negligibly when applied to pertinent new data.
Validation based on only the first type (data that was used in the construction of the model) is often inadequate. An extreme example is illustrated in Figure 1. The figure displays data (black dots) that was generated via a straight line + noise. The figure also displays a curvy line, which is a polynomial chosen to fit the data perfectly. The residuals for the curvy line are all zero. Hence validation based on only the first type of data would conclude that the curvy line was a good model. Yet the curvy line is obviously a poor model: interpolation, especially between −5 and −4, would tend to be highly misleading; moreover, any substantial extrapolation would be bad.
Thus, validation is usually not based on only considering data that was used in the construction of the model; rather, validation also employs data that was not used in the construction. In other words, validation usually includes testing some of the model's predictions.
For some classes of statistical models, specialized methods of performing validation are available. As an example, if the statistical model was obtained via a regression, then specialized analyses for regression validation exist and are generally employed.
When doing a validation, there are three notable causes of potential difficulty, according to the Encyclopedia of Statistical Sciences (2006).[1] The three causes are these: lack of data; lack of control of the input variables; uncertainty about the underlying probability distributions and correlations.
See also
Notes
- ^ Deaton, M. L. (2006), "Simulation models, validation of", in Kotz, S.; et al. (eds.), Encyclopedia of Statistical Sciences, Wiley.
References
- National Research Council (2012), "Chapter 5: Model validation and prediction", Assessing the Reliability of Complex Models: Mathematical and statistical foundations of verification, validation, and uncertainty quantification, Washington, DC: National Academies Press, pp. 52–85
{{citation}}: CS1 maint: multiple names: authors list (link).
Bibliography
- Altman, D. G.; Royston, P. (2000), "What do we mean by validating a prognostic model?", Statistics in Medicine, 19: 453–473, doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
- Barlas, Y. (1996), "Formal aspects of model validity and validation in system dynamics", System Dynamics Review, 12: 183–210, doi:10.1002/(SICI)1099-1727(199623)12:3<183::AID-SDR103>3.0.CO;2-4
- Batzel, J. J.; Bachar, M.; Kappel, F., eds. (2013), Mathematical Modeling and Validation in Physiology, Springer
- Huber-Carol, C.; Balakrishnan, N.; Nikulin, M. S.; Mesbah, M., eds. (2002), Goodness-of-Fit Tests and Model Validity, Springer
- Mayer, D. G.; Butler, D.G. (1993), "Statistical validation", Ecological Modelling, 68: 21–32, doi:10.1016/0304-3800(93)90105-2
External links
- How can I tell if a model fits my data? —Handbook of Statistical Methods (NIST)
- What are core statistical model validation techniques? —Stack Exchange
 | This statistics-related article is a stub. You can help Wikipedia by expanding it. |