Talk:Cross-validation (statistics)

Claim about OLS' downward bias in the expected MSE

The article makes the following claim:

${\begin{aligned}{\text{MSE}}&={\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2}={\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-a-{\boldsymbol {\beta }}^{T}\mathbf {x} _{i})^{2}\\&={\frac {1}{n}}\sum _{i=1}^{n}(y_{i}-a-\beta _{1}x_{i1}-\dots -\beta _{p}x_{ip})^{2}\end{aligned}}$

If the model is correctly specified, it can be shown under mild assumptions that the expected value of the MSE for the training set is (n − p − 1)/(n + p + 1) < 1 times the expected value of the MSE for the validation set (the expected value is taken over the distribution of training sets).

The text cites Trippa et al. (2015) specifically about the bias factor $(n-p-1)/(n+p+1)$. However, the paper does not seem to contain any discussion of this bias factor for OLS. Is there an algebraic proof available for OLS?

Based on a simple simulation, the claim seems to be true.

Simulation in R

draw_sample <- function(n) {
    X <- rnorm(n)
    Z <- rnorm(n)
    epsilon <- rnorm(n)

    data.frame(
        Y = .1 + .3 * X + .4 * Z + epsilon,
        X = X,
        Z = Z)
}

mse <- function(model, data) {
    Y_hat <- predict(model, data)

    mean((data$Y - Y_hat)^2)
}

draw_mse <- function(n_training, n_validation) {
    data <- draw_sample(n_training + n_validation)
    data_training <- data[1:n_training,]
    data_validation <- data[(n_training + 1):nrow(data),]

    model <- lm(Y ~ X + Z, data = data_training)

    c(mse(model, data_training),
      mse(model, data_validation))
}

simulate <- function(n_samples) {
    sapply(
        1:n_samples,
        function(x) {
            draw_mse(n_training = 50, n_validation = 50)
        })
}

x <- simulate(10000)
mean(log(x[1,]) - log(x[2,]))

The resulting mean log ratio of the MSEs on the training set and the validation set is very similar to the formula given by the article. E.g., $-0.1241675$ which is close to $\ln((50-3)/(50+3))\approx -0.1201443$.

chery (talk) 16:32, 17 June 2022 (UTC); edited 17:37, 17 June 2022 (UTC)

"Swap sampling"

Is there another paper describing this method? The cited paper doesn't even call it "swap sampling" 24.13.125.183 (talk) 00:57, 15 February 2023 (UTC)