Explained sum of squares

In statistics, an explained sum of squares (ESS) is the sum of squared predicted values in a standard regression model (for example y_i = a + bx_i + ε_i), where y_i is the response variable, x_i is the explanatory variable, a and b are coefficients, i indexes the observations from 1 to n, and ε_i is the error term. In general, the greater the ESS, the better the model performs in its estimation.

If ${\hat {a}}$ and ${\hat {b}}$ are the estimated coefficients, then

${\hat {y}}_{i}={\hat {a}}+{\hat {b}}x_{i}$

is the predicted variable. The ESS is the sum of the squares of the differences of the predicted values and the grand mean:

${\text{ESS}}=\sum _{i=1}^{n}\left({\hat {y}}_{i}-{\bar {y}}\right)^{2}.$

In general: total sum of squares = explained sum of squares + residual sum of squares.

Type I SS

Type one estimates of the sum of squares explained by a model in a variable are obtained when sums of squares for a model are calculated sequentially (e.g. with the model Y = aX₁ + bX₂ + cX₃). Sums of squares are calculated for a using the model Y = aX₁ and sums of squares for b are calculated using the model Y = aX₁ + bX₂, and sums of squares for c are calculated using the model Y = aX₁ + bX₂ + cX₃.

Type II SS

Type two sums of squares are calculated for a variable after adjusting for every other factor of equal or lesser order. For example in the model Y = A + B + C + AB + BC + AC + ABC, the type II sum of squares for A is obtained by adjusting for B and C.

Type III SS

The type III sum of squares is calculated by comparing the full model, to the full model without the variable of interest. So it is considered to be the additional variability explained by adding the variable of interest. It is the same as the Type I ss when the variable is the last variable in the model.

Partitioning in simple linear regression

The following equality is generally true in simple linear regression.

$\sum _{i=1}^{n}\left(y_{i}-{\bar {y}}\right)^{2}=\sum _{i=1}^{n}\left(y_{i}-{\hat {y}}_{i}\right)^{2}+\sum _{i=1}^{n}\left({\hat {y}}_{i}-{\bar {y}}\right)^{2}.$

Simple Derivation

${\begin{aligned}\sum _{i=1}^{n}(y_{i}-{\bar {y}})&=\sum _{i=1}^{n}(y_{i}-{\bar {y}}+({\hat {y}}_{i}-{\hat {y}}_{i}))\\&=\sum _{i=1}^{n}((y_{i}-{\hat {y}}_{i})+({\hat {y}}_{i}-{\bar {y}}))\\\end{aligned}}$

Square the terms on both sides. In general, these sums would not be equal, but in this case, each $i$th term on both sides of the sum are the same.

$\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}=\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2}+\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}2({\hat {y}}_{i}-{\bar {y}})(y_{i}-{\hat {y}}_{i})$

simple linear regression gives us ${\hat {a}}={\bar {y}}-{\hat {b}}{\bar {x}}$. What follows depends on this.

${\begin{aligned}\sum _{i=1}^{n}2({\hat {y}}_{i}-{\bar {y}})(y_{i}-{\hat {y}}_{i})&=\sum _{i=1}^{n}2(({\bar {y}}-{\hat {b}}{\bar {x}}+{\hat {b}}x_{i})-{\bar {y}})(y_{i}-{\hat {y}}_{i})\\&=\sum _{i=1}^{n}2(({\bar {y}}+{\hat {b}}(x_{i}-{\bar {x}}))-{\bar {y}})(y_{i}-{\hat {y}}_{i})\\&=\sum _{i=1}^{n}2({\hat {b}}(x_{i}-{\bar {x}}))(y_{i}-{\hat {y}}_{i})\\&=\sum _{i=1}^{n}2{\hat {b}}(x_{i}-{\bar {x}})(y_{i}-({\bar {y}}+{\hat {b}}(x_{i}-{\bar {x}})))\\&=\sum _{i=1}^{n}2{\hat {b}}((y_{i}-{\bar {y}})(x_{i}-{\bar {x}})-{\hat {b}}(x_{i}-{\bar {x}})^{2})\end{aligned}}$

Again simple linear regression gives us ${\hat {b}}=(\sum _{i=1}^{n}(x_{i}-{\bar {x}})(y_{i}-{\bar {y}}))/(\sum _{i=1}^{n}(x_{i}-{\bar {x}})^{2})$

${\begin{aligned}\sum _{i=1}^{n}2({\hat {y}}_{i}-{\bar {y}})(y_{i}-{\hat {y}}_{i})&=\sum _{i=1}^{n}2{\hat {b}}((y_{i}-{\bar {y}})(x_{i}-{\bar {x}})-{\hat {b}}(x_{i}-{\bar {x}})^{2})\\&=2{\hat {b}}\sum _{i=1}^{n}((y_{i}-{\bar {y}})(x_{i}-{\bar {x}})-(y_{i}-{\bar {y}})(x_{i}-{\bar {x}}))\\&=2{\hat {b}}\cdot 0=0\end{aligned}}$

See also

• Sum of squares