Scoring algorithm

In statistics, Fisher's Scoring algorithm is a form of Newton's method used to solve maximum likelihood equations numerically.

Sketch of Derivation

Let $Y_{1},\ldots ,Y_{n}$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) $\theta ^{*}$ of $\theta$. First, suppose we have a starting point for our algorithm $\theta _{0}$, and consider a Taylor expansion of the score function, $V(\theta )$, about $\theta _{0}$:

$V(\theta )\approx V(\theta _{0})-{\mathcal {J}}(\theta _{0})(\theta -\theta _{0})$,

where

${\mathcal {J}}(\theta _{0})=-\prod _{i=1}^{n}\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\right|_{\theta =\theta _{0}}\log f(Y_{i};\theta )$

is the observed information at $\theta _{0}$. Now, setting $\theta =\theta ^{*}$, and using that

$V(\theta ^{*})=0$,

then rearranging gives us:

$\theta ^{*}=\theta _{0}+{\mathcal {J}}^{-1}(\theta _{0})V(\theta _{0})$.

We therefore use the algorithm

$\theta _{m+1}=\theta _{m}+{\mathcal {J}}^{-1}(\theta _{m})V(\theta _{m})$,

and under certain regularity conditions, it can be shown that $\theta _{m}\rightarrow {\hat {\theta }}$.

Fisher Scoring

In practice, ${\mathcal {J}}(\theta )$ is usually replaced by ${\mathcal {I}}(\theta )=\mathrm {E} [J(\theta )]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

$\theta _{m+1}=\theta _{m}+{\mathcal {I}}^{-1}(\theta _{m})V(\theta _{m})$.

Application to Linear Models

The method of Fisher Scoring is often used in the theory of linear models. Suppose we have a standard linear model

$Y=X\beta +\varepsilon$,

where $\varepsilon _{i}\sim N(0,\sigma ^{2})$ independently; now suppose we want to estimate β. It can be shown that

$\beta ^{*}\sim N(\beta ,\sigma ^{2}{\mathcal {I}}^{-1}(\beta ))$

where $\beta ^{*}$ is the M.L.E. of β. It is therefore desirable to find $\beta ^{*}$ and use it as an estimator of β.