Scoring algorithm

In statistics, Fisher's Scoring algorithm is a form of Newton's method used to solve maximum likelihood equations numerically.

Suppose that $Y_{1},\ldots ,Y_{n}$ are random variables, independent and identically distributed with p.d.f. $f_{Y}(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) ${\hat {\theta }}$ of $\theta$. First, suppose we have a starting point for our algorithm $\theta _{0}$, and consider a Taylor expansion of the score function, $V(\theta )$, about $\theta _{0}$:

$V(\theta )\approx V(\theta _{0})-{\mathcal {J}}(\theta _{0})(\theta -\theta _{0})$,

where

${\mathcal {J}}(\theta _{0})=-\prod _{i=1}^{n}\left.{\frac {\partial ^{2}}{\partial \theta ^{2}}}\right|_{\theta =\theta _{0}}\log f(Y_{i};\theta )$

is the observed information at $\theta _{0}$. Now, since

$V({\hat {\theta }})=0$,

then rearranging gives us an algorithm with which to work:

$\theta _{m}=\theta _{m-1}+{\mathcal {J}}^{-1}(\theta _{m-1})V(\theta _{m-1})$.

Then under certain regularity conditions, it can be shown that $\theta _{m}\rightarrow {\hat {\theta }}$.