Scoring algorithm explained

Scoring algorithm, also known as Fisher's scoring,^[1] is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.

Sketch of derivation

Let

Y _{1,\ldots,Y} _{n} be random variables, independent and identically distributed with twice differentiable p.d.f. f(y; \theta) , and we wish to calculate the maximum likelihood estimator (M.L.E.) \theta ^{*} of \theta . First, suppose we have a starting point for our algorithm \theta _{0} , and consider a Taylor expansion of the score function, V(\theta) , about \theta _{0} :V(\theta) ≈ V(\theta _{0)} - l{J}(\theta _{0)(\theta} - \theta _{0),}

where

l{J}(\theta _{0)} = - \sum ^{n} _{i =1} \left. \nabla \nabla ^{\top} \right| _{\theta = \theta _{0}} log f(Y _{i} ; \theta)

is the observed information matrix at

\theta _{0} . Now, setting \theta = \theta ^{*} , using that V(\theta ^{*)} = 0 and rearranging gives us:\theta ^{*} ≈ \theta _{0} + l{J} ^{-1} (\theta _{0} )V(\theta _{0} ).

We therefore use the algorithm

\theta _{m+1} = \theta _{m} + l{J} ^{-1} (\theta _{m} )V(\theta _{m} ),

and under certain regularity conditions, it can be shown that

\theta _{m} → \theta ^{*} .

Fisher scoring

In practice,

l{J}(\theta) is usually replaced by l{I}(\theta)= E [ l{J}(\theta)] , the Fisher information, thus giving us the Fisher Scoring Algorithm:\theta _{m+1} = \theta _{m} + l{I} ^{-1} (\theta _{m} )V(\theta _{m} ) ..

Under some regularity conditions, if

\theta _{m} is a consistent estimator, then \theta _{m+1} (the correction after a single step) is 'optimal' in the sense that its error distribution is asymptotically identical to that of the true max-likelihood estimate.

See also

• Score (statistics)
• Score test
• Fisher information

Further reading

• Jennrich . R. I. . Sampson . P. F. . amp . 1976 . Newton-Raphson and Related Algorithms for Maximum Likelihood Variance Component Estimation . . 18 . 1 . 11–17 . 10.1080/00401706.1976.10489395 . 31 January 2024 . 1267911 .

Notes and References

1. Nicholas T. . Longford . A fast scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects . Biometrika . 74 . 4 . 1987 . 817–827 . 10.1093/biomet/74.4.817 .