Scoring algorithm, also known as Fisher's scoring, is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.
Let
Y
1
,
…
,
Y
n
be random variables, independent and identically distributed with twice differentiable p.d.f.
f
(
y
;
θ
)
, and we wish to calculate the maximum likelihood estimator (M.L.E.)
θ
∗
of
θ
. First, suppose we have a starting point for our algorithm
θ
0
, and consider a Taylor expansion of the score function,
V
(
θ
)
, about
θ
0
:
V
(
θ
)
≈
V
(
θ
0
)
−
J
(
θ
0
)
(
θ
−
θ
0
)
,
where
J
(
θ
0
)
=
−
∑
i
=
1
n
∇
∇
⊤
|
θ
=
θ
0
log
f
(
Y
i
;
θ
)
is the observed information matrix at
θ
0
. Now, setting
θ
=
θ
∗
, using that
V
(
θ
∗
)
=
0
and rearranging gives us:
θ
∗
≈
θ
0
+
J
−
1
(
θ
0
)
V
(
θ
0
)
.
We therefore use the algorithm
θ
m
+
1
=
θ
m
+
J
−
1
(
θ
m
)
V
(
θ
m
)
,
and under certain regularity conditions, it can be shown that
θ
m
→
θ
∗
.
In practice,
J
(
θ
)
is usually replaced by
I
(
θ
)
=
E
[
J
(
θ
)
]
, the Fisher information, thus giving us the Fisher Scoring Algorithm:
θ
m
+
1
=
θ
m
+
I
−
1
(
θ
m
)
V
(
θ
m
)
.
