Cramér–Rao bound

Statement

The Cramer–Rao bound is stated in this section for several increasingly general cases, beginning with the case in which the parameter is a scalar and its estimator is unbiased. All versions of the bound require certain regularity conditions, which hold for most well-behaved distributions. These conditions are listed later in this section.

Scalar unbiased case

Suppose θ is an unknown deterministic parameter which is to be estimated from measurements x , distributed according to some probability density function f ( x ; θ ) . The variance of any unbiased estimator θ ^ of θ is then bounded by the reciprocal of the Fisher information I ( θ ) :

v a r ( θ ^ ) ≥ 1 I ( θ )

where the Fisher information I ( θ ) is defined by

I ( θ ) = E [ ( ∂ ℓ ( x ; θ ) ∂ θ ) 2 ] = − E [ ∂ 2 ℓ ( x ; θ ) ∂ θ 2 ]

and ℓ ( x ; θ ) = log ⁡ ( f ( x ; θ ) ) is the natural logarithm of the likelihood function and E denotes the expected value (over x ).

The efficiency of an unbiased estimator θ ^ measures how close this estimator's variance comes to this lower bound; estimator efficiency is defined as

e ( θ ^ ) = I ( θ ) − 1 v a r ( θ ^ )

or the minimum possible variance for an unbiased estimator divided by its actual variance. The Cramér–Rao lower bound thus gives

e ( θ ^ ) ≤ 1.  

General scalar case

A more general form of the bound can be obtained by considering an unbiased estimator T ( X ) of the parameter θ . Here, unbiasedness is understood as stating that E { T ( X ) } = ψ ( θ ) . In this case, the bound is given by

v a r ( T ) ≥ [ ψ ′ ( θ ) ] 2 I ( θ )

where ψ ′ ( θ ) is the derivative of ψ ( θ ) (by θ ), and I ( θ ) is the Fisher information defined above.

Bound on the variance of biased estimators

Apart from being a bound on estimators of functions of the parameter, this approach can be used to derive a bound on the variance of biased estimators with a given bias, as follows. Consider an estimator θ ^ with bias b ( θ ) = E { θ ^ } − θ , and let ψ ( θ ) = b ( θ ) + θ . By the result above, any unbiased estimator whose expectation is ψ ( θ ) has variance greater than or equal to ( ψ ′ ( θ ) ) 2 / I ( θ ) . Thus, any estimator θ ^ whose bias is given by a function b ( θ ) satisfies

V a r ( θ ^ ) ≥ [ 1 + b ′ ( θ ) ] 2 I ( θ ) .

The unbiased version of the bound is a special case of this result, with b ( θ ) = 0 .

It's trivial to have a small variance − an "estimator" that is constant has a variance of zero. But from the above equation we find that the mean squared error of a biased estimator is bounded by

E ( ( θ ^ − θ ) 2 ) ≥ [ 1 + b ′ ( θ ) ] 2 I ( θ ) + b ( θ ) 2 ,

using the standard decomposition of the MSE. Note, however, that if 1 + b ′ ( θ ) < 1 this bound might be less than the unbiased Cramér–Rao bound 1 / I ( θ ) . For instance, in the example of estimating variance below, 1 + b ′ ( θ ) = n n + 2 < 1 .

Multivariate case

Extending the Cramér–Rao bound to multiple parameters, define a parameter column vector

θ = [ θ 1 , θ 2 , … , θ d ] T ∈ R d

with probability density function f ( x ; θ ) which satisfies the two regularity conditions below.

The Fisher information matrix is a d × d matrix with element I m , k defined as

I m , k = E [ ∂ ∂ θ m log ⁡ f ( x ; θ ) ∂ ∂ θ k log ⁡ f ( x ; θ ) ] = − E [ ∂ 2 ∂ θ m ∂ θ k log ⁡ f ( x ; θ ) ] .

Let T ( X ) be an estimator of any vector function of parameters, T ( X ) = ( T 1 ( X ) , … , T d ( X ) ) T , and denote its expectation vector E [ T ( X ) ] by ψ ( θ ) . The Cramér–Rao bound then states that the covariance matrix of T ( X ) satisfies

c o v θ ( T ( X ) ) ≥ ∂ ψ ( θ ) ∂ θ [ I ( θ ) ] − 1 ( ∂ ψ ( θ ) ∂ θ ) T

where

The matrix inequality A ≥ B is understood to mean that the matrix A − B is positive semidefinite, and

∂ ψ ( θ ) / ∂ θ is the Jacobian matrix whose i j element is given by ∂ ψ i ( θ ) / ∂ θ j .

If T ( X ) is an unbiased estimator of θ (i.e., ψ ( θ ) = θ ), then the Cramér–Rao bound reduces to

c o v θ ( T ( X ) ) ≥ I ( θ ) − 1 .

If it is inconvenient to compute the inverse of the Fisher information matrix, then one can simply take the reciprocal of the corresponding diagonal element to find a (possibly loose) lower bound.

v a r θ ( T m ( X ) ) = [ c o v θ ( T ( X ) ) ] m m ≥ [ I ( θ ) − 1 ] m m ≥ ( [ I ( θ ) ] m m ) − 1 .

Regularity conditions

The bound relies on two weak regularity conditions on the probability density function, f ( x ; θ ) , and the estimator T ( X ) :

The Fisher information is always defined; equivalently, for all x such that f ( x ; θ ) > 0 ,

exists, and is finite.

The operations of integration with respect to x and differentiation with respect to θ can be interchanged in the expectation of T ; that is,

whenever the right-hand side is finite. This condition can often be confirmed by using the fact that integration and differentiation can be swapped when either of the following cases hold:

1. The function f ( x ; θ ) has bounded support in x , and the bounds do not depend on θ ;
2. The function f ( x ; θ ) has infinite support, is continuously differentiable, and the integral converges uniformly for all θ .

Simplified form of the Fisher information

Suppose, in addition, that the operations of integration and differentiation can be swapped for the second derivative of f ( x ; θ ) as well, i.e.,

∂ 2 ∂ θ 2 [ ∫ T ( x ) f ( x ; θ ) d x ] = ∫ T ( x ) [ ∂ 2 ∂ θ 2 f ( x ; θ ) ] d x .

In this case, it can be shown that the Fisher information equals

I ( θ ) = − E [ ∂ 2 ∂ θ 2 log ⁡ f ( X ; θ ) ] .

The Cramèr–Rao bound can then be written as

v a r ( θ ^ ) ≥ 1 I ( θ ) = 1 − E [ ∂ 2 ∂ θ 2 log ⁡ f ( X ; θ ) ] .

In some cases, this formula gives a more convenient technique for evaluating the bound.

Single-parameter proof

The following is a proof of the general scalar case of the Cramér–Rao bound described above. Assume that T = t ( X ) is an unbiased estimator for the value ψ ( θ ) (based on the observations X ), and so E ( T ) = ψ ( θ ) . The goal is to prove that, for all θ ,

v a r ( t ( X ) ) ≥ [ ψ ′ ( θ ) ] 2 I ( θ ) .

Let X be a random variable with probability density function f ( x ; θ ) . Here T = t ( X ) is a statistic, which is used as an estimator for ψ ( θ ) . Define V as the score:

V = ∂ ∂ θ ln ⁡ f ( X ; θ ) = 1 f ( X ; θ ) ∂ ∂ θ f ( X ; θ )

where the chain rule is used in the final equality above. Then the expectation of V , written E ( V ) , is zero. This is because:

E ( V ) = ∫ f ( x ; θ ) [ 1 f ( x ; θ ) ∂ ∂ θ f ( x ; θ ) ] d x = ∂ ∂ θ ∫ f ( x ; θ ) d x = 0

where the integral and partial derivative have been interchanged (justified by the second regularity condition).

If we consider the covariance c o v ( V , T ) of V and T , we have c o v ( V , T ) = E ( V T ) , because E ( V ) = 0 . Expanding this expression we have

c o v ( V , T ) = E ( T ⋅ [ 1 f ( X ; θ ) ∂ ∂ θ f ( X ; θ ) ] ) = ∫ t ( x ) [ 1 f ( x ; θ ) ∂ ∂ θ f ( x ; θ ) ] f ( x ; θ ) d x = ∂ ∂ θ [ ∫ t ( x ) f ( x ; θ ) d x ] = ψ ′ ( θ )

again because the integration and differentiation operations commute (second condition).

The Cauchy–Schwarz inequality shows that

v a r ( T ) v a r ( V ) ≥ | c o v ( V , T ) | = | ψ ′ ( θ ) |

therefore

v a r ( T ) ≥ [ ψ ′ ( θ ) ] 2 v a r ( V ) = [ ψ ′ ( θ ) ] 2 I ( θ )

which proves the proposition.

Multivariate normal distribution

For the case of a d-variate normal distribution

x ∼ N d ( μ ( θ ) , C ( θ ) )

the Fisher information matrix has elements

I m , k = ∂ μ T ∂ θ m C − 1 ∂ μ ∂ θ k + 1 2 t r ( C − 1 ∂ C ∂ θ m C − 1 ∂ C ∂ θ k )

where "tr" is the trace.

For example, let w [ n ] be a sample of N independent observations with unknown mean θ and known variance σ 2 .

w [ n ] ∼ N N ( θ 1 , σ 2 I ) .

Then the Fisher information is a scalar given by

I ( θ ) = ( ∂ μ ( θ ) ∂ θ ) T C − 1 ( ∂ μ ( θ ) ∂ θ ) = ∑ i = 1 N 1 σ 2 = N σ 2 ,

and so the Cramér–Rao bound is

v a r ( θ ^ ) ≥ σ 2 N .

Normal variance with known mean

Suppose X is a normally distributed random variable with known mean μ and unknown variance σ 2 . Consider the following statistic:

T = ∑ i = 1 n ( X i − μ ) 2 n .

Then T is unbiased for σ 2 , as E ( T ) = σ 2 . What is the variance of T?

v a r ( T ) = v a r ( X − μ ) 2 n = 1 n [ E { ( X − μ ) 4 } − ( E { ( X − μ ) 2 } ) 2 ]

(the second equality follows directly from the definition of variance). The first term is the fourth moment about the mean and has value 3 ( σ 2 ) 2 ; the second is the square of the variance, or ( σ 2 ) 2 . Thus

v a r ( T ) = 2 ( σ 2 ) 2 n .

Now, what is the Fisher information in the sample? Recall that the score V is defined as

V = ∂ ∂ σ 2 log ⁡ L ( σ 2 , X )

where L is the likelihood function. Thus in this case,

V = ∂ ∂ σ 2 log ⁡ [ 1 2 π σ 2 e − ( X − μ ) 2 / 2 σ 2 ] = ( X − μ ) 2 2 ( σ 2 ) 2 − 1 2 σ 2

where the second equality is from elementary calculus. Thus, the information in a single observation is just minus the expectation of the derivative of V, or

I = − E ( ∂ V ∂ σ 2 ) = − E ( − ( X − μ ) 2 ( σ 2 ) 3 + 1 2 ( σ 2 ) 2 ) = σ 2 ( σ 2 ) 3 − 1 2 ( σ 2 ) 2 = 1 2 ( σ 2 ) 2 .

Thus the information in a sample of n independent observations is just n times this, or n 2 ( σ 2 ) 2 .

The Cramer Rao bound states that

v a r ( T ) ≥ 1 I .

In this case, the inequality is saturated (equality is achieved), showing that the estimator is efficient.

However, we can achieve a lower mean squared error using a biased estimator. The estimator

T = ∑ i = 1 n ( X i − μ ) 2 n + 2 .

obviously has a smaller variance, which is in fact

v a r ( T ) = 2 n ( σ 2 ) 2 ( n + 2 ) 2 .

Its bias is

( 1 − n n + 2 ) σ 2 = 2 σ 2 n + 2

so its mean squared error is

M S E ( T ) = ( 2 n ( n + 2 ) 2 + 4 ( n + 2 ) 2 ) ( σ 2 ) 2 = 2 ( σ 2 ) 2 n + 2

which is clearly less than the Cramér–Rao bound found above.

When the mean is not known, the minimum mean squared error estimate of the variance of a sample from Gaussian distribution is achieved by dividing by n + 1, rather than n − 1 or n + 2.