Trisha Shetty (Editor)

Stein's lemma

Updated on
Edit
Like
Comment
Share on FacebookTweet on TwitterShare on LinkedInShare on Reddit

Stein's lemma, named in honor of Charles Stein, is a theorem of probability theory that is of interest primarily because of its applications to statistical inference — in particular, to James–Stein estimation and empirical Bayes methods — and its applications to portfolio choice theory. The theorem gives a formula for the covariance of one random variable with the value of a function of another, when the two random variables are jointly normally distributed.

Contents

Statement of the lemma

Suppose X is a normally distributed random variable with expectation μ and variance σ2. Further suppose g is a function for which the two expectations E(g(X) (X − μ) ) and E( g ′(X) ) both exist (the existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value). Then

E ( g ( X ) ( X − μ ) ) = σ 2 E ( g ′ ( X ) ) .

In general, suppose X and Y are jointly normally distributed. Then

Cov ⁡ ( g ( X ) , Y ) = E ( g ′ ( X ) ) Cov ⁡ ( X , Y ) .

Proof

In order to prove the univariate version of this lemma, recall that the probability density function for the normal distribution with expectation 0 and variance 1 is

φ ( x ) = 1 2 π e − x 2 / 2

and that for a normal distribution with expectation μ and variance σ2 is

1 σ φ ( x − μ σ ) .

Then use integration by parts.

More general statement

Suppose X is in an exponential family, that is, X has the density

f η ( x ) = exp ⁡ ( η ′ T ( x ) − Ψ ( η ) ) h ( x ) .

Suppose this density has support ( a , b ) where a , b could be − ∞ , ∞ and as x → a  or  b , exp ⁡ ( η ′ T ( x ) ) h ( x ) g ( x ) → 0 where g is any differentiable function such that E | g ′ ( X ) | < ∞ or exp ⁡ ( η ′ T ( x ) ) h ( x ) → 0 if a , b finite. Then

E ( ( h ′ ( X ) / h ( X ) + ∑ η i T i ′ ( X ) ) g ( X ) ) = − E g ′ ( X ) .

The derivation is same as the special case, namely, integration by parts.

If we only know X has support R , then it could be the case that E | g ( X ) | < ∞  and  E | g ′ ( X ) | < ∞ but lim x → ∞ f η ( x ) g ( x ) ≠ 0 . To see this, simply put g ( x ) = 1 and f η ( x ) with infinitely spikes towards infinity but still integrable. One such example could be adapted from f ( x ) = { 1 x ∈ [ n , n + 2 − n ) 0 otherwise so that f is smooth.

Extensions to elliptically-contoured distributions also exist.

References

Stein's lemma Wikipedia
(Text) CC BY-SA