Stein's lemma explained

Stein's lemma, named in honor of Charles Stein, is a theorem of probability theory that is of interest primarily because of its applications to statistical inference - in particular, to James–Stein estimation and empirical Bayes methods - and its applications to portfolio choice theory.^[1] The theorem gives a formula for the covariance of one random variable with the value of a function of another, when the two random variables are jointly normally distributed.

Note that the name "Stein's lemma" is also commonly used^[2] to refer to a different result in the area of statistical hypothesis testing, which connects the error exponents in hypothesis testing with the Kullback–Leibler divergence. This result is also known as the Chernoff–Stein lemma^[3] and is not related to the lemma discussed in this article.

Statement of the lemma

Suppose X is a normally distributed random variable with expectation μ and variance σ². Further suppose g is a differentiable function for which the two expectations E(g(X) (X - μ)) and E(g ′(X)) both exist. (The existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value.) Then

E l (g(X)(X - \mu) r ) = \sigma ^{2} E l (g'(X) r ).

In general, suppose X and Y are jointly normally distributed. Then

\operatorname{Cov}(g(X),Y) = \operatorname{Cov}(X,Y)E(g'(X)).

For a general multivariate Gaussian random vector

(X _{1,} ..., X _{n)} \sim N(\mu, \Sigma) it follows thatE l (g(X)(X - \mu) r ) = \Sigma ⋅ E l (\nabla g(X) r ).

Proof

The univariate probability density function for the univariate normal distribution with expectation 0 and variance 1 is

\varphi(x) = { 1 \over \sqrt{ 2 \pi}}e ^{- x ^{2/2}}

Since

\int x \exp( - x ^{2/2)dx} = - \exp( - x ^{2/2)} we get from integration by parts:E[g(X)X] = (1)/(\sqrt{ 2 \pi) }\int g(x) x \exp(-x^2/2)\,dx= \frac\int g'(x) \exp(-x^2/2)\,dx= E[g'(X)].

The case of general variance

\sigma ^{2} follows by substitution.

More general statement

Isserlis' theorem is equivalently stated as$$\backslash operatorname(X\_1\; f(X\_1,\backslash ldots,X\_n))=\backslash sum\_^\; \backslash operatorname(X\_1,X\_i)\backslash operatorname(\backslash partial\_f(X\_1,\backslash ldots,X\_n)).$$where

(X _{1,...} X _{n} ) is a zero-mean multivariate normal random vector.

Suppose X is in an exponential family, that is, X has the density

f _{η(x)=\exp(η'T(x)} - \Psi( η ))h(x).

Suppose this density has support

(a,b) where a,b could be -inf ty , inf ty and as x → a or b , \exp ( η 'T(x))h(x) g(x) → 0 where g is any differentiable function such that E|g'(X)|< inf ty or \exp ( η 'T(x))h(x) → 0 if a,b finite. Then E\left[\left( (h'(X))/(h(X)) + \sum η _{i} T _{i'(X)\right) ⋅} g(X)\right] = - E[g'(X)].

The derivation is same as the special case, namely, integration by parts.

If we only know

X has support R , then it could be the case that E|g(X)| < inf ty and E|g'(X)| < inf ty but \lim _{x →} f _{η(x)} g(x) \not = 0 . To see this, simply put g(x) =1 and f _{η(x)} with infinitely spikes towards infinity but still integrable. One such example could be adapted from f(x) = \begin{cases} 1 & x \in [n, n + 2 ^{-n} ) \ 0 & otherwise \end{cases} so that f is smooth.

Extensions to elliptically-contoured distributions also exist.^{[4] [5] [6]}

See also

• Stein's method
• Taylor expansions for the moments of functions of random variables
• Stein discrepancy

References

1. Ingersoll, J., Theory of Financial Decision Making, Rowman and Littlefield, 1987: 13-14.
2. Book: Information Theory: Coding Theorems for Discrete Memoryless Systems. Imre. Csiszár. János. Körner. Cambridge University Press. 2011. 9781139499989. 14.
3. Book: Thomas M. Cover, Joy A. Thomas . Elements of Information Theory . John Wiley & Sons, New York . 2006. 9781118585771 .
4. Cellier . Dominique . Fourdrinier . Dominique . Robert . Christian . 1989 . Robust shrinkage estimators of the location parameter for elliptically symmetric distributions . Journal of Multivariate Analysis . 29 . 1 . 39 - 52 . 10.1016/0047-259X(89)90075-4 .
5. Hamada . Mahmoud . Valdez . Emiliano A. . 2008 . CAPM and option pricing with elliptically contoured distributions . The Journal of Risk & Insurance . 75 . 2 . 387 - 409 . 10.1111/j.1539-6975.2008.00265.x. 10.1.1.573.4715 .
6. Landsman . Zinoviy . Nešlehová . Johanna. Johanna G. Nešlehová . 2008 . Stein's Lemma for elliptical random vectors . Journal of Multivariate Analysis . 99 . 5 . 912 - –927 . 10.1016/j.jmva.2007.05.006. free.