Stein's unbiased risk estimate explained

In statistics, Stein's unbiased risk estimate (SURE) is an unbiased estimator of the mean-squared error of "a nearly arbitrary, nonlinear biased estimator." In other words, it provides an indication of the accuracy of a given estimator. This is important since the true mean-squared error of an estimator is a function of the unknown parameter to be estimated, and thus cannot be determined exactly. The technique is named after its discoverer, Charles Stein.^[1]

Formal statement

Let

\mu \in { R} ^{d} be an unknown parameter and let x \in { R} ^{d} be a measurement vector whose components are independent and distributed normally with mean \mu _{i,} i =1 ,...,d, and variance \sigma ^{2} . Suppose h(x) is an estimator of \mu from x , and can be written h(x) = x + g(x) , where g is weakly differentiable. Then, Stein's unbiased risk estimate is given by^[2] \operatorname{SURE}(h) = d\sigma ^{2} + \|g(x)\| ^{2} + 2 \sigma ^{2} \sum ^{d} _{i =1} (\partial)/(\partial x _{i}) g _{i(x)} = - d\sigma ^{2} + \|g(x)\| ^{2} + 2 \sigma ^{2} \sum ^{d} _{i =1} (\partial)/(\partial x _{i}) h _{i(x),} where g _{i(x)} is the i th component of the function g(x) , and \| ⋅ \| is the Euclidean norm.

The importance of SURE is that it is an unbiased estimate of the mean-squared error (or squared error risk) of

h(x) , i.e.\operatorname E _{\mu} \{ \operatorname{SURE}(h) \} = \operatorname{MSE}(h), with\operatorname{MSE}(h) = \operatorname E _{\mu} \|h(x) - \mu\| ^{2.}

Thus, minimizing SURE can act as a surrogate for minimizing the MSE. Note that there is no dependence on the unknown parameter

\mu in the expression for SURE above. Thus, it can be manipulated (e.g., to determine optimal estimation settings) without knowledge of \mu .

Proof

We wish to show that

\operatorname E _{\mu} \|h(x) - \mu\| ^{2} = \operatorname E _{\mu} \{ \operatorname{SURE}(h) \}. We start by expanding the MSE as\begin{align} \operatorname E _{\mu} \| h(x) - \mu\| ^{2} & = \operatorname E _{\mu} \|g(x) + x - \mu\| ^{2} \\ & = \operatorname E _{\mu} \|g(x)\| ^{2} + \operatorname E _{\mu} \|x - \mu\| ^{2} + 2 \operatorname E _{\mu} g(x) ^{T} (x - \mu) \\ & = \operatorname E _{\mu} \|g(x)\| ^{2} + d \sigma ^{2} + 2 \operatorname E _{\mu} g(x) ^{T(x} - \mu). \end{align} Now we use integration by parts to rewrite the last term:\begin{align} \operatorname E _{\mu} g(x) ^{T(x} - \mu) & = \int ^{d}} _{{ R} (1)/(\sqrt{ 2 \pi \sigma ^{2d}) } \exp\left(-\frac \right) \sum_^d g_i(x) (x_i - \mu_i) d^d x \\& = \sigma^2 \sum_^d\int_ \frac \exp\left(-\frac \right) \frac d^d x \\& = \sigma^2 \sum_^d \operatorname E_\mu \frac. \endSubstituting this into the expression for the MSE, we arrive at\operatorname E _{\mu} \|h(x) - \mu\| ^{2} = \operatorname E _{\mu} \left( d\sigma ^{2} + \|g(x)\| ^{2} + 2 \sigma ^{2} \sum ^{d} _{i =1} (dg _{i})/(dx _{i}) \right).

Applications

A standard application of SURE is to choose a parametric form for an estimator, and then optimize the values of the parameters to minimize the risk estimate. This technique has been applied in several settings. For example, a variant of the James–Stein estimator can be derived by finding the optimal shrinkage estimator. The technique has also been used by Donoho and Johnstone to determine the optimal shrinkage factor in a wavelet denoising setting.^[3]

Notes and References

1. Estimation of the Mean of a Multivariate Normal Distribution. The Annals of Statistics. Charles M.. Stein. 9. 6. 1135–1151. November 1981. 10.1214/aos/1176345632. 2240405. free.
2. Book: Wasserman, Larry. All of Nonparametric Statistics. 2005.
3. Adapting to Unknown Smoothness via Wavelet Shrinkage. Journal of the American Statistical Association. David L.. Donoho. David Donoho. Iain M. Johnstone . 90. 432. 1200–1244. December 1995. 10.2307/2291512. 2291512. 10.1.1.161.8697.