Chapman–Robbins bound explained

In statistics, the Chapman–Robbins bound or Hammersley–Chapman–Robbins bound is a lower bound on the variance of estimators of a deterministic parameter. It is a generalization of the Cramér–Rao bound; compared to the Cramér–Rao bound, it is both tighter and applicable to a wider range of problems. However, it is usually more difficult to compute.

The bound was independently discovered by John Hammersley in 1950, and by Douglas Chapman and Herbert Robbins in 1951.

Statement

Let

\Theta be the set of parameters for a family of probability distributions \{\mu _{\theta} : \theta\in\Theta\} on \Omega .

For any two

\theta, \theta' \in \Theta , let \chi ^{2 (\mu} _{\theta'} ; \mu _{\theta} ) be the \chi ^{2} -divergence from \mu _{\theta} to \mu _{\theta'} . Then:

A generalization to the multivariable case is:

Proof

By the variational representation of chi-squared divergence:^[1] $$\backslash chi^2(P;\; Q)\; =\; \backslash sup\_g\; \backslash frac$$Plug in

g = \hat g, P = \mu _{\theta'} , Q = \mu _{\theta} , to obtain: $$\backslash chi^2(\backslash mu\_;\; \backslash mu\_\backslash theta)\; \backslash geq\; \backslash frac$$Switch the denominator and the left side and take supremum over \theta' to obtain the single-variate case. For the multivariate case, we define $h\; =\; \backslash sum\_^m\; v\_i\; \backslash hat\; g\_i$ for any v ≠ 0 \in \R ^{m} . Then plug in g = h in the variational representation to obtain: $$\backslash chi^2(\backslash mu\_;\; \backslash mu\_\backslash theta)\; \backslash geq\; \backslash frac\; =\; \backslash frac$$Take supremum over v ≠ 0 \in\R ^{m} , using the linear algebra fact that \sup _{v ≠} (v ^{T} ww ^{T} v)/(v ^{T} M v) = w ^{T} M ^{-1} w , we obtain the multivariate case.

Relation to Cramér–Rao bound

Usually,

\Omega = l X ^{n} is the sample space of n independent draws of a l X -valued random variable X with distribution λ _{\theta} from a by \theta \in \Theta \subseteq R ^{m} parameterized family of probability distributions, \mu _{\theta} = λ ^{⊗ n} _{\theta} is its n -fold product measure, and \hat g : l X ^{n} \to \Theta is an estimator of \theta . Then, for m =1 , the expression inside the supremum in the Chapman–Robbins bound converges to the Cramér–Rao bound of \hat g when \theta' \to \theta , assuming the regularity conditions of the Cramér–Rao bound hold. This implies that, when both bounds exist, the Chapman–Robbins version is always at least as tight as the Cramér–Rao bound; in many cases, it is substantially tighter.

The Chapman–Robbins bound also holds under much weaker regularity conditions. For example, no assumption is made regarding differentiability of the probability density function p(x; θ) of

λ _{\theta} . When p(x; θ) is non-differentiable, the Fisher information is not defined, and hence the Cramér–Rao bound does not exist.

See also

• Cramér–Rao bound
• Estimation theory

Notes and References

1. Web site: Polyanskiy . Yury . 2017 . Lecture notes on information theory, chapter 29, ECE563 (UIUC) . live . https://web.archive.org/web/20220524014051/https://people.lids.mit.edu/yp/homepage/data/LN_stats.pdf . 2022-05-24 . 2022-05-24 . Lecture notes on information theory.