Kullback's inequality explained

In information theory and statistics, Kullback's inequality is a lower bound on the Kullback–Leibler divergence expressed in terms of the large deviations rate function.^[1] If P and Q are probability distributions on the real line, such that P is absolutely continuous with respect to Q, i.e. P << Q, and whose first moments exist, then$$D\_(P\backslash parallel\; Q)\; \backslash ge\; \backslash Psi\_Q^*(\backslash mu\text{'}\_1(P)),$$where

\Psi ^{*} _{Q} is the rate function, i.e. the convex conjugate of the cumulant-generating function, of Q , and \mu' _{1(P)} is the first moment of P.

The Cramér–Rao bound is a corollary of this result.

Proof

Let P and Q be probability distributions (measures) on the real line, whose first moments exist, and such that P << Q. Consider the natural exponential family of Q given by$$Q\_\backslash theta(A)\; =\; \backslash frac\; =\; \backslash frac\; \backslash int\_A\; e^Q(dx)$$for every measurable set A, where

M _{Q} is the moment-generating function of Q. (Note that Q₀ = Q.) Then$$D\_(P\backslash parallel\; Q)\; =\; D\_(P\backslash parallel\; Q\_\backslash theta)\; +\; \backslash int\_\backslash left(\backslash log\backslash frac\backslash right)\backslash mathrm\; dP.$$By Gibbs' inequality we have D _{KL} (P\parallel Q _{\theta)} \ge 0 so that$$D\_(P\backslash parallel\; Q)\; \backslash ge\; \backslash int\_\backslash left(\backslash log\backslash frac\backslash right)\backslash mathrm\; dP\; =\; \backslash int\_\backslash left(\backslash log\backslash frac\backslash right)\; P(dx)$$Simplifying the right side, we have, for every real θ where M _{Q(\theta)} < inf ty: $$D\_(P\backslash parallel\; Q)\; \backslash ge\; \backslash mu\text{'}\_1(P)\; \backslash theta\; -\; \backslash Psi\_Q(\backslash theta),$$where \mu' _{1(P)} is the first moment, or mean, of P, and \Psi _{Q} = log M _{Q} is called the cumulant-generating function. Taking the supremum completes the process of convex conjugation and yields the rate function:$$D\_(P\backslash parallel\; Q)\; \backslash ge\; \backslash sup\_\backslash theta\; \backslash left\backslash \; =\; \backslash Psi\_Q^*(\backslash mu\text{'}\_1(P)).$$

Corollary: the Cramér–Rao bound

See main article: Cramér–Rao bound.

Start with Kullback's inequality

Let X_θ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then$$\backslash lim\_\; \backslash frac\; \backslash ge\; \backslash lim\_\; \backslash frac,$$

where

\Psi ^{*} _{\theta} is the convex conjugate of the cumulant-generating function of X _{\theta} and \mu _{\theta+h} is the first moment of X _{\theta+h} .

Left side

The left side of this inequality can be simplified as follows:which is half the Fisher information of the parameter θ.

Right side

The right side of the inequality can be developed as follows:$$\backslash lim\_\; \backslash frac\; =\; \backslash lim\_\; \backslash frac\; 1\; .$$This supremum is attained at a value of t=τ where the first derivative of the cumulant-generating function is

\Psi' _{\theta(\tau)} = \mu _{\theta+h} , but we have \Psi' _{\theta(0)} = \mu _{\theta,} so that$$\backslash Psi$$_\theta(0) = \frac \lim_ \frac h \tau.Moreover,$$\backslash lim\_\; \backslash frac\; =\; \backslash frac\; 1\; \backslash left(\backslash frac\; \backslash right)^2\; =\; \backslash frac\; 1\; \backslash left(\backslash frac\; \backslash right)^2.$$

Putting both sides back together

We have:$$\backslash frac\; 1\; 2\; \backslash mathcal\; I\_X(\backslash theta)\; \backslash ge\; \backslash frac\; 1\; \backslash left(\backslash frac\; \backslash right)^2,$$which can be rearranged as:$$\backslash operatorname(X\_\backslash theta)\; \backslash ge\; \backslash frac\; .$$

See also

• Kullback–Leibler divergence
• Cramér–Rao bound
• Fisher information
• Large deviations theory
• Convex conjugate
• Rate function
• Moment-generating function

Notes and references

1. Aimé . Fuchs . Giorgio . Letta . L'inégalité de Kullback. Application à la théorie de l'estimation . Séminaire de Probabilités de Strasbourg . Séminaire de probabilités . Strasbourg . 4 . 108–131 . 1970 .