Law of the unconscious statistician explained

In probability theory and statistics, the law of the unconscious statistician, or LOTUS, is a theorem which expresses the expected value of a function of a random variable in terms of and the probability distribution of .

The form of the law depends on the type of random variable in question. If the distribution of is discrete and one knows its probability mass function, then the expected value of is$$\backslash operatorname[g(X)]\; =\; \backslash sum\_x\; g(x)\; p\_X(x),\; \backslash ,$$where the sum is over all possible values of . If instead the distribution of is continuous with probability density function, then the expected value of is$$\backslash operatorname[g(X)]\; =\; \backslash int\_^\backslash infty\; g(x)\; f\_X(x)\; \backslash ,\; \backslash mathrmx$$

Both of these special cases can be expressed in terms of the cumulative probability distribution function of, with the expected value of now given by the Lebesgue - Stieltjes integral$$\backslash operatorname[g(X)]\; =\; \backslash int\_^\backslash infty\; g(x)\; \backslash ,\; \backslash mathrmF\_X(x).$$

In even greater generality, could be a random element in any measurable space, in which case the law is given in terms of measure theory and the Lebesgue integral. In this setting, there is no need to restrict the context to probability measures, and the law becomes a general theorem of mathematical analysis on Lebesgue integration relative to a pushforward measure.

Etymology

This proposition is (sometimes) known as the law of the unconscious statistician because of a purported tendency to think of the identity as the very definition of the expected value, rather than (more formally) as a consequence of its true definition. The naming is sometimes attributed to Sheldon Ross' textbook Introduction to Probability Models, although he removed the reference in later editions. Many statistics textbooks do present the result as the definition of expected value.

Joint distributions

A similar property holds for joint distributions, or equivalently, for random vectors. For discrete random variables X and Y, a function of two variables g, and joint probability mass function

p _{X,} (x, y) :$$\backslash operatorname[g(X,\; Y)]\; =\; \backslash sum\_y\; \backslash sum\_x\; g(x,\; y)\; p\_(x,\; y)$$In the absolutely continuous case, with f _{X,} (x, y) being the joint probability density function,$$\backslash operatorname[g(X,\; Y)]\; =\; \backslash int\_^\backslash infty\; \backslash int\_^\backslash infty\; g(x,\; y)\; f\_(x,\; y)\; \backslash ,\; \backslash mathrmx\; \backslash ,\; \backslash mathrmy$$

Special cases

A number of special cases are given here. In the simplest case, where the random variable takes on countably many values (so that its distribution is discrete), the proof is particularly simple, and holds without modification if is a discrete random vector or even a discrete random element.

The case of a continuous random variable is more subtle, since the proof in generality requires subtle forms of the change-of-variables formula for integration. However, in the framework of measure theory, the discrete case generalizes straightforwardly to general (not necessarily discrete) random elements, and the case of a continuous random variable is then a special case by making use of the Radon–Nikodym theorem.

Discrete case

Suppose that is a random variable which takes on only finitely or countably many different values, with probabilities . Then for any function of these values, the random variable has values, although some of these may coincide with each other. For example, this is the case if can take on both values and and .

Let enumerate the possible distinct values of

g(X) , and for each let denote the collection of all with . Then, according to the definition of expected value, there is$$\backslash operatorname[g(X)]=\backslash sum\_i\; y\_i\; p\_(y\_i).$$

Since a

y _{i} can be the image of multiple, distinct x _{j} , it holds that$$p\_(y\_i)\; =\; \backslash sum\_\; p\_X(x\_j).$$

Then the expected value can be rewritten as$$\backslash sum\_i\; y\_i\; p\_(y\_i)\; =\; \backslash sum\_i\; y\_i\; \backslash sum\_\; p\_X(x\_j)\; =\; \backslash sum\_i\; \backslash sum\_\; g(x\_j)\; p\_X(x\_j)\; =\; \backslash sum\_x\; g(x)p\_X(x).$$This equality relates the average of the outputs of as weighted by the probabilities of the outputs themselves to the average of the outputs of as weighted by the probabilities of the outputs of .

If takes on only finitely many possible values, the above is fully rigorous. However, if takes on countably many values, the last equality given does not always hold, as seen by the Riemann series theorem. Because of this, it is necessary to assume the absolute convergence of the sums in question.

Continuous case

Suppose that is a random variable whose distribution has a continuous density . If is a general function, then the probability that is valued in a set of real numbers equals the probability that is valued in, which is given by$$\backslash int\_\; f(x)\backslash ,\backslash mathrmx.$$Under various conditions on, the change-of-variables formula for integration can be applied to relate this to an integral over, and hence to identify the density of in terms of the density of . In the simplest case, if is differentiable with nowhere-vanishing derivative, then the above integral can be written as$$\backslash int\_K\; f(g^(y))(g^)\text{'}(y)\backslash ,\backslash mathrmy,$$thereby identifying as possessing the density . The expected value of is then identified as$$\backslash int\_^\backslash infty\; yf(g^(y))(g^)\text{'}(y)\backslash ,\backslash mathrmy=\backslash int\_^\backslash infty\; g(x)f(x)\backslash ,\backslash mathrmx,$$where the equality follows by another use of the change-of-variables formula for integration. This shows that the expected value of is encoded entirely by the function and the density of .

The assumption that is differentiable with nonvanishing derivative, which is necessary for applying the usual change-of-variables formula, excludes many typical cases, such as . The result still holds true in these broader settings, although the proof requires more sophisticated results from mathematical analysis such as Sard's theorem and the coarea formula. In even greater generality, using the Lebesgue theory as below, it can be found that the identity$$\backslash operatorname[g(X)]=\backslash int\_^\backslash infty\; g(x)f(x)\backslash ,\backslash mathrmx$$holds true whenever has a density (which does not have to be continuous) and whenever is a measurable function for which has finite expected value. (Every continuous function is measurable.) Furthermore, without modification to the proof, this holds even if is a random vector (with density) and is a multivariable function; the integral is then taken over the multi-dimensional range of values of .

Measure-theoretic formulation

An abstract and general form of the result is available using the framework of measure theory and the Lebesgue integral. Here, the setting is that of a measure space and a measurable map from to a measurable space . The theorem then says that for any measurable function on which is valued in real numbers (or even the extended real number line), there is$$\backslash int\_\backslash Omega\; g\; \backslash circ\; X\; \backslash ,\; \backslash mathrm\backslash mu\; =\; \backslash int\_\; g\; \backslash ,\; \backslash mathrm(X\_\backslash sharp\; \backslash mu),$$(interpreted as saying, in particular, that either side of the equality exists if the other side exists). Here denotes the pushforward measure on . The 'discrete case' given above is the special case arising when takes on only countably many values and is a probability measure. In fact, the discrete case (although without the restriction to probability measures) is the first step in proving the general measure-theoretic formulation, as the general version follows therefrom by an application of the monotone convergence theorem. Without any major changes, the result can also be formulated in the setting of outer measures.

If is a σ-finite measure, the theory of the Radon–Nikodym derivative is applicable. In the special case that the measure is absolutely continuous relative to some background σ-finite measure on, there is a real-valued function on representing the Radon–Nikodym derivative of the two measures, and then$$\backslash int\_\; g\; \backslash ,\; \backslash mathrm(X\_\backslash sharp\; \backslash mu)=\backslash int\_gf\_X\backslash ,\backslash mathrm\backslash nu.$$In the further special case that is the real number line, as in the contexts discussed above, it is natural to take to be the Lebesgue measure, and this then recovers the 'continuous case' given above whenever is a probability measure. (In this special case, the condition of σ-finiteness is vacuous, since Lebesgue measure and every probability measure are trivially σ-finite.)

References

• Book: 2267655. Bogachev. V. I.. Measure theory. Volume I. Springer-Verlag. Berlin. 2007. 978-3-540-34513-8. Vladimir Bogachev. 1120.28001. 10.1007/978-3-540-34514-5.
• Book: Casella. George. Berger. Roger L.. Statistical inference. Duxbury Advanced Series. Duxbury. Pacific Grove, CA. 2001. Second edition of 1990 original. 0-534-11958-1. George Casella. Roger Lee Berger. 0699.62001.
• Book: 3098996. Cohn. Donald L.. Measure theory. Second edition of 1980 original. Birkhäuser Advanced Texts: Basler Lehrbücher. Birkhäuser/Springer. New York. 2013. 978-1-4614-6955-1. 10.1007/978-1-4614-6956-8. 1292.28002.
• Book: DeGroot. Morris H. . Schervish . Mark J. . Probability and statistics . . Fourth edition of 1975 original . 2014. 0373075. Morris H. DeGroot. 0619.62001. 0-321-50046-6.
• Book: Federer, Herbert. Herbert Federer. Geometric measure theory. Berlin–Heidelberg–New York. Springer-Verlag. Die Grundlehren der mathematischen Wissenschaften. 153. 1969. 978-3-540-60656-7. 0257325. 0176.00801 . 10.1007/978-3-642-62010-2.
• Book: Feller. William. An introduction to probability theory and its applications. Volume I. Third edition of 1950 original. John Wiley & Sons, Inc.. New York–London–Sydney. 1968. William Feller. 0228020. 0155.23101.
• Book: 0033869. Halmos. Paul R.. Measure theory. D. Van Nostrand Co., Inc.. New York. 1950. Paul Halmos. 10.1007/978-1-4684-9440-2. 0040.16802.
• Book: Papoulis. Athanasios. Pillai. S. Unnikrishna. Probability, random variables, and stochastic processes. Fourth edition of 1965 original. Athanasios Papoulis. Unnikrishna Pillai. 2002. McGraw-Hill. New York. 0-07-366011-6.
• Book: Ross. Sheldon M.. Introduction to probability models. Twelfth edition of 1972 original. Academic Press. London. 2019. 978-0-12-814346-9. 3931305. Sheldon M. Ross. 10.1016/C2017-0-01324-1. 1408.60002.