Law of total variance explained

In probability theory, the law of total variance^[1] or variance decomposition formula or conditional variance formulas or law of iterated variances also known as Eve's law,^[2] states that if

X and Y are random variables on the same probability space, and the variance of Y is finite, then

$$\backslash operatorname(Y)\; =\; \backslash operatorname[\backslash operatorname\{Var\}(Y\; \backslash mid\; X)]\; +\; \backslash operatorname(\backslash operatorname[Y\; \backslash mid\; X]).$$

In language perhaps better known to statisticians than to probability theorists, the two terms are the "unexplained" and the "explained" components of the variance respectively (cf. fraction of variance unexplained, explained variation). In actuarial science, specifically credibility theory, the first component is called the expected value of the process variance (EVPV) and the second is called the variance of the hypothetical means (VHM).^[3] These two components are also the source of the term "Eve's law", from the initials EV VE for "expectation of variance" and "variance of expectation".

Explanation

To understand the formula above, we need to comprehend the random variables

\operatorname{E}[Y|X] and \operatorname{Var}(Y|X) . These variables depend on the value of X : for a given x , \operatorname{E}[Y|X = x] and \operatorname{Var}(Y|X = x) are constant numbers. Essentially, we use the possible values of X to group the outcomes and then compute the expected values and variances for each group.

The "unexplained" component

\operatorname{E}(\operatorname{Var}[Y|X]) is simply the average of all the variances of Y within each group.The "explained" component \operatorname{Var}(\operatorname{E}[Y|X]) is the variance of the expected values, i.e., it represents the part of the variance that is explained by the variation of the average value of Y for each group.

For an illustration, consider the example of a dog show (a selected excerpt of Analysis_of_variance#Example). Let the random variable

Y correspond to the dog weight and X correspond to the breed. In this situation, it is reasonable to expect that the breed explains a major portion of the variance in weight since there is a big variance in the breeds' average weights. Of course, there is still some variance in weight for each breed, which is taken into account in the "unexplained" term.

Note that the "explained" term actually means "explained by the averages." If variances for each fixed

X (e.g., for each breed in the example above) are very distinct, those variances are still combined in the "unexplained" term.

Examples

Example 1

Five graduate students take an exam that is graded from 0 to 100. Let

Y denote the student's grade and X indicate whether the student is international or domestic. The data is summarized as follows:

Student	Y	X
1	20	International
2	30	International
3	100	International
4	40	Domestic
5	60	Domestic

Among international students, the mean is

\operatorname E[Y|X = International ] = 50 and the variance is \operatorname{Var}(Y|X = International ) = (3800)/(3) = 1266 .\overline{ 6 } .

Among domestic students, the mean is

\operatorname E[Y|X = Domestic ] = 50 and the variance is \operatorname{Var}(Y|X = Domestic ) = 100 .

X	P(X)	\operatorname E[Y|X]	\operatorname{Var}(Y	X)
International	3/5	50	1266.6
Domestic	2/5	50	100

The part of the variance of

Y "unexplained" by X is the mean of the variances for each group. In this case, it is \left( (3)/(\right)\left( 5) (3800)/(3) \right) + \left( (2)/(5) \right) ( 100 ) = 800 . The part of the variance of Y "explained" by X is the variance of the means of Y inside each group defined by the values of the X . In this case, it is zero, since the mean is the same for each group. So the total variation is\operatorname{Var}(Y) = \operatorname{E}[\operatorname{Var}(Y|X)] + \operatorname{Var}(\operatorname E[Y|X]) = 800 + 0 = 800 .

Example 2

Suppose is a coin flip with the probability of heads being . Suppose that when then is drawn from a normal distribution with mean and standard deviation, and that when then is drawn from normal distribution with mean and standard deviation . Then the first, "unexplained" term on the right-hand side of the above formula is the weighted average of the variances,, and the second, "explained" term is the variance of the distribution that gives with probability and gives with probability .

Formulation

There is a general variance decomposition formula for

c \geq 2 components (see below).^[4] For example, with two conditioning random variables:$$\backslash operatorname[Y]\; =\; \backslash operatorname\backslash left[\backslash operatorname\{Var\}\backslash left(Y\; \backslash mid\; X\_1,\; X\_2\backslash right)\backslash right]\; +\; \backslash operatorname[\backslash operatorname\{Var\}(\backslash operatorname\{E\}\backslash left[Y\; \backslash mid\; X\_1,\; X\_2\backslash right]\; \backslash mid\; X\_1)]\; +\; \backslash operatorname(\backslash operatorname\backslash left[Y\; \backslash mid\; X\_1\backslash right]),$$which follows from the law of total conditional variance:$$\backslash operatorname(Y\; \backslash mid\; X\_1)\; =\; \backslash operatorname\; \backslash left[\backslash operatorname\{Var\}(Y\; \backslash mid\; X\_1,\; X\_2)\; \backslash mid\; X\_1\backslash right]\; +\; \backslash operatorname\; \backslash left(\backslash operatorname\backslash left[Y\; \backslash mid\; X\_1,\; X\_2\; \backslash right]\; \backslash mid\; X\_1\backslash right).$$\operatorname{E}(Y \mid X) is a random variable in its own right, whose value depends on the value of X. Notice that the conditional expected value of Y given the X = x is a function of x (this is where adherence to the conventional and rigidly case-sensitive notation of probability theory becomes important!). If we write \operatorname{E}(Y \mid X = x) = g(x) then the random variable \operatorname{E}(Y \mid X) is just g(X). Similar comments apply to the conditional variance.

One special case, (similar to the law of total expectation) states that if

A _{1,} \ldots, A _{n} is a partition of the whole outcome space, that is, these events are mutually exclusive and exhaustive, then

In this formula, the first component is the expectation of the conditional variance; the other two components are the variance of the conditional expectation.

Proof

Finite Case

Let

(x _{1,} y _{1),} \ldots, (x _{n,} y _{n)} be observed values of (X,Y) , with repetitions.

Set

\bar{y} = \operatorname{E}[Y] and, for each possible value x of X , set \bar{y} _{x _{i}} = \operatorname{E}[Y | X = x _{i]} .

Note that

(y _{i} - \bar{y}) ^{2} = \left(y _{i} - \bar{y} _{x _{i}} + \bar{y} _{x _{i}} - \bar{y}\right) ^{2} = (y _{i} - \bar{y} _{x _{i}} ) ^{2} + (\bar{y} _{x _{i}} - \bar{y}) ^{2} + 2 (y _{i} - \bar{y} _{x _{i}} )(\bar{y} _{x _{i}} - \bar{y}) .

Summing these for

1 \leq i \leq n , the last parcel becomes\sum ^{n} _{i =1} 2 (y _{i} - \bar{y} _{x _{i}} )(\bar{y} _{x _{i}} - \bar{y}) = 2 \sum _{x} \left( \sum _{\{ 1 \leq i \leq n | x _{i} = x\} } (y_i - \bar_) \right) (\bar_ - \bar) = 2 \sum_ 0 \cdot (\bar_ - \bar) = 0.

Hence,

\operatorname{Var} (Y) = (1)/(n) \sum ^{n} _{i =1} (y _{i} - \bar{y}) ^{2} = (1)/(n) \sum ^{n} _{i =1} (y _{i} - \bar{y} _{x _{i}} ) ^{2} + (1)/(n) \sum ^{n} _{i =1} (\bar{y} _{x _{i}} - \bar{y}) ^{2} = \operatorname{E} [\operatorname{Var}(Y \mid X)] + \operatorname{Var} (\operatorname{E}[Y \mid X]).

General Case

The law of total variance can be proved using the law of total expectation.^[5] First,$$\backslash operatorname(Y)\; =\; \backslash operatorname\backslash left[Y^2\backslash right]\; -\; \backslash operatorname[Y]^2$$from the definition of variance. Again, from the definition of variance, and applying the law of total expectation, we have$$\backslash operatorname\backslash left[Y^2\backslash right]\; =\; \backslash operatorname\backslash left[\backslash operatorname\{E\}[Y^2\backslash mid\; X]\backslash right]\; =\; \backslash operatorname\; \backslash left[\backslash operatorname\{Var\}(Y\; \backslash mid\; X)\; +\; \backslash operatorname\{E\}[Y\; \backslash mid\; X]^2\backslash right].$$

Now we rewrite the conditional second moment of

Y in terms of its variance and first moment, and apply the law of total expectation on the right hand side:$$\backslash operatorname\backslash left[Y^2\backslash right]\; -\; \backslash operatorname[Y]^2\; =\; \backslash operatorname\; \backslash left[\backslash operatorname\{Var\}(Y\; \backslash mid\; X)\; +\; \backslash operatorname\{E\}[Y\; \backslash mid\; X]^2\backslash right]\; -\; \backslash operatorname\; [\backslash operatorname\{E\}[Y\; \backslash mid\; X]]^2.$$

Since the expectation of a sum is the sum of expectations, the terms can now be regrouped:$$=\; \backslash left(\backslash operatorname\; [\backslash operatorname\{Var\}(Y\; \backslash mid\; X)]\backslash right)\; +\; \backslash left(\backslash operatorname\; \backslash left[\backslash operatorname\{E\}[Y\; \backslash mid\; X]^2\backslash right]\; -\; \backslash operatorname\; [\backslash operatorname\{E\}[Y\; \backslash mid\; X]]^2\backslash right).$$

Finally, we recognize the terms in the second set of parentheses as the variance of the conditional expectation

\operatorname{E}[Y \mid X] :$$=\; \backslash operatorname\; [\backslash operatorname\{Var\}(Y\; \backslash mid\; X)]\; +\; \backslash operatorname\; (\backslash operatorname[Y\; \backslash mid\; X]).$$

General variance decomposition applicable to dynamic systems

The following formula shows how to apply the general, measure theoretic variance decomposition formula to stochastic dynamic systems. Let

Y(t) be the value of a system variable at time t. Suppose we have the internal histories (natural filtrations) H _{1t} ,H _{2t} ,\ldots,H _{c-1,t} , each one corresponding to the history (trajectory) of a different collection of system variables. The collections need not be disjoint. The variance of Y(t) can be decomposed, for all times t, into c \geq 2 components as follows:

The decomposition is not unique. It depends on the order of the conditioning in the sequential decomposition.

The square of the correlation and explained (or informational) variation

In cases where

(Y, X) are such that the conditional expected value is linear; that is, in cases where$$\backslash operatorname(Y\; \backslash mid\; X)\; =\; a\; X\; +\; b,$$it follows from the bilinearity of covariance that $$a=$$and$$b\; =\; \backslash operatorname(Y)-\; \backslash operatorname(X)$$and the explained component of the variance divided by the total variance is just the square of the correlation between Y and X; that is, in such cases,$$=\; \backslash operatorname(X,\; Y)^2.$$

One example of this situation is when

(X, Y) have a bivariate normal (Gaussian) distribution.

More generally, when the conditional expectation

\operatorname{E}(Y \mid X) is a non-linear function of X $$\backslash iota\_\; =\; =\; \backslash operatorname(\backslash operatorname(Y\; \backslash mid\; X),\; Y)^2,$$which can be estimated as the R squared from a non-linear regression of Y on X, using data drawn from the joint distribution of (X, Y). When \operatorname{E}(Y \mid X) has a Gaussian distribution (and is an invertible function of X ), or Y itself has a (marginal) Gaussian distribution, this explained component of variation sets a lower bound on the mutual information:$$\backslash operatorname(Y;\; X)\; \backslash geq\; \backslash ln\; \backslash left([1\; -\; \backslash iota\_\{Y\; \backslash mid\; X\}]^\backslash right).$$

Higher moments

\mu _{3} says$$\backslash mu\_3(Y)=\backslash operatorname\backslash left(\backslash mu\_3(Y\; \backslash mid\; X)\backslash right)\; +\; \backslash mu\_3(\backslash operatorname(Y\; \backslash mid\; X))\; +\; 3\backslash operatorname(\backslash operatorname(Y\; \backslash mid\; X),\; \backslash operatorname(Y\; \backslash mid\; X)).$$

For higher cumulants, a generalization exists. See law of total cumulance.

See also

• − a generalization

References

• Web site: Blitzstein. Joe. Stat 110 Final Review (Eve's Law). stat110.net. Harvard University, Department of Statistics. 9 July 2014.
• Book: Billingsley, Patrick . Probability and Measure . John Wiley & Sons, Inc. . New York, NY . 1995 . 0-471-00710-2. (Problem 34.10(b))

Notes and References

1. Neil A. Weiss, A Course in Probability, Addison - Wesley, 2005, pages 385 - 386.
2. Joseph K. Blitzstein and Jessica Hwang: "Introduction to Probability"
3. Book: Mahler. Howard C. . Dean. Curtis Gary. 2001 . Chapter 8: Credibility . http://people.stat.sfu.ca/~cltsai/ACMA315/Ch8_Credibility.pdf . . Foundations of Casualty Actuarial Science. 4th . . 525–526 . 978-0-96247-622-8. June 25, 2015.
4. Bowsher, C.G. and P.S. Swain, Identifying sources of variation and the flow of information in biochemical networks, PNAS May 15, 2012 109 (20) E1320-E1328.
5. Neil A. Weiss, A Course in Probability, Addison - Wesley, 2005, pages 380 - 383.