Compound probability distribution explained

In probability and statistics, a compound probability distribution (also known as a mixture distribution or contagious distribution) is the probability distribution that results from assuming that a random variable is distributed according to some parametrized distribution, with (some of) the parameters of that distribution themselves being random variables.If the parameter is a scale parameter, the resulting mixture is also called a scale mixture.

The compound distribution ("unconditional distribution") is the result of marginalizing (integrating) over the latent random variable(s) representing the parameter(s) of the parametrized distribution ("conditional distribution").

Definition

A compound probability distribution is the probability distribution that results from assuming that a random variable

is distributed according to some parametrized distribution

with an unknown parameter

\theta

that is again distributed according to some other distribution

. The resulting distribution

is said to be the distribution that results from compounding

with

. The parameter's distribution

is also called the mixing distribution or latent distribution. Technically, the unconditional distribution

results from marginalizing over

, i.e., from integrating out the unknown parameter(s)

\theta

. Its probability density function is given by:

p_H(x)={\displaystyle\int\limitsp_F(x|\theta)p_G(\theta)\operatorname{d}\theta}

The same formula applies analogously if some or all of the variables are vectors.

From the above formula, one can see that a compound distribution essentially is a special case of a marginal distribution: The joint distribution of

and

\theta

is given by

p(x,\theta)=p(x|\theta)p(\theta)

, and the compound results as its marginal distribution:

{stylep(x)=\intp(x,\theta)\operatorname{d}\theta}

.If the domain of

\theta

is discrete, then the distribution is again a special case of a mixture distribution.

Properties

General

The compound distribution

will depend on the specific expression of each distribution, as well as which parameter of

is distributed according to the distribution

, and the parameters of

will include any parameters of

that are not marginalized, or integrated, out.The support of

is the same as that of

, and if the latter is a two-parameter distribution parameterized with the mean and variance, some general properties exist.

Mean and variance

The compound distribution's first two moments are given by the law of total expectation and the law of total variance:

$\operatorname_H[X] = \operatorname_G\bigl[\operatorname{E}_F[X|\theta]\bigr]$

$\operatorname_H(X) = \operatorname_G\bigl[\operatorname{Var}_F(X|\theta)\bigr] + \operatorname_G\bigl(\operatorname_F[X|\theta]\bigr)$

If the mean of

is distributed as

, which in turn has mean

\mu

and variance

\sigma²

the expressions above imply

\operatorname{E}_H[X]=\operatorname{E}_G[\theta]=\mu

and

\operatorname{Var}_H(X)=\operatorname{Var}_F(X|\theta)+\operatorname{Var}_G(Y)=\tau²+\sigma²

, where

\tau²

is the variance of

Proof

let

and

be probability distributions parameterized with mean a variance as

\beginx &\sim \mathcal(\theta,\tau^2) \\\theta &\sim \mathcal(\mu,\sigma^2)\end

then denoting the probability density functions as

f(x|\theta)=p_F(x|\theta)

and

g(\theta)=p_G(\theta)

respectively, and

h(x)

being the probability density of

we have

\begin\operatorname_H[X] = \int_F x h(x)dx &= \int_F x \int_G f(x|\theta) g(\theta) d\theta dx \\ &= \int_G \int_F x f(x|\theta) dx\ g(\theta) d\theta \\ &= \int_G \operatorname_F[X|\theta] g(\theta) d\theta\end

and we have from the parameterization

l{F}

and

l{G}

that

\begin\operatorname_F[X|\theta] &= \int_F x f(x|\theta)dx = \theta \\\operatorname_G[\theta] &= \int_G \theta g(\theta)d\theta = \mu\end

and therefore the mean of the compound distribution

\operatorname{E}_H[X]=\mu

as per the expression for its first moment above.

The variance of

is given by

	2]
\operatorname{E}
	H[X

	2
(\operatorname{E}
	H[X])

, and

\begin\operatorname_H[X^2] = \int_F x^2 h(x)dx &= \int_F x^2 \int_G f(x|\theta) g(\theta) d\theta dx \\ &= \int_G g(\theta)\int_F x^2 f(x|\theta) dx\ d\theta \\ &= \int_G g(\theta)(\tau^2+\theta^2)d\theta\\ &= \tau^2\int_G g(\theta)d\theta+\int_Gg(\theta)\theta^2d\theta\\ &= \tau^2+(\sigma^2+\mu^2),\end

given the fact that

\int_Fx²f(x\mid\theta)

	2\mid
dx=\operatorname{E}
	F[X

\theta]=\operatorname{Var}_{F(X\mid\theta)+(\operatorname{E}}_F[X\mid\theta])²

and

\int_G\theta²

	2
g(\theta)d\theta=\operatorname{E}
	G[\theta

]=\operatorname{Var}_G(\theta)+

	2
(\operatorname{E}
	G[\theta])

. Finally we get

\begin\operatorname_H(X) &= \operatorname_H[X^2] - (\operatorname_H[X])^2 \\ &= \tau^2 + \sigma^2\end

Applications

Testing

Distributions of common test statistics result as compound distributions under their null hypothesis, for example in Student's t-test (where the test statistic results as the ratio of a normal and a chi-squared random variable), or in the F-test (where the test statistic is the ratio of two chi-squared random variables).

Overdispersion modeling

Compound distributions are useful for modeling outcomes exhibiting overdispersion, i.e., a greater amount of variability than would be expected under a certain model. For example, count data are commonly modeled using the Poisson distribution, whose variance is equal to its mean. The distribution may be generalized by allowing for variability in its rate parameter, implemented via a gamma distribution, which results in a marginal negative binomial distribution. This distribution is similar in its shape to the Poisson distribution, but it allows for larger variances. Similarly, a binomial distribution may be generalized to allow for additional variability by compounding it with a beta distribution for its success probability parameter, which results in a beta-binomial distribution.

Bayesian inference

Besides ubiquitous marginal distributions that may be seen as special cases of compound distributions, in Bayesian inference, compound distributions arise when, in the notation above, F represents the distribution of future observations and G is the posterior distribution of the parameters of F, given the information in a set of observed data. This gives a posterior predictive distribution. Correspondingly, for the prior predictive distribution, F is the distribution of a new data point while G is the prior distribution of the parameters.

Convolution

Convolution of probability distributions (to derive the probability distribution of sums of random variables) may also be seen as a special case of compounding; here the sum's distribution essentially results from considering one summand as a random location parameter for the other summand.^[1]

Computation

Compound distributions derived from exponential family distributions often have a closed form.If analytical integration is not possible, numerical methods may be necessary.

Compound distributions may relatively easily be investigated using Monte Carlo methods, i.e., by generating random samples. It is often easy to generate random numbers from the distributions

p(\theta)

as well as

p(x|\theta)

and then utilize these to perform collapsed Gibbs sampling to generate samples from

p(x)

A compound distribution may usually also be approximated to a sufficient degree by a mixture distribution using a finite number of mixture components, allowing to derive approximate density, distribution function etc.

Parameter estimation (maximum-likelihood or maximum-a-posteriori estimation) within a compound distribution model may sometimes be simplified by utilizing the EM-algorithm.^[2]

Examples

Gaussian scale mixtures:^[3] ^[4]
- Compounding a normal distribution with variance distributed according to an inverse gamma distribution (or equivalently, with precision distributed as a gamma distribution) yields a non-standardized Student's t-distribution.^[5] This distribution has the same symmetrical shape as a normal distribution with the same central point, but has greater variance and heavy tails.
- Compounding a Gaussian (or normal) distribution with variance distributed according to an exponential distribution (or with standard deviation according to a Rayleigh distribution) yields a Laplace distribution. More generally, compounding a Gaussian (or normal) distribution with variance distributed according to a gamma distribution yields a variance-gamma distribution.
- Compounding a Gaussian distribution with variance distributed according to an exponential distribution whose rate parameter is itself distributed according to a gamma distribution yields a Normal-exponential-gamma distribution. (This involves two compounding stages. The variance itself then follows a Lomax distribution; see below.)
- Compounding a Gaussian distribution with standard deviation distributed according to a (standard) inverse uniform distribution yields a Slash distribution.
- Compounding a Gaussian (normal) distribution with a Kolmogorov distribution yields a logistic distribution.
other Gaussian mixtures:
- Compounding a Gaussian distribution with mean distributed according to another Gaussian distribution yields (again) a Gaussian distribution.
- Compounding a Gaussian distribution with mean distributed according to a shifted exponential distribution yields an exponentially modified Gaussian distribution.
Compounding a Bernoulli distribution with probability of success

distributed according to a distribution

that has a defined expected value yields a Bernoulli distribution with success probability

E[X]

. An interesting consequence is that the dispersion of

does not influence the dispersion of the resulting compound distribution.

Compounding a binomial distribution with probability of success distributed according to a beta distribution yields a beta-binomial distribution. It possesses three parameters, a parameter

(number of samples) from the binomial distribution and shape parameters

\alpha

and

\beta

from the beta distribution.^[6] ^[7]

Compounding a multinomial distribution with probability vector distributed according to a Dirichlet distribution yields a Dirichlet-multinomial distribution.
Compounding a Poisson distribution with rate parameter distributed according to a gamma distribution yields a negative binomial distribution.^[8] ^[9]
Compounding a Poisson distribution with rate parameter distributed according to a exponential distribution yields a geometric distribution.
Compounding an exponential distribution with its rate parameter distributed according to a gamma distribution yields a Lomax distribution.^[10]
Compounding a gamma distribution with inverse scale parameter distributed according to another gamma distribution yields a three-parameter beta prime distribution.^[11]
Compounding a half-normal distribution with its scale parameter distributed according to a Rayleigh distribution yields an exponential distribution. This follows immediately from the Laplace distribution resulting as a normal scale mixture; see above. The roles of conditional and mixing distributions may also be exchanged here; consequently, compounding a Rayleigh distribution with its scale parameter distributed according to a half-normal distribution also yields an exponential distribution.
A Gamma(k=2,θ) - distributed random variable whose scale parameter θ again is uniformly distributed marginally yields an exponential distribution.

Similar terms

The notion of "compound distribution" as used e.g. in the definition of a Compound Poisson distribution or Compound Poisson process is different from the definition found in this article. The meaning in this article corresponds to what is used in e.g. Bayesian hierarchical modeling.

The special case for compound probability distributions where the parametrized distribution

is the Poisson distribution is also called mixed Poisson distribution.

Notes and References

Röver. C.. Friede. T.. Discrete approximation of a mixture distribution via restricted divergence. Journal of Computational and Graphical Statistics. 26. 1. 2017. 217–222. 10.1080/10618600.2016.1276840. free. 1602.04060.
Book: Gelman. A.. Carlin. J. B.. Stern. H.. Rubin. D. B.. Bayesian Data Analysis. 1st. Chapman & Hall / CRC. Boca Raton. 1997. 9.5 Finding marginal posterior modes using EM and related algorithms. 276.
Lee. S.X.. McLachlan. G.J.. Scale mixture distribution. Wiley StatsRef: Statistics Reference Online. 2019. 10.1002/9781118445112.stat08201.
Gneiting. T.. Normal scale mixtures and dual probability densities. Journal of Statistical Computation and Simulation. 59. 4. 1997. 375–384. 10.1080/00949659708811867.
Book: Mood. A. M.. Graybill. F. A.. Boes. D. C.. Introduction to the theory of statistics. 3rd. McGraw-Hill. New York. 1974.
Book: Johnson. N. L.. Kemp. A. W.. Kotz. S.. Univariate discrete distributions. 3rd. Wiley. New York. 2005. 6.2.2. 253.
Book: Gelman. A.. Carlin. J. B.. Stern. H.. Dunson. D. B.. Vehtari. A.. Rubin. D. B.. Bayesian Data Analysis. 3rd. Chapman & Hall / CRC. Boca Raton. 2014.
1987. Negative binomial and mixed Poisson regression. The Canadian Journal of Statistics. 15. 3. 209–225. Lawless. J.F.. 10.2307/3314912. 3314912.
Teich, M. C.. Diament, P.. Multiply stochastic representations for K distributions and their Poisson transforms. Journal of the Optical Society of America A. 1989. 6. 1. 80–91. 10.1364/JOSAA.6.000080. 1989JOSAA...6...80T. 10.1.1.64.596.
Book: Johnson. N. L.. Kotz. S.. Balakrishnan. N.. Continuous univariate distributions. 2nd. 1. Wiley. New York. 1994. 20 Pareto distributions. 573.
Dubey . S. D. . 10.1007/BF02613934 . Compound gamma, beta and F distributions . Metrika . 16 . 27–31 . 1970 .

Compound probability distribution explained

Definition

Properties

General

Mean and variance

Proof

Applications

Testing

Overdispersion modeling

Bayesian inference

Convolution

Computation

Examples

Similar terms

See also

Notes and References