Normal-inverse-Wishart distribution explained

In probability theory and statistics, the normal-inverse-Wishart distribution (or Gaussian-inverse-Wishart distribution) is a multivariate four-parameter family of continuous probability distributions. It is the conjugate prior of a multivariate normal distribution with unknown mean and covariance matrix (the inverse of the precision matrix).^[1]

Definition

Suppose

\boldsymbol\mu|\boldsymbol\mu _{0,λ,\boldsymbol\Sigma} \sim l{N}\left(\boldsymbol\mu | \boldsymbol\mu _{0 , (1)/(λ)} \boldsymbol\Sigma\right) has a multivariate normal distribution with mean \boldsymbol\mu _{0} and covariance matrix \tfrac{ 1 }{ λ }\boldsymbol\Sigma , where\boldsymbol\Sigma|\boldsymbol\Psi,\nu \sim l{W} ^{-1} (\boldsymbol\Sigma|\boldsymbol\Psi,\nu) has an inverse Wishart distribution. Then (\boldsymbol\mu,\boldsymbol\Sigma) has a normal-inverse-Wishart distribution, denoted as(\boldsymbol\mu,\boldsymbol\Sigma) \sim NIW (\boldsymbol\mu _{0,λ,\boldsymbol\Psi,\nu)} .

Characterization

Probability density function

f(\boldsymbol\mu,\boldsymbol\Sigma|\boldsymbol\mu _{0,λ,\boldsymbol\Psi,\nu)} = l{N}\left(\boldsymbol\mu | \boldsymbol\mu _{0 , (1)/(λ)} \boldsymbol\Sigma\right) l{W} ^{-1} (\boldsymbol\Sigma|\boldsymbol\Psi,\nu)

The full version of the PDF is as follows:^[2]

f(\boldsymbol{\mu},\boldsymbol{\Sigma} | \boldsymbol{\mu} _{0,λ,\boldsymbol{\Psi},\nu} ) = (λ ^{D/2} |\boldsymbol{\Psi)/(|) ^{\nu} |\boldsymbol{\Sigma}| ^{- (\nu + D +2)/(2)} }\text\left\

Here

\Gamma _{D[ ⋅ ]} is the multivariate gamma function and Tr(\boldsymbol{\Psi}) is the Trace of the given matrix.

Properties

Marginal distributions

By construction, the marginal distribution over

\boldsymbol\Sigma is an inverse Wishart distribution, and the conditional distribution over \boldsymbol\mu given \boldsymbol\Sigma is a multivariate normal distribution. The marginal distribution over \boldsymbol\mu is a multivariate t-distribution.

Posterior distribution of the parameters

Suppose the sampling density is a multivariate normal distribution

\boldsymbol{y _{i}|\boldsymbol\mu,\boldsymbol\Sigma} \sim l{N} _{p(\boldsymbol\mu,\boldsymbol\Sigma)}

where

\boldsymbol{y} is an n x p matrix and \boldsymbol{y _{i}} (of length p ) is row i of the matrix .

With the mean and covariance matrix of the sampling distribution is unknown, we can place a Normal-Inverse-Wishart prior on the mean and covariance parameters jointly

(\boldsymbol\mu,\boldsymbol\Sigma) \sim NIW (\boldsymbol\mu _{0,λ,\boldsymbol\Psi,\nu).}

The resulting posterior distribution for the mean and covariance matrix will also be a Normal-Inverse-Wishart

(\boldsymbol\mu,\boldsymbol\Sigma|y) \sim NIW (\boldsymbol\mu _{n,λ} _{n,\boldsymbol\Psi} _{n,\nu} _{n),}

where

\boldsymbol\mu _{n} = (λ \boldsymbol\mu _{0} + n \bar{\boldsymbol y)/() }λ _{n} = λ + n \nu _{n} = \nu + n \boldsymbol\Psi _{n} = \boldsymbol{\Psi + S} + (λ n)/(λ+ n) (\boldsymbol{\bar{y} - \mu _{0})(\boldsymbol{\bar{y}-\mu} ^{T ~~~ with ~~\boldsymbol{S} =} ^{0 })} \sum ^{n} _{i =1} (\boldsymbol{y _{i-\bar{y}})(\boldsymbol{y} ^{T} ^{i - \bar{y}})} .

To sample from the joint posterior of

(\boldsymbol\mu,\boldsymbol\Sigma) , one simply draws samples from \boldsymbol\Sigma|\boldsymbol y \sim l{W} ^{-1} (\boldsymbol\Psi _{n,\nu} _{n)} , then draw \boldsymbol\mu | \boldsymbol{\Sigma,y} \sim l{N} _{p(\boldsymbol\mu} _{n,\boldsymbol\Sigma/λ} _{n)} . To draw from the posterior predictive of a new observation, draw \boldsymbol\tilde{y}|\boldsymbol{\mu,\Sigma,y} \sim l{N} _{p(\boldsymbol\mu,\boldsymbol\Sigma)} , given the already drawn values of \boldsymbol\mu and \boldsymbol\Sigma .^[3]

Generating normal-inverse-Wishart random variates

Generation of random variates is straightforward:

1. Sample

\boldsymbol\Sigma from an inverse Wishart distribution with parameters \boldsymbol\Psi and \nu

1. Sample

\boldsymbol\mu from a multivariate normal distribution with mean \boldsymbol\mu _{0} and variance \boldsymbol \tfrac{ 1 }{ λ } \boldsymbol\Sigma

Related distributions

• The normal-Wishart distribution is essentially the same distribution parameterized by precision rather than variance. If

(\boldsymbol\mu,\boldsymbol\Sigma) \sim NIW (\boldsymbol\mu _{0,λ,\boldsymbol\Psi,\nu)} then (\boldsymbol\mu,\boldsymbol\Sigma ^{-1} ) \sim NW (\boldsymbol\mu ^{-1} _{0 , λ ,\boldsymbol\Psi} ,\nu) .

• The normal-inverse-gamma distribution is the one-dimensional equivalent.
• The multivariate normal distribution and inverse Wishart distribution are the component distributions out of which this distribution is made.

References

• Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media.
• Murphy, Kevin P. (2007). "Conjugate Bayesian analysis of the Gaussian distribution." http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf

Notes and References

1. Murphy, Kevin P. (2007). "Conjugate Bayesian analysis of the Gaussian distribution." http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
2. Simon J.D. Prince(June 2012). Computer Vision: Models, Learning, and Inference. Cambridge University Press. 3.8: "Normal inverse Wishart distribution".
3. Gelman, Andrew, et al. Bayesian data analysis. Vol. 2, p.73. Boca Raton, FL, USA: Chapman & Hall/CRC, 2014.