Laplace's approximation explained

Laplace's approximation provides an analytical expression for a posterior probability distribution by fitting a Gaussian distribution with a mean equal to the MAP solution and precision equal to the observed Fisher information.^{[1] [2]} The approximation is justified by the Bernstein–von Mises theorem, which states that, under regularity conditions, the error of the approximation tends to 0 as the number of data points tends to infinity.^{[3] [4]}

For example, consider a regression or classification model with data set

\{x _{n,y} _{n\}} _{n=1,\ldots,N} comprising inputs x and outputs y with (unknown) parameter vector \theta of length D . The likelihood is denoted p({\bf y}|{\bf x},\theta) and the parameter prior p(\theta) . Suppose one wants to approximate the joint density of outputs and parameters p({\bf y},\theta|{\bf x}) . Bayes' formula reads:p({\bf y},\theta|{\bf x}) = p({\bf y}|{\bf x},\theta)p(\theta|{\bf x}) = p({\bf y}|{\bf x})p(\theta|{\bf y},{\bf x}) \simeq \tilde q(\theta) = Zq(\theta). p({\bf y}|{\bf x}) and posterior p(\theta|{\bf y},{\bf x}) . Seen as a function of \theta the joint is an un-normalised density.

In Laplace's approximation, we approximate the joint by an un-normalised Gaussian

\tilde q(\theta) = Zq(\theta) , where we use q to denote approximate density, \tilde q for un-normalised density and Z the normalisation constant of \tilde q (independent of \theta ). Since the marginal likelihood p({\bf y}|{\bf x}) doesn't depend on the parameter \theta and the posterior p(\theta|{\bf y},{\bf x}) normalises over \theta we can immediately identify them with Z and q(\theta) of our approximation, respectively.

Laplace's approximation is

p({\bf y},\theta|{\bf x}) \simeq p({\bf y},\hat\theta|{\bf x})\exp ( - \tfrac{ 1 }{ 2 }(\theta - \hat\theta) ^{\top} S ^{-1} (\theta - \hat\theta) ) = \tilde q(\theta),

where we have defined

\begin{align} \hat\theta & = \operatorname{argmax} _{\theta} log p({\bf y},\theta|{\bf x}),\\ S ^{-1} & = - \left.\nabla _{\theta\nabla} _{\thetalog} p({\bf y},\theta|{\bf x})\right| _{\theta=\hat\theta} , \end{align}

where

\hat\theta is the location of a mode of the joint target density, also known as the maximum a posteriori or MAP point and S ^{-1} is the D x D positive definite matrix of second derivatives of the negative log joint target density at the mode \theta = \hat\theta . Thus, the Gaussian approximation matches the value and the log-curvature of the un-normalised target density at the mode. The value of \hat\theta is usually found using a gradient based method.

In summary, we have

\begin{align} q(\theta) & = {\cal N}(\theta|\mu = \hat\theta,\Sigma = S),\\ log Z & = log p({\bf y},\hat\theta|{\bf x}) + \tfrac{ 1 }{ 2 } log |S| + \tfrac{D}{ 2 } log ( 2 \pi), \end{align}

for the approximate posterior over

\theta and the approximate log marginal likelihood respectively.

The main weaknesses of Laplace's approximation are that it is symmetric around the mode and that it is very local: the entire approximation is derived from properties at a single point of the target density. Laplace's method is widely used and was pioneered in the context of neural networks by David MacKay,^[5] and for Gaussian processes by Williams and Barber.^[6]

Further reading

• Book: Amaral Turkman, M. Antónia . Carlos Daniel . Paulino . Peter . Müller . Computational Bayesian Statistics : An Introduction . The Classical Laplace Method . 154–159 . Cambridge . Cambridge University Press . 2019 . 978-1-108-48103-8 .
• Book: Tanner, Martin A. . Tools for Statistical Inference . New York . Springer . 1996 . Posterior Moments and Marginalization Based on Laplace's Method . 44–51 . 0-387-94688-8 .

Notes and References

1. Book: Robert E. . Kass . Luke . Tierney . Joseph B. . Kadane . Laplace’s method in Bayesian analysis . Statistical Multiple Integration . Contemporary Mathematics . 1991 . 115 . 89–100 . 0-8218-5122-5 . 10.1090/conm/115/07 .
2. Web site: Information Theory, Inference and Learning Algorithms, chapter 27: Laplace's method. David J. C.. MacKay. 2003.
3. Book: Hartigan, J. A. . John A. Hartigan

   . John A. Hartigan . Asymptotic Normality of Posterior Distributions . Bayes Theory . Springer Series in Statistics . New York . Springer . 1983 . 107–118 . 978-1-4613-8244-7. 10.1007/978-1-4613-8242-3_11 .

4. Book: Robert E. . Kass . Luke . Tierney . Joseph B. . Kadane . The Validity of Posterior Expansions Based on Laplace's Method . 473–488 . S. . Geisser . J. S. . Hodges . S. J. . Press . A. . Zellner . Bayesian and Likelihood Methods in Statistics and Econometrics . Elsevier . 1990 . 0-444-88376-2 .
5. MacKay . David J. C. . 1992 . Neural Computation . Bayesian Interpolation. MIT Press . 4 . 3 . 415–447 . 10.1162/neco.1992.4.3.415 . 1762283 .
6. Williams . Christopher K. I. . Barber . David . 1998 . PAMI . Bayesian classification with Gaussian Processes . IEEE . 20 . 12 . 1342–1351 . 10.1109/34.735807 .