Evidence lower bound explained

In variational Bayesian methods, the evidence lower bound (often abbreviated ELBO, also sometimes called the variational lower bound^[1] or negative variational free energy) is a useful lower bound on the log-likelihood of some observed data.

The ELBO is useful because it provides a guarantee on the worst-case for the log-likelihood of some distribution (e.g.

p(X) ) which models a set of data. The actual log-likelihood may be higher (indicating an even better fit to the distribution) because the ELBO includes a Kullback-Leibler divergence (KL divergence) term which decreases the ELBO due to an internal part of the model being inaccurate despite good fit of the model overall. Thus improving the ELBO score indicates either improving the likelihood of the model p(X) or the fit of a component internal to the model, or both, and the ELBO score makes a good loss function, e.g., for training a deep neural network to improve both the model overall and the internal component. (The internal component is q _{\phi( ⋅} | x) , defined in detail later in this article.)

Definition

Let

X and Z be random variables, jointly distributed with distribution p _{\theta} . For example, p _{\theta(} X) is the marginal distribution of X , and p _{\theta(} Z \mid X) is the conditional distribution of Z given X . Then, for a sample x\sim p _{data} , and any distribution q _{\phi} , the ELBO is defined as$$L(\backslash phi,\; \backslash theta;\; x)\; :=\; \backslash mathbb\; E\_\; \backslash left[\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}\; \backslash right]\; .$$The ELBO can equivalently be written as^[2]

In the first line,

H[ q _{\phi(z|x)} ] is the entropy of q _{\phi} , which relates the ELBO to the Helmholtz free energy.^[3] In the second line, ln p _{\theta(x)} is called the evidence for x , and D _{KL} ( q _{\phi(z|x)} || p _{\theta(z|x)} ) is the Kullback-Leibler divergence between q _{\phi} and p _{\theta} . Since the Kullback-Leibler divergence is non-negative, L(\phi, \theta; x) forms a lower bound on the evidence (ELBO inequality)$$\backslash ln\; p\_\backslash theta(x)\; \backslash ge\; \backslash mathbb\; \backslash mathbb\; E\_\backslash left[\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z\backslash vert\; x)\}\; \backslash right].$$

Motivation

Variational Bayesian inference

Suppose we have an observable random variable

X , and we want to find its true distribution p ^{*} . This would allow us to generate data by sampling, and estimate probabilities of future events. In general, it is impossible to find p ^{*} exactly, forcing us to search for a good approximation.

That is, we define a sufficiently large parametric family

\{p _{\theta\}} _{\theta\in\Theta} of distributions, then solve for min _{\theta} L(p _{\theta,} p ^{*)} for some loss function L . One possible way to solve this is by considering small variation from p _{\theta} to p _{\theta} , and solve for L(p _{\theta,} p ^{*)} - L(p _{\theta+\delta} , p ^{*)} =0 . This is a problem in the calculus of variations, thus it is called the variational method.

Since there are not many explicitly parametrized distribution families (all the classical distribution families, such as the normal distribution, the Gumbel distribution, etc, are far too simplistic to model the true distribution), we consider implicitly parametrized probability distributions:

• First, define a simple distribution

p(z) over a latent random variable Z . Usually a normal distribution or a uniform distribution suffices.

• Next, define a family of complicated functions

f _{\theta} (such as a deep neural network) parametrized by \theta .

• Finally, define a way to convert any

f _{\theta(z)} into a simple distribution over the observable random variable X . For example, let f _{\theta(z)} = (f _{1(z),} f _{2(z))} have two outputs, then we can define the corresponding distribution over X to be the normal distribution l N(f _{1(z),} e ^{f _{2(z)}} ) .

This defines a family of joint distributions

p _{\theta} over (X, Z) . It is very easy to sample (x, z) \sim p _{\theta} : simply sample z\sim p , then compute f _{\theta(z)} , and finally sample x \sim p _{\theta( ⋅} | z) using f _{\theta(z)} .

In other words, we have a generative model for both the observable and the latent.Now, we consider a distribution

p _{\theta} good, if it is a close approximation of p ^{*} :$$p\_\backslash theta(X)\; \backslash approx\; p^*(X)$$since the distribution on the right side is over X only, the distribution on the left side must marginalize the latent variable Z away.
In general, it's impossible to perform the integral p _{\theta(x)} = \int p _{\theta(x|z)p(z)dz} , forcing us to perform another approximation.

Since

p _{\theta(x)} = (p _{\theta(x|z)p(z)})/(p _{\theta(z|x)}) (Bayes' Rule), it suffices to find a good approximation of p _{\theta(z|x)} . So define another distribution family q _{\phi(z|x)} and use it to approximate p _{\theta(z|x)} . This is a discriminative model for the latent.

The entire situation is summarized in the following table:

!X : observable!X, Z !Z : latent

p ^{*(x) ≈} p _{\theta(x)} ≈ (p _{\theta(x|z)p(z)})/(q _{\phi(z|x)}) approximable		p(z) , easy
	p _{\theta(x}	z)p(z), easy
p _{\theta(z}	x) \approx q_\phi(z	x) approximable		p _{\theta(x}	z), easy

In Bayesian language, X is the observed evidence, and Z is the latent/unobserved. The distribution p over Z is the prior distribution over Z , p _{\theta(x|z)} is the likelihood function, and p _{\theta(z|x)} is the posterior distribution over Z .

Given an observation

x , we can infer what z likely gave rise to x by computing p _{\theta(z|x)} . The usual Bayesian method is to estimate the integral p _{\theta(x)} = \int p _{\theta(x|z)p(z)dz} , then compute by Bayes' rule p _{\theta(z|x)} = (p _{\theta(x|z)p(z)})/(p _{\theta(x)}) . This is expensive to perform in general, but if we can simply find a good approximation q _{\phi(z|x)} ≈ p _{\theta(z|x)} for most x, z , then we can infer z from x cheaply. Thus, the search for a good q _{\phi} is also called amortized inference.

All in all, we have found a problem of variational Bayesian inference.

Deriving the ELBO

A basic result in variational inference is that minimizing the Kullback–Leibler divergence (KL-divergence) is equivalent to maximizing the log-likelihood:$$\backslash mathbb\_[\backslash ln\; p\_\backslash theta\; (x)]\; =\; -H(p^*)\; -\; D\_(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$where

H(p ^{*)} = - E _{x\sim p ^{*}} [ ln p ^{*(x)]} is the entropy of the true distribution. So if we can maximize E _{x\sim p ^{*(x)}} [ ln p _{\theta} (x)] , we can minimize D _{KL} (p ^{*(x)} \| p _{\theta(x))} , and consequently find an accurate approximation p _{\theta} ≈ p ^{*} .

To maximize

E _{x\sim p ^{*(x)}} [ ln p _{\theta} (x)] , we simply sample many x _{i\sim} p ^{*(x)} , i.e. use importance sampling$$N\backslash max\_\backslash theta\; \backslash mathbb\_[\backslash ln\; p\_\backslash theta\; (x)]\backslash approx\; \backslash max\_\backslash theta\; \backslash sum\_i\; \backslash ln\; p\_\backslash theta\; (x\_i)$$where N is the number of samples drawn from the true distribution. This approximation can be seen as overfitting.

In order to maximize

\sum _{i} ln p _{\theta} (x _{i)} , it's necessary to find ln p _{\theta(x)} :$$\backslash ln\; p\_\backslash theta(x)\; =\; \backslash ln\; \backslash int\; p\_\backslash theta(x|z)\; p(z)dz$$This usually has no closed form and must be estimated. The usual way to estimate integrals is Monte Carlo integration with importance sampling:$$\backslash int\; p\_\backslash theta(x|z)\; p(z)dz\; =\; \backslash mathbb\; E\_\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$where q _{\phi(z|x)} is a sampling distribution over z that we use to perform the Monte Carlo integration.

So we see that if we sample

z\sim q _{\phi( ⋅ |x)} , then (p _{\theta} (x, z))/(q _{\phi(z|x)}) is an unbiased estimator of p _{\theta(x)} . Unfortunately, this does not give us an unbiased estimator of ln p _{\theta(x)} , because ln is nonlinear. Indeed, we have by Jensen's inequality, $$\backslash ln\; p\_\backslash theta(x)=\; \backslash ln\; \backslash mathbb\; E\_\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; \backslash geq\; \backslash mathbb\; E\_\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$In fact, all the obvious estimators of ln p _{\theta(x)} are biased downwards, because no matter how many samples of z _{i\sim} q _{\phi( ⋅} | x) we take, we have by Jensen's inequality:$$\backslash mathbb\; E\_\backslash left[\backslash ln\; \backslash left(\backslash frac\; 1N\; \backslash sum\_i\; \backslash frac\{p\_\backslash theta\; (x,\; z\_i)\}\{q\_\backslash phi(z\_i|x)\}\backslash right)\; \backslash right]\; \backslash leq\; \backslash ln\; \backslash mathbb\; E\_\backslash left[\backslash frac\; 1N\; \backslash sum\_i\; \backslash frac\{p\_\backslash theta\; (x,\; z\_i)\}\{q\_\backslash phi(z\_i|x)\}\; \backslash right]\; =\; \backslash ln\; p\_\backslash theta(x)$$Subtracting the right side, we see that the problem comes down to a biased estimator of zero:$$\backslash mathbb\; E\_\backslash left[\backslash ln\; \backslash left(\backslash frac\; 1N\; \backslash sum\_i\; \backslash frac\{p\_\backslash theta\; (z\_i|x)\}\{q\_\backslash phi(z\_i|x)\}\backslash right)\; \backslash right]\; \backslash leq\; 0$$At this point, we could branch off towards the development of an importance-weighted autoencoder, but we will instead continue with the simplest case with N =1 :$$\backslash ln\; p\_\backslash theta(x)=\; \backslash ln\; \backslash mathbb\; E\_\backslash left[\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; \backslash geq\; \backslash mathbb\; E\_\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]$$The tightness of the inequality has a closed form:$$\backslash ln\; p\_\backslash theta(x)-\; \backslash mathbb\; E\_\backslash left[\backslash ln\backslash frac\{p\_\backslash theta\; (x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; =\; D\_(q\_\backslash phi(\backslash cdot\; |\; x)\backslash |\; p\_\backslash theta(\backslash cdot\; |\; x))\backslash geq\; 0$$We have thus obtained the ELBO function:$$L(\backslash phi,\; \backslash theta;\; x)\; :=\; \backslash ln\; p\_\backslash theta(x)\; -\; D\_(q\_\backslash phi(\backslash cdot\; |\; x)\backslash |\; p\_\backslash theta(\backslash cdot\; |\; x))$$

Maximizing the ELBO

For fixed

x , the optimization max _{\theta,} L(\phi, \theta; x) simultaneously attempts to maximize ln p _{\theta(x)} and minimize D _{KL} (q _{\phi( ⋅} | x)\| p _{\theta( ⋅} | x)) . If the parametrization for p _{\theta} and q _{\phi} are flexible enough, we would obtain some \hat\phi, \hat \theta , such that we have simultaneously

$$\backslash ln\; p\_(x)\; \backslash approx\; \backslash max\_\backslash theta\; \backslash ln\; p\_\backslash theta(x);\; \backslash quad\; q\_(\backslash cdot\; |\; x)\backslash approx\; p\_(\backslash cdot\; |\; x)$$Since$$\backslash mathbb\_[\backslash ln\; p\_\backslash theta\; (x)]\; =\; -H(p^*)\; -\; D\_(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$we have$$\backslash ln\; p\_(x)\; \backslash approx\; \backslash max\_\backslash theta\; -H(p^*)\; -\; D\_(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$and so$$\backslash hat\backslash theta\; \backslash approx\; \backslash arg\backslash min\; D\_(p^*(x)\; \backslash |\; p\_\backslash theta(x))$$In other words, maximizing the ELBO would simultaneously allow us to obtain an accurate generative model

p _{\hat\theta} ≈ p ^{*} and an accurate discriminative model q _{\hat\phi} ( ⋅ | x) ≈ p _{\hat\theta} ( ⋅ | x) .

Main forms

The ELBO has many possible expressions, each with some different emphasis.

$$\backslash mathbb\_\backslash left[\backslash ln\backslash frac\{p\_\backslash theta(x,\; z)\}\{q\_\backslash phi(z|x)\}\backslash right]\; =\; \backslash int\; q\_\backslash phi(z|x)\backslash ln\backslash fracdz$$

This form shows that if we sample

z\sim q _{\phi( ⋅} | x) , then ln (p _{\theta(x,} z))/(q _{\phi(z|x)}) is an unbiased estimator of the ELBO.

$$\backslash ln\; p\_\backslash theta(x)\; -\; D\_(q\_\backslash phi(\backslash cdot\; |\; x)\; \backslash ;\backslash |\backslash ;\; p\_\backslash theta(\backslash cdot\; |\; x))$$

This form shows that the ELBO is a lower bound on the evidence

ln p _{\theta(x)} , and that maximizing the ELBO with respect to \phi is equivalent to minimizing the KL-divergence from p _{\theta( ⋅} | x) to q _{\phi( ⋅} | x) .

$$\backslash mathbb\_[\backslash ln\; p\_\backslash theta(x|z)]\; -\; D\_(q\_\backslash phi(\backslash cdot\; |\; x)\; \backslash ;\backslash |\backslash ;\; p)$$

This form shows that maximizing the ELBO simultaneously attempts to keep

q _{\phi( ⋅} | x) close to p and concentrate q _{\phi( ⋅} | x) on those z that maximizes ln p _{\theta} (x|z) . That is, the approximate posterior q _{\phi( ⋅} | x) balances between staying close to the prior p and moving towards the maximum likelihood \arg max _{z} ln p _{\theta} (x|z) .

$$H(q\_\backslash phi(\backslash cdot\; |\; x))\; +\; \backslash mathbb\_[\backslash ln\; p\_\backslash theta(z|x)]\; +\; \backslash ln\; p\_\backslash theta(x)$$

This form shows that maximizing the ELBO simultaneously attempts to keep the entropy of

q _{\phi( ⋅} | x) high, and concentrate q _{\phi( ⋅} | x) on those z that maximizes ln p _{\theta} (z|x) . That is, the approximate posterior q _{\phi( ⋅} | x) balances between being a uniform distribution and moving towards the maximum a posteriori \arg max _{z} ln p _{\theta} (z|x) .

Data-processing inequality

Suppose we take

N independent samples from p ^{*} , and collect them in the dataset D = \{x _{1,} ..., x _{N\}} , then we have empirical distribution q _{D(x)} = (1 N)/(\sum) _{i} \delta _{x _{i}} .

Fitting

p _{\theta(x)} to q _{D(x)} can be done, as usual, by maximizing the loglikelihood ln p _{\theta(D)} :$$D\_(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; =\; -\backslash frac\; 1N\; \backslash sum\_i\; \backslash ln\; p\_\backslash theta(x\_i)\; -\; H(q\_D)=\; -\backslash frac\; 1N\; \backslash ln\; p\_\backslash theta(D)\; -\; H(q\_D)$$Now, by the ELBO inequality, we can bound ln p _{\theta(D)} , and thus$$D\_(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; \backslash leq\; -\backslash frac\; 1N\; L(\backslash phi,\; \backslash theta;\; D)\; -\; H(q\_D)$$The right-hand-side simplifies to a KL-divergence, and so we get:$$D\_(q\_D(x)\; \backslash |\; p\_\backslash theta(x))\; \backslash leq\; -\backslash frac\; 1N\; \backslash sum\_i\; L(\backslash phi,\; \backslash theta;\; x\_i)\; -\; H(q\_D)=\; D\_(q\_(x,\; z);\; p\_\backslash theta(x,\; z))$$This result can be interpreted as a special case of the data processing inequality.

In this interpretation, maximizing

L(\phi, \theta; D) = \sum _{i} L(\phi, \theta; x _{i)} is minimizing D _{KL} (q _{D,} (x, z); p _{\theta(x,} z)) , which upper-bounds the real quantity of interest D _{KL} (q _{D} (x); p _{\theta(x))} via the data-processing inequality. That is, we append a latent space to the observable space, paying the price of a weaker inequality for the sake of more computationally efficient minimization of the KL-divergence.^[4]

References

1. Kingma. Diederik P.. Welling. Max. 2014-05-01. Auto-Encoding Variational Bayes. stat.ML. 1312.6114.
2. Book: Goodfellow . Ian . Deep learning . Bengio . Yoshua . Courville . Aaron . 2016 . The MIT press . 978-0-262-03561-3 . Adaptive computation and machine learning . Cambridge, Mass . Chapter 19.
3. Hinton . Geoffrey E . Zemel . Richard . 1993 . Autoencoders, Minimum Description Length and Helmholtz Free Energy . Advances in Neural Information Processing Systems . Morgan-Kaufmann . 6.
4. Kingma . Diederik P. . Welling . Max . 2019-11-27 . An Introduction to Variational Autoencoders . Foundations and Trends in Machine Learning . English . 12 . 4 . Section 2.7 . 10.1561/2200000056 . 1935-8237. 1906.02691 . 174802445 .