Neural network Gaussian process explained

A Neural Network Gaussian Process (NNGP) is a Gaussian process (GP) obtained as the limit of a certain type of sequence of neural networks. Specifically, a wide variety of network architectures converges to a GP in the infinitely wide limit, in the sense of distribution.^{[1] [2] [3] [4] [5] [6] [7] [8]} The concept constitutes an intensional definition, i.e., a NNGP is just a GP, but distinguished by how it is obtained.

Motivation

Bayesian networks are a modeling tool for assigning probabilities to events, and thereby characterizing the uncertainty in a model's predictions. Deep learning and artificial neural networks are approaches used in machine learning to build computational models which learn from training examples. Bayesian neural networks merge these fields. They are a type of neural network whose parameters and predictions are both probabilistic.^{[9] [10]} While standard neural networks often assign high confidence even to incorrect predictions,^[11] Bayesian neural networks can more accurately evaluate how likely their predictions are to be correct.

Computation in artificial neural networks is usually organized into sequential layers of artificial neurons. The number of neurons in a layer is called the layer width. When we consider a sequence of Bayesian neural networks with increasingly wide layers (see figure), they converge in distribution to a NNGP. This large width limit is of practical interest, since the networks often improve as layers get wider.^{[12] [13]} And the process may give a closed form way to evaluate networks.

NNGPs also appears in several other contexts: It describes the distribution over predictions made by wide non-Bayesian artificial neural networks after random initialization of their parameters, but before training; it appears as a term in neural tangent kernel prediction equations; it is used in deep information propagation to characterize whether hyperparameters and architectures will be trainable.^[14] It is related to other large width limits of neural networks.

Scope

The first correspondence result had been established in the 1995 PhD thesis of Radford M. Neal, then supervised by Geoffrey Hinton at University of Toronto. Neal cites David J. C. MacKay as inspiration, who worked in Bayesian learning.

Today the correspondence is proven for: Single hidden layer Bayesian neural networks; deep fully connected networks as the number of units per layer is taken to infinity; convolutional neural networks as the number of channels is taken to infinity; transformer networks as the number of attention heads is taken to infinity;^[15] recurrent networks as the number of units is taken to infinity.In fact, this NNGP correspondence holds for almost any architecture: Generally, if an architecture can be expressed solely via matrix multiplication and coordinatewise nonlinearities (i.e., a tensor program), then it has an infinite-width GP.This in particular includes all feedforward or recurrent neural networks composed of multilayer perceptron, recurrent neural networks (e.g., LSTMs, GRUs), (nD or graph) convolution, pooling, skip connection, attention, batch normalization, and/or layer normalization.

Illustration

Every setting of a neural network's parameters

\theta corresponds to a specific function computed by the neural network. A prior distribution p(\theta) over neural network parameters therefore corresponds to a prior distribution over functions computed by the network. As neural networks are made infinitely wide, this distribution over functions converges to a Gaussian process for many architectures.

The notation used in this section is the same as the notation used below to derive the correspondence between NNGPs and fully connected networks, and more details can be found there.

The figure to the right plots the one-dimensional outputs

z ^{L( ⋅ ;\theta)} of a neural network for two inputs x and x ^{*} against each other. The black dots show the function computed by the neural network on these inputs for random draws of the parameters from p(\theta) . The red lines are iso-probability contours for the joint distribution over network outputs z ^{L(x;\theta)} and z ^{L(x} ^{*;\theta)} induced by p(\theta) . This is the distribution in function space corresponding to the distribution p(\theta) in parameter space, and the black dots are samples from this distribution. For infinitely wide neural networks, since the distribution over functions computed by the neural network is a Gaussian process, the joint distribution over network outputs is a multivariate Gaussian for any finite set of network inputs.

Discussion

Infinitely wide fully connected network

This section expands on the correspondence between infinitely wide neural networks and Gaussian processes for the specific case of a fully connected architecture. It provides a proof sketch outlining why the correspondence holds, and introduces the specific functional form of the NNGP for fully connected networks. The proof sketch closely follows the approach by Novak and coauthors.

Network architecture specification

Consider a fully connected artificial neural network with inputs

x , parameters \theta consisting of weights W ^{l} and biases b ^{l} for each layer l in the network, pre-activations (pre-nonlinearity) z ^{l} , activations (post-nonlinearity) y ^{l} , pointwise nonlinearity \phi( ⋅ ) , and layer widths n ^{l} . For simplicity, the width n ^{L+1} of the readout vector z ^{L} is taken to be 1. The parameters of this network have a prior distribution p(\theta) , which consists of an isotropic Gaussian for each weight and bias, with the variance of the weights scaled inversely with layer width. This network is illustrated in the figure to the right, and described by the following set of equations:

Notes and References

1. Williams . Christopher K. I. . 1997 . Computing with infinite networks . Neural Information Processing Systems.
2. Lee . Jaehoon . Bahri . Yasaman . Novak . Roman . Schoenholz . Samuel S. . Pennington . Jeffrey . Sohl-Dickstein . Jascha . 2017 . Deep Neural Networks as Gaussian Processes . International Conference on Learning Representations . 1711.00165 . 2017arXiv171100165L.
3. G. de G. Matthews . Alexander . Rowland . Mark . Hron . Jiri . Turner . Richard E. . Ghahramani . Zoubin . 2017 . Gaussian Process Behaviour in Wide Deep Neural Networks . International Conference on Learning Representations . 1804.11271 . 2018arXiv180411271M.
4. Novak . Roman . Xiao . Lechao . Lee . Jaehoon . Bahri . Yasaman . Yang . Greg . Abolafia . Dan . Pennington . Jeffrey . Sohl-Dickstein . Jascha . 2018 . Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes . International Conference on Learning Representations . 1810.05148 . 2018arXiv181005148N.
5. Garriga-Alonso . Adrià . Aitchison . Laurence . Rasmussen . Carl Edward . 2018 . Deep Convolutional Networks as shallow Gaussian Processes . International Conference on Learning Representations . 1808.05587 . 2018arXiv180805587G.
6. 1810.10798 . stat.ML . Anastasia . Borovykh . A Gaussian Process perspective on Convolutional Neural Networks . 2018.
7. 2002.08517 . cs.LG . Russell . Tsuchida . Tim . Pearce . Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks . 2020 . van der Heide . Christopher . Roosta . Fred . Gallagher . Marcus.
8. Yang . Greg . 2019 . Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes . Advances in Neural Information Processing Systems . 1910.12478 . 2019arXiv191012478Y.
9. MacKay . David J. C. . 1992 . A Practical Bayesian Framework for Backpropagation Networks . Neural Computation . 4 . 3 . 448–472 . 10.1162/neco.1992.4.3.448 . 0899-7667 . 16543854.
10. Book: Neal, Radford M. . Bayesian Learning for Neural Networks . Springer Science and Business Media . 2012.
11. Guo . Chuan . Pleiss . Geoff . Sun . Yu . Weinberger . Kilian Q. . 2017 . On calibration of modern neural networks . Proceedings of the 34th International Conference on Machine Learning-Volume 70 . 1706.04599.
12. Novak . Roman . Bahri . Yasaman . Abolafia . Daniel A. . Pennington . Jeffrey . Sohl-Dickstein . Jascha . 2018-02-15 . Sensitivity and Generalization in Neural Networks: an Empirical Study . International Conference on Learning Representations . 1802.08760 . 2018arXiv180208760N.
13. Neyshabur . Behnam . Li . Zhiyuan . Bhojanapalli . Srinadh . LeCun . Yann . Srebro . Nathan . 2019 . Towards understanding the role of over-parametrization in generalization of neural networks . International Conference on Learning Representations . 1805.12076 . 2018arXiv180512076N.
14. Schoenholz . Samuel S. . Gilmer . Justin . Ganguli . Surya . Sohl-Dickstein . Jascha . 2016 . Deep information propagation . International Conference on Learning Representations . 1611.01232.
15. Hron . Jiri . Bahri . Yasaman . Sohl-Dickstein . Jascha . Novak . Roman . 2020-06-18 . Infinite attention: NNGP and NTK for deep attention networks . International Conference on Machine Learning . 2020 . 2006.10540 . 2020arXiv200610540H.