Pruning (artificial neural network) explained

In the context of artificial neural networks, pruning is the practice of removing parameters (which may entail removing individual parameters, or parameters in groups such as by neurons) from an existing network.^[1] The goal of this process is to maintain accuracy of the network while increasing its efficiency. This can be done to reduce the computational resources required to run the neural network. A biological process of synaptic pruning takes place in the brain of mammals during development^[2] (see also Neural Darwinism).

Node (neuron) pruning

A basic algorithm for pruning is as follows:^{[3] [4]}

1. Evaluate the importance of each neuron.
2. Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
3. Remove the least important neuron.
4. Check a termination condition (to be determined by the user) to see whether to continue pruning.

Edge (weight) pruning

Most work on neural network pruning focuses on removing weights, namely, setting their values to zero. Early work suggested to also change the values of non-pruned weights.^[5]

Notes and References

1. Blalock. Davis. Ortiz. Jose Javier Gonzalez. Frankle. Jonathan. Guttag. John. 2020-03-06. What is the State of Neural Network Pruning?. cs.LG. 2003.03033.
2. Chechik . Gal . Meilijson . Isaac . Ruppin . Eytan . October 1998 . Synaptic Pruning in Development: A Computational Account . Neural Computation . 10 . 7 . 1759–1777 . 10.1162/089976698300017124 . 9744896 . 14629275 . 0899-7667.
3. Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
4. Web site: Gildenblat . Jacob . 2017-06-23 . Pruning deep neural networks to make them fast and small . 2024-02-04 . Github . en.
5. Chechik . Gal . Meilijson . Isaac . Ruppin . Eytan . April 2001 . Effective Neuronal Learning with Ineffective Hebbian Learning Rules . Neural Computation . 13 . 4 . 817–840 . 10.1162/089976601300014367 . 11255571 . 133186 . 0899-7667.