Delta rule explained

In machine learning, the delta rule is a gradient descent learning rule for updating the weights of the inputs to artificial neurons in a single-layer neural network.^[1] It can be derived as the backpropagation algorithm for a single-layer neural network with mean-square error loss function.

For a neuron

j with activation function g(x) , the delta rule for neuron j 's i -th weight w _{ji} is given by

$$\backslash Delta\; w\_\; =\; \backslash alpha(t\_j-y\_j)\; g\text{'}(h\_j)\; x\_i,$$

where

\alpha is a small constant called learning rateg(x) is the neuron's activation functiong' is the derivative of g t _{j} is the target outputh _{j} is the weighted sum of the neuron's inputsy _{j} is the actual outputx _{i} is the i -th input.

It holds that $h\_j\; =\; \backslash sum\_i\; x\_i\; w\_$ and

y _{j=g(h} _{j)} .

The delta rule is commonly stated in simplified form for a neuron with a linear activation function as $$\backslash Delta\; w\_\; =\; \backslash alpha\; \backslash left(t\_j-y\_j\backslash right)\; x\_i$$

While the delta rule is similar to the perceptron's update rule, the derivation is different. The perceptron uses the Heaviside step function as the activation function

g(h) , and that means that g'(h) does not exist at zero, and is equal to zero elsewhere, which makes the direct application of the delta rule impossible.

Derivation of the delta rule

The delta rule is derived by attempting to minimize the error in the output of the neural network through gradient descent. The error for a neural network with

j outputs can be measured as $$E\; =\; \backslash sum\_\; \backslash tfrac\; \backslash left(t\_j-y\_j\backslash right)^2\; .$$

In this case, we wish to move through "weight space" of the neuron (the space of all possible values of all of the neuron's weights) in proportion to the gradient of the error function with respect to each weight. In order to do that, we calculate the partial derivative of the error with respect to each weight. For the

i th weight, this derivative can be written as $$\backslash frac\; .$$

Because we are only concerning ourselves with the

j -th neuron, we can substitute the error formula above while omitting the summation:$$\backslash frac\; =\; \backslash frac\; \backslash left\; [\backslash frac\{1\}\{2\}\; \backslash left(t\_j-y\_j\; \backslash right)\; ^2\; \backslash right\; ]$$

Next we use the chain rule to split this into two derivatives:$$\backslash frac\; =\; \backslash frac\; \backslash frac$$

To find the left derivative, we simply apply the power rule and the chain rule:$$\backslash frac\; =\; -\; \backslash left\; (t\_j-y\_j\; \backslash right)\; \backslash frac$$

To find the right derivative, we again apply the chain rule, this time differentiating with respect to the total input to

j , h _{j} :$$\backslash frac\; =\; -\; \backslash left\; (t\_j-y\_j\; \backslash right)\; \backslash frac\; \backslash frac$$

Note that the output of the

j th neuron, y _{j} , is just the neuron's activation function g applied to the neuron's input h _{j} . We can therefore write the derivative of y _{j} with respect to h _{j} simply as g 's first derivative:$$\backslash frac\; =\; -\; \backslash left\; (t\_j-y\_j\; \backslash right)\; g\text{'}(h\_j)\; \backslash frac$$

Next we rewrite

h _{j} in the last term as the sum over all k weights of each weight w _{jk} times its corresponding input x _{k} :$$\backslash frac\; =\; -\; \backslash left\; (t\_j-y\_j\; \backslash right)\; g\text{'}(h\_j)\; \backslash ;\; \backslash frac\; \backslash !\backslash !\backslash left[\backslash sum\_\{i\}\; x\_i\; w\_\{ji\}\; \backslash right]$$

Because we are only concerned with the

i th weight, the only term of the summation that is relevant is x _{i} w _{ji} . Clearly, $$\backslash frac\; =\; x\_i.$$giving us our final equation for the gradient:$$\backslash frac\; =\; -\; \backslash left\; (t\_j-y\_j\; \backslash right)\; g\text{'}(h\_j)\; x\_i$$

As noted above, gradient descent tells us that our change for each weight should be proportional to the gradient. Choosing a proportionality constant

\alpha and eliminating the minus sign to enable us to move the weight in the negative direction of the gradient to minimize error, we arrive at our target equation:$$\backslash Delta\; w\_=\backslash alpha(t\_j-y\_j)\; g\text{'}(h\_j)\; x\_i\; .$$

See also

• Stochastic gradient descent
• Backpropagation
• Rescorla–Wagner model – the origin of delta rule

Notes and References

1. Web site: Russell . Ingrid . The Delta Rule . University of Hartford . 5 November 2012 . dead . https://web.archive.org/web/20160304032228/http://uhavax.hartford.edu/compsci/neural-networks-delta-rule.html . 4 March 2016 .