Descent direction explained

In optimization, a descent direction is a vector

p \in R ^{n} that points towards a local minimum x ^{*} of an objective function f: R ^{n\toR} .

Computing

x ^{*} by an iterative method, such as line search defines a descent direction p ^{n} _{k\in R} at the k th iterate to be any p _{k} such that \langle p _{k,\nabla} f( x _{k)\rangle} < 0 , where \langle , \rangle denotes the inner product. The motivation for such an approach is that small steps along p _{k} guarantee that \displaystyle f is reduced, by Taylor's theorem.

Using this definition, the negative of a non-zero gradient is always adescent direction, as

\langle - \nabla f( x _{k),} \nabla f( x _{k)} \rangle = - \langle \nabla f( x _{k),} \nabla f( x _{k)} \rangle < 0 .

Numerous methods exist to compute descent directions, all with differing merits, such as gradient descent or the conjugate gradient method.

More generally, if

P is a positive definite matrix, thenp _{k} = - P \nabla f(x _{k)} is a descent direction at x _{k} .^[1] This generality is used in preconditioned gradient descent methods.

See also

• Directional derivative

Notes and References

1. Book: J. M. Ortega and W. C. Rheinbold . Iterative Solution of Nonlinear Equations in Several Variables . 243 . 1970 . 10.1137/1.9780898719468 .