In optimization, a descent direction is a vector p ∈ R n that, in the sense below, moves us closer towards a local minimum x ∗ of our objective function f : R n → R .
Suppose we are computing x ∗ by an iterative method, such as line search. We define a descent direction p k ∈ R n at the k th iterate to be any p k such that ⟨ p k , ∇ f ( x k ) ⟩ < 0 , where ⟨ , ⟩ denotes the inner product. The motivation for such an approach is that small steps along p k guarantee that f is reduced, by Taylor's theorem.
Using this definition, the negative of a non-zero gradient is always a descent direction, as ⟨ − ∇ f ( x k ) , ∇ f ( x k ) ⟩ = − ⟨ ∇ f ( x k ) , ∇ f ( x k ) ⟩ < 0 .
Numerous methods exist to compute descent directions, all with differing merits. For example, one could use gradient descent or the conjugate gradient method.
More generally, if P is a positive definite matrix, then d = − P ∇ f ( x ) is a descent direction at x . This generality is used in preconditioned gradient descent methods.
