Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a differentiable approximation to the maximum function

and the concept of smooth minimum is similarly defined.

For large positive values of the parameter α > 0 , the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

S α has the following properties:

1. S α → max as α → ∞
2. S 0 is the average of its inputs
3. S α → min as α → − ∞

The gradient of S α is closely related to softmax and is given by

This makes the softmax function useful for optimization techniques that use gradient descent.

Another formulation is:

The ( n − 1 ) term corrects for the fact that exp ⁡ ( 0 ) = 1 by canceling out all but one zero expoential