In computer science, the use of log probabilities means representing probabilities in logarithmic space, instead of the standard
A log probability is simply the logarithm of a probability. The logarithm function is not defined for zero, so log probabilities can only represent non-zero probabilities. Since the logarithm of a number in
The product of probabilities
The sum of probabilities
However, in many applications a multiplication of probabilities (giving the probability of all independent events occurring) is used more often than their addition (giving the probability of at least one of them occurring). Additionally, the cost of computing the addition can be avoided in some situations by simply using the highest probability as an approximation. Since probabilities are non-negative this gives a lower bound. This approximation is used in reverse to get a continuous approximation of the max function.
Representing probabilities in this way has two main advantages:
- Speed. Since multiplication is more expensive than addition, taking the product of a high number of probabilities is faster if they are represented in log form. (The conversion to log form is expensive, but is only incurred once.)
- Accuracy. The use of log probabilities improves numerical stability, when the probabilities are very small.
The use of log probabilities is widespread in several fields of computer science such as information theory and natural language processing as it represents the surprisal, the minimum length of the message that specifies the outcome in an optimally efficient code.
Addition in log space
The formula above is more accurate than
Note that the above formula alone will incorrectly produce an indeterminate result in the case where both arguments are
Note also that for numerical reasons, one should use a function that computes