In information theory, the binary entropy function, denoted H ( p ) or H b ( p ) , is defined as the entropy of a Bernoulli process with probability of success p . Mathematically, the Bernoulli trial is modelled as a random variable X that can take on only two values: 0 and 1. The event X = 1 is considered a success and the event X = 0 is considered a failure. (These two events are mutually exclusive and exhaustive.)
If Pr ( X = 1 ) = p , then Pr ( X = 0 ) = 1 − p and the entropy of X (in shannons) is given by
H ( X ) = H b ( p ) = − p log 2 p − ( 1 − p ) log 2 ( 1 − p ) ,
where 0 log 2 0 is taken to be 0. The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See binary logarithm.
When p = 1 2 , the binary entropy function attains its maximum value. This is the case of the unbiased bit, the most common unit of information entropy.
H ( p ) is distinguished from the entropy function H ( X ) in that the former takes a single real number as a parameter whereas the latter takes a distribution or random variables as a parameter. Sometimes the binary entropy function is also written as H 2 ( p ) . However, it is different from and should not be confused with the Rényi entropy, which is denoted as H 2 ( X ) .
In terms of information theory, entropy is considered to be a measure of the uncertainty in a message. To put it intuitively, suppose p = 0 . At this probability, the event is certain never to occur, and so there is no uncertainty at all, leading to an entropy of 0. If p = 1 , the result is again certain, so the entropy is 0 here as well. When p = 1 / 2 , the uncertainty is at a maximum; if one were to place a fair bet on the outcome in this case, there is no advantage to be gained with prior knowledge of the probabilities. In this case, the entropy is maximum at a value of 1 bit. Intermediate values fall between these cases; for instance, if p = 1 / 4 , there is still a measure of uncertainty on the outcome, but one can still predict the outcome correctly more often than not, so the uncertainty measure, or entropy, is less than 1 full bit.
The derivative of the binary entropy function may be expressed as the negative of the logit function:
d d p H b ( p ) = − logit 2 ( p ) = − log 2 ( p 1 − p ) .
The Taylor series of the binary entropy function in a neighborhood of 1/2 is
H b ( p ) = 1 − 1 2 ln 2 ∑ n = 1 ∞ ( 1 − 2 p ) 2 n n ( 2 n − 1 ) for 0 ≤ p ≤ 1 .
