In information theory, self-information or surprisal is a synonym for the entropy of a random variable; it is the expected value of the 'surprisal' of a random event, but is not the same as the surprisal. It is expressed in a unit of information, for example bits, nats, or hartleys, depending on the base of the logarithm used in its calculation.
Contents
The term self-information is also sometimes used as a synonym of the related information-theoretic concept of entropy. These two meanings are not equivalent, and this article covers the first sense only.
Definition
By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero.
For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin, “Weather forecast for tonight: dark. Continued dark overnight, with widely scattered light by morning.” Assuming one not residing near the Earth's poles or polar circles, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.
When the content of a message is known a priori with certainty, with probability of 1, there is no actual information conveyed in the message. Only when the advanced knowledge of the content of the message by the receiver is less certain than 100% does the message actually convey information.
Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event,
for some function
Further, by definition, the measure of self-information is nonnegative and additive. If a message informing of event
Because of the independence of events
However, applying function
The class of function
is the logarithm function of any base. The only operational difference between logarithms of different bases is that of different scaling constants.
Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that
Taking into account these properties, the self-information
The smaller the probability of event
As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 bits (probability 1/16), and the information content associated with getting a result other than the one specified would be 0.09 bits (probability 15/16). See below for detailed examples.
This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term was coined by Myron Tribus in his 1961 book Thermostatics and Thermodynamics.
The information entropy of a random event is the expected value of its self-information.
Self-information is an example of a proper scoring rule.
Examples
This outcome equals the sum of the individual amounts of self-information associated with {throw 1 = 'two'} and {throw 2 = 'four'}; namely 2.585 + 2.585 = 5.170 bits.
Relationship to entropy
The entropy is the expected value of the self-information of the values of a discrete random variable. Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies