![]() | ||
In probability theory, and in particular, information theory, the conditional mutual information is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.
Contents
Definition
For discrete random variables
where the marginal, joint, and/or conditional probability mass functions are denoted by
Alternatively, we may write in terms of joint and conditional entropies as
This can be rewritten to show its relationship to mutual information
usually rearranged as the chain rule for mutual information
Another equivalent form of the above is
Like mutual information, conditional mutual information can be expressed as a Kullback–Leibler divergence:
Or as an expected value of simpler Kullback–Leibler divergences:
More general definition
A more general definition of conditional mutual information, applicable to random variables with continuous or other arbitrary distributions, will depend on the concept of regular conditional probability. (See also.)
Let
Consider the Borel measure (on the σ-algebra generated by the open sets) in the state space of each random variable defined by assigning each Borel set the
Now we can formally define the conditional probability measure given the value of one (or, via the product topology, more) of the random variables. Let
where the limit is taken over the open neighborhoods
Finally we can define the conditional mutual information via Lebesgue integration:
where the integrand is the logarithm of a Radon–Nikodym derivative involving some of the conditional probability measures we have just defined.
Note on notation
In an expression such as
Properties
It is always true that
for discrete, jointly distributed random variables X, Y, Z. This result has been used as a basic building block for proving other inequalities in information theory, in particular, those known as Shannon-type inequalities.
Conditioning on a third random variable may either increase or decrease the mutual information: that is, the difference
Multivariate mutual information
The conditional mutual information can be used to inductively define a multivariate mutual information in a set- or measure-theoretic sense in the context of information diagrams. In this sense we define the multivariate mutual information as follows:
where
This definition is identical to that of interaction information except for a change in sign in the case of an odd number of random variables. A complication is that this multivariate mutual information (as well as the interaction information) can be positive, negative, or zero, which makes this quantity difficult to interpret intuitively. In fact, for n random variables, there are