Dual total correlation - Alchetron, The Free Social Encyclopedia

In information theory, dual total correlation (Han 1978), excess entropy (Olbrich 2008), or binding information (Abdallah and Plumbley 2010) is one of the two known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation (Ay 2001).

Definition

For a set of n random variables { X 1 , … , X n } , the dual total correlation D ( X 1 , … , X n ) is given by

D ( X 1 , … , X n ) = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) ,

where H ( X 1 , … , X n ) is the joint entropy of the variable set { X 1 , … , X n } and H ( X i ∣ ⋯ ) is the conditional entropy of variable X i , given the rest.

Normalized

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value H ( X 1 , … , X n ) ,

N D ( X 1 , … , X n ) = D ( X 1 , … , X n ) H ( X 1 , … , X n ) .

Bounds

Dual total correlation is non-negative and bounded above by the joint entropy H ( X 1 , … , X n ) .

0 ≤ D ( X 1 , … , X n ) ≤ H ( X 1 , … , X n ) .

Secondly, Dual total correlation has a close relationship with total correlation, C ( X 1 , … , X n ) . In particular,

C ( X 1 , … , X n ) n − 1 ≤ D ( X 1 , … , X n ) ≤ ( n − 1 ) C ( X 1 , … , X n ) .

Relation to other quantities

In measure theoretic terms, by the definition of dual total correlation:

D ( X 1 , … , X n ) = μ ( ⋃ i X ~ i ∖ ( ⋃ j X ~ j ∖ ⋃ k ≠ j X ~ k ) ) )

which is equal to the union of the pairwise mutual informations:

D ( X 1 , … , X n ) = μ ( ⋃ i ⋃ j ≠ i ( X ~ i ∩ X ~ j ) )

History

Han (1978) originally defined the dual total correlation as,

D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) .

However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:

D ( X 1 , … , X n ) ≡ [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] − ( n − 1 ) H ( X 1 , … , X n ) = [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) ] + ( 1 − n ) H ( X 1 , … , X n ) = H ( X 1 , … , X n ) + [ ∑ i = 1 n H ( X 1 , … , X i − 1 , X i + 1 , … , X n ) − H ( X 1 , … , X n ) ] = H ( X 1 , … , X n ) − ∑ i = 1 n H ( X i ∣ X 1 , … , X i − 1 , X i + 1 , … , X n ) .

References

Dual total correlation Wikipedia

(Text) CC BY-SA