Supriya Ghosh (Editor)

Dual total correlation

Updated on
Edit
Like
Comment
Share on FacebookTweet on TwitterShare on LinkedInShare on Reddit
Dual total correlation

In information theory, dual total correlation (Han 1978), excess entropy (Olbrich 2008), or binding information (Abdallah and Plumbley 2010) is one of the two known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation (Ay 2001).

Contents

Definition

For a set of n random variables { X 1 , , X n } , the dual total correlation D ( X 1 , , X n ) is given by

D ( X 1 , , X n ) = H ( X 1 , , X n ) i = 1 n H ( X i X 1 , , X i 1 , X i + 1 , , X n ) ,

where H ( X 1 , , X n ) is the joint entropy of the variable set { X 1 , , X n } and H ( X i ) is the conditional entropy of variable X i , given the rest.

Normalized

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value H ( X 1 , , X n ) ,

N D ( X 1 , , X n ) = D ( X 1 , , X n ) H ( X 1 , , X n ) .

Bounds

Dual total correlation is non-negative and bounded above by the joint entropy H ( X 1 , , X n ) .

0 D ( X 1 , , X n ) H ( X 1 , , X n ) .

Secondly, Dual total correlation has a close relationship with total correlation, C ( X 1 , , X n ) . In particular,

C ( X 1 , , X n ) n 1 D ( X 1 , , X n ) ( n 1 ) C ( X 1 , , X n ) .

Relation to other quantities

In measure theoretic terms, by the definition of dual total correlation:

D ( X 1 , , X n ) = μ ( i X ~ i ( j X ~ j k j X ~ k ) ) )

which is equal to the union of the pairwise mutual informations:

D ( X 1 , , X n ) = μ ( i j i ( X ~ i X ~ j ) )

History

Han (1978) originally defined the dual total correlation as,

D ( X 1 , , X n ) [ i = 1 n H ( X 1 , , X i 1 , X i + 1 , , X n ) ] ( n 1 ) H ( X 1 , , X n ) .

However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:

D ( X 1 , , X n ) [ i = 1 n H ( X 1 , , X i 1 , X i + 1 , , X n ) ] ( n 1 ) H ( X 1 , , X n ) = [ i = 1 n H ( X 1 , , X i 1 , X i + 1 , , X n ) ] + ( 1 n ) H ( X 1 , , X n ) = H ( X 1 , , X n ) + [ i = 1 n H ( X 1 , , X i 1 , X i + 1 , , X n ) H ( X 1 , , X n ) ] = H ( X 1 , , X n ) i = 1 n H ( X i X 1 , , X i 1 , X i + 1 , , X n ) .

References

Dual total correlation Wikipedia