Partial information decomposition explained

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.^[1]

Motivation

Information theory can quantify the amount of information a single source variable

X _{1} has about a target variable Y via the mutual information I(X _{1;Y)} . If we now consider a second source variable X _{2} , classical information theory can only describe the mutual information of the joint variable \{X _{1,X} _{2\}} with Y , given by I(X _{1,X} _{2;Y)} . In general however, it would be interesting to know how exactly the individual variables X _{1} and X _{2} and their interactions relate to Y .

Consider that we are given two source variables

X _{1,} X _{2} \in \{ 0 , 1 \} and a target variable Y = XOR(X _{1,X} _{2)} . In this case the total mutual information I(X _{1,X} _{2;Y)=1} , while the individual mutual information I(X _{1;Y)=I(X} _{2;Y)=0} . That is, there is synergistic information arising from the interaction of X _{1,X} _{2} about Y , which cannot be easily captured with classical information theoretic quantities.

Definition

Partial information decomposition further decomposes the mutual information between the source variables

\{X _{1,X} _{2\}} with the target variable Y asI(X _{1,X} _{2;Y)=Unq(X} _{1;Y} \setminus X _{2)} + Unq (X _{2;Y} \setminus X _{1)} + Syn (X _{1,X} _{2;Y)} + Red (X _{1,X} _{2;Y)}

Here the individual information atoms are defined as

Unq (X _{1;Y} \setminus X _{2)} is the unique information that X _{1} has about Y , which is not in X _{2} Syn (X _{1,X} _{2;Y)} is the synergistic information that is in the interaction of X _{1} and X _{2} about Y Red (X _{1,X} _{2;Y)} is the redundant information that is in both X _{1} or X _{2} about Y

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.^{[2] [3] [4]}

Applications

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology,^[5] neuroscience^{[6] [7] [8]} sociology,^[9] and machine learning^[10] Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems^[11] and may be relevant to formal theories of consciousness.^[12]

See also

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information

Notes and References

1. Williams PL, Beer RD . 2010-04-14 . Nonnegative Decomposition of Multivariate Information . cs.IT . 1004.2515 .
2. Quax R, Har-Shemesh O, Sloot PM . February 2017 . Quantifying Synergistic Information Using Intermediate Stochastic Variables . Entropy . en . 19 . 2 . 85 . 10.3390/e19020085 . 1099-4300. free . 1602.01265 .
3. Rosas FE, Mediano PA, Rassouli B, Barrett AB . 2020-12-04 . An operational information decomposition via synergistic disclosure . Journal of Physics A: Mathematical and Theoretical . 53 . 48 . 485001 . 10.1088/1751-8121/abb723 . 2001.10387 . 2020JPhA...53V5001R . 210932609 . 1751-8113.
4. Kolchinsky A . A Novel Approach to the Partial Information Decomposition . Entropy . 24 . 3 . 403 . March 2022 . 35327914 . 10.3390/e24030403 . 8947370 . 1908.08642 . 2022Entrp..24..403K . free .
5. Goodwell AE, Jiang P, Ruddell BL, Kumar P . February 2020 . Debates—Does Information Theory Provide a New Paradigm for Earth Science? Causality, Interaction, and Feedback . Water Resources Research . en . 56 . 2 . 10.1029/2019WR024940 . 2020WRR....5624940G . 216201598 . 0043-1397. free .
6. Newman EL, Varley TF, Parakkattu VK, Sherrill SP, Beggs JM . Revealing the Dynamics of Neural Information Processing with Multivariate Information Decomposition . Entropy . 24 . 7 . 930 . July 2022 . 35885153 . 10.3390/e24070930 . 9319160 . 2022Entrp..24..930N . free .
7. Luppi AI, Mediano PA, Rosas FE, Holland N, Fryer TD, O'Brien JT, Rowe JB, Menon DK, Bor D, Stamatakis EA . 6 . A synergistic core for human brain evolution and cognition . Nature Neuroscience . 25 . 6 . 771–782 . June 2022 . 35618951 . 10.1038/s41593-022-01070-0 . 249096746 . 7614771 .
8. Wibral M, Priesemann V, Kay JW, Lizier JT, Phillips WA . Partial information decomposition as a unified approach to the specification of neural goal functions . Brain and Cognition . 112 . 25–38 . March 2017 . 26475739 . 10.1016/j.bandc.2015.09.004 . Perspectives on Human Probabilistic Inferences and the 'Bayesian Brain' . 4394452 . free . 1510.00831 .
9. Varley TF, Kaminski P . October 2022 . Untangling Synergistic Effects of Intersecting Social Identities with Partial Information Decomposition . Entropy . en . 24 . 10 . 1387 . 10.3390/e24101387 . 37420406 . 9611752 . 2022Entrp..24.1387V . 1099-4300. free .
10. Tax TM, Mediano PA, Shanahan M . September 2017 . The Partial Information Decomposition of Generative Neural Network Models . Entropy . en . 19 . 9 . 474 . 10.3390/e19090474 . 2017Entrp..19..474T . 1099-4300. free . 10044/1/50586 . free .
11. Mediano PA, Rosas FE, Luppi AI, Jensen HJ, Seth AK, Barrett AB, Carhart-Harris RL, Bor D . 6 . Greater than the parts: a review of the information decomposition approach to causal emergence . Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences . 380 . 2227 . 20210246 . July 2022 . 35599558 . 9125226 . 10.1098/rsta.2021.0246 .
12. Luppi AI, Mediano PA, Rosas FE, Harrison DJ, Carhart-Harris RL, Bor D, Stamatakis EA . What it is like to be a bit: an integrated information decomposition account of emergent mental phenomena . Neuroscience of Consciousness . 2021 . 2 . niab027 . 2021 . 34804593 . 8600547 . 10.1093/nc/niab027 .