Partial information decomposition explained

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.^[1]

Motivation

Information theory can quantify the amount of information a single source variable

has about a target variable via the mutual information . If we now consider a second source variable , classical information theory can only describe the mutual information of the joint variable with , given by . In general however, it would be interesting to know how exactly the individual variables and and their interactions relate to .

Consider that we are given two source variables

and a target variable . In this case the total mutual information , while the individual mutual information . That is, there is synergistic information arising from the interaction of about , which cannot be easily captured with classical information theoretic quantities.

Definition

Partial information decomposition further decomposes the mutual information between the source variables

with the target variable as

Here the individual information atoms are defined as

is the unique information that has about , which is not in is the synergistic information that is in the interaction of and about is the redundant information that is in both or about

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.^{[2] [3] [4]}

Applications

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology,^[5] neuroscience^{[6] [7] [8]} sociology,^[9] and machine learning^[10] Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems^[11] and may be relevant to formal theories of consciousness.^[12]

See also

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information

Notes and References

1. Williams PL, Beer RD . 2010-04-14 . Nonnegative Decomposition of Multivariate Information . cs.IT . 1004.2515 .
2. Quax R, Har-Shemesh O, Sloot PM . February 2017 . Quantifying Synergistic Information Using Intermediate Stochastic Variables . Entropy . en . 19 . 2 . 85 . 10.3390/e19020085 . 1099-4300. free . 1602.01265 .
3. Rosas FE, Mediano PA, Rassouli B, Barrett AB . 2020-12-04 . An operational information decomposition via synergistic disclosure . Journal of Physics A: Mathematical and Theoretical . 53 . 48 . 485001 . 10.1088/1751-8121/abb723 . 2001.10387 . 2020JPhA...53V5001R . 210932609 . 1751-8113.
4. Kolchinsky A . A Novel Approach to the Partial Information Decomposition . Entropy . 24 . 3 . 403 . March 2022 . 35327914 . 10.3390/e24030403 . 8947370 . 1908.08642 . 2022Entrp..24..403K . free .
5. Goodwell AE, Jiang P, Ruddell BL, Kumar P . February 2020 . Debates—Does Information Theory Provide a New Paradigm for Earth Science? Causality, Interaction, and Feedback . Water Resources Research . en . 56 . 2 . 10.1029/2019WR024940 . 2020WRR....5624940G . 216201598 . 0043-1397. free .
6. Newman EL, Varley TF, Parakkattu VK, Sherrill SP, Beggs JM . Revealing the Dynamics of Neural Information Processing with Multivariate Information Decomposition . Entropy . 24 . 7 . 930 . July 2022 . 35885153 . 10.3390/e24070930 . 9319160 . 2022Entrp..24..930N . free .
7. Luppi AI, Mediano PA, Rosas FE, Holland N, Fryer TD, O'Brien JT, Rowe JB, Menon DK, Bor D, Stamatakis EA . 6 . A synergistic core for human brain evolution and cognition . Nature Neuroscience . 25 . 6 . 771–782 . June 2022 . 35618951 . 10.1038/s41593-022-01070-0 . 249096746 . 7614771 .
8. Wibral M, Priesemann V, Kay JW, Lizier JT, Phillips WA . Partial information decomposition as a unified approach to the specification of neural goal functions . Brain and Cognition . 112 . 25–38 . March 2017 . 26475739 . 10.1016/j.bandc.2015.09.004 . Perspectives on Human Probabilistic Inferences and the 'Bayesian Brain' . 4394452 . free . 1510.00831 .
9. Varley TF, Kaminski P . October 2022 . Untangling Synergistic Effects of Intersecting Social Identities with Partial Information Decomposition . Entropy . en . 24 . 10 . 1387 . 10.3390/e24101387 . 37420406 . 9611752 . 2022Entrp..24.1387V . 1099-4300. free .
10. Tax TM, Mediano PA, Shanahan M . September 2017 . The Partial Information Decomposition of Generative Neural Network Models . Entropy . en . 19 . 9 . 474 . 10.3390/e19090474 . 2017Entrp..19..474T . 1099-4300. free . 10044/1/50586 . free .
11. Mediano PA, Rosas FE, Luppi AI, Jensen HJ, Seth AK, Barrett AB, Carhart-Harris RL, Bor D . 6 . Greater than the parts: a review of the information decomposition approach to causal emergence . Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences . 380 . 2227 . 20210246 . July 2022 . 35599558 . 9125226 . 10.1098/rsta.2021.0246 .
12. Luppi AI, Mediano PA, Rosas FE, Harrison DJ, Carhart-Harris RL, Bor D, Stamatakis EA . What it is like to be a bit: an integrated information decomposition account of emergent mental phenomena . Neuroscience of Consciousness . 2021 . 2 . niab027 . 2021 . 34804593 . 8600547 . 10.1093/nc/niab027 .