Transfer entropy explained

Transfer entropy is a non-parametric statistic measuring the amount of directed (time-asymmetric) transfer of information between two random processes.^{[1] [2] [3]} Transfer entropy from a process X to another process Y is the amount of uncertainty reduced in future values of Y by knowing the past values of X given past values of Y. More specifically, if

X _{t} and Y _{t} for t\in N denote two random processes and the amount of information is measured using Shannon's entropy, the transfer entropy can be written as:T _{X →} = H\left( Y _{t} \mid Y _{t-1:t-L} \right) - H\left( Y _{t} \mid Y _{t-1:t-L} , X _{t-1:t-L} \right),

where H(X) is Shannon's entropy of X. The above definition of transfer entropy has been extended by other types of entropy measures such as Rényi entropy.^[4]

Transfer entropy is conditional mutual information,^{[5] [6]} with the history of the influenced variable

Y _{t-1:t-L} in the condition:T _{X →} = I(Y _{t} ; X _{t-1:t-L} \mid Y _{t-1:t-L} ).

Transfer entropy reduces to Granger causality for vector auto-regressive processes.^[7] Hence, it is advantageous when the model assumption of Granger causality doesn't hold, for example, analysis of non-linear signals.^[8] However, it usually requires more samples for accurate estimation.^[9] The probabilities in the entropy formula can be estimated using different approaches (binning, nearest neighbors) or, in order to reduce complexity, using a non-uniform embedding.^[10] While it was originally defined for bivariate analysis, transfer entropy has been extended to multivariate forms, either conditioning on other potential source variables^[11] or considering transfer from a collection of sources,^[12] although these forms require more samples again.

Transfer entropy has been used for estimation of functional connectivity of neurons,^{[13] [14]} social influence in social networks^[15] and statistical causality between armed conflict events.^[16] Transfer entropy is a finite version of the directed information which was defined in 1990 by James Massey^[17] as

I(X ^{n\to} Y ^{n)} = \sum ^{n} _{i =1} I(X ^{i -1} _{i|Y} ) , where X ^{n} denotes the vector X _{1,X} _{2,...,X} _{n} and Y ^{n} denotes Y _{1,Y} _{2,...,Y} _{n} . The directed information places an important role in characterizing the fundamental limits (channel capacity) of communication channels with or without feedback^{[18] [19]} and gambling with causal side information.^[20]

See also

• Directed information
• Mutual information
• Conditional mutual information
• Causality
• Causality (physics)
• Structural equation modeling
• Rubin causal model

External links

• Web site: Transfer Entropy Toolbox. Google Code., a toolbox, developed in C++ and MATLAB, for computation of transfer entropy between spike trains.
• Web site: Java Information Dynamics Toolkit (JIDT). GitHub. 2019-01-16., a toolbox, developed in Java and usable in MATLAB, GNU Octave and Python, for computation of transfer entropy and related information-theoretic measures in both discrete and continuous-valued data.
• Web site: Multivariate Transfer Entropy (MuTE) toolbox. GitHub. 2019-01-09., a toolbox, developed in MATLAB, for computation of transfer entropy with different estimators.

Notes and References

1. Schreiber. Thomas. Measuring information transfer. Physical Review Letters. 1 July 2000. 85. 2. 461–464. 10.1103/PhysRevLett.85.461. 10991308. nlin/0001042. 2000PhRvL..85..461S. 7411376.
2. 2007 . Granger causality . 2 . 7 . 1667 . Seth . Anil. . 10.4249/scholarpedia.1667 . 2007SchpJ...2.1667S. free .
3. Hlaváčková-Schindler. Katerina. Palus, M . Vejmelka, M . Bhattacharya, J . Causality detection based on information-theoretic approaches in time series analysis. Physics Reports. 1 March 2007. 441. 1. 1–46. 10.1016/j.physrep.2006.12.004. 2007PhR...441....1H. 10.1.1.183.1617.
4. Jizba. Petr. Kleinert. Hagen. Shefaat. Mohammad. 2012-05-15. Rényi's information transfer between financial time series. Physica A: Statistical Mechanics and Its Applications. en. 391. 10. 2971–2989. 10.1016/j.physa.2011.12.064. 0378-4371. 1106.5913. 2012PhyA..391.2971J. 51789622.
5. Wyner. A. D. . A definition of conditional mutual information for arbitrary ensembles. Information and Control. 1978. 38. 1. 51–59. 10.1016/s0019-9958(78)90026-8. free.
6. Dobrushin. R. L. . General formulation of Shannon's main theorem in information theory. Uspekhi Mat. Nauk. 1959. 14. 3–104.
7. Barnett. Lionel. Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables. Physical Review Letters. 1 December 2009. 103. 23. 10.1103/PhysRevLett.103.238701. 2009PhRvL.103w8701B. 20366183. 238701. 0910.4514. 1266025.
8. Lungarella. M.. Ishiguro, K. . Kuniyoshi, Y. . Otsu, N. . Methods for quantifying the causal structure of bivariate time series. International Journal of Bifurcation and Chaos. 1 March 2007. 17. 3. 903–921. 10.1142/S0218127407017628. 2007IJBC...17..903L. 10.1.1.67.3585.
9. Pereda. E. Quiroga, RQ . Bhattacharya, J . Nonlinear multivariate analysis of neurophysiological signals.. Progress in Neurobiology. Sep–Oct 2005. 77. 1–2. 1–37. 16289760. 10.1016/j.pneurobio.2005.10.003. nlin/0510077. 2005nlin.....10077P. 9529656.
10. Montalto. A. Faes, L . Marinazzo, D . MuTE: A MATLAB Toolbox to Compare Established and Novel Estimators of the Multivariate Transfer Entropy.. PLOS ONE. Oct 2014. 25314003. 10.1371/journal.pone.0109462. 9. 10. 4196918. e109462. 2014PLoSO...9j9462M. free.
11. Lizier. Joseph. Prokopenko, Mikhail . Zomaya, Albert . Local information transfer as a spatiotemporal filter for complex systems. Physical Review E. 2008. 77. 2. 026110. 10.1103/PhysRevE.77.026110. 18352093. 0809.3275. 2008PhRvE..77b6110L. 15634881.
12. Lizier. Joseph. Heinzle, Jakob . Horstmann, Annette . Haynes, John-Dylan . Prokopenko, Mikhail . Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. Journal of Computational Neuroscience. 2011. 30. 1. 85–107. 10.1007/s10827-010-0271-2. 20799057. 3012713.
13. Vicente. Raul. Wibral, Michael . Lindner, Michael . Pipa, Gordon . Transfer entropy—a model-free measure of effective connectivity for the neurosciences . Journal of Computational Neuroscience. February 2011. 30. 1. 45–67. 10.1007/s10827-010-0262-3. 20706781. 3040354.
14. Shimono. Masanori. Beggs, John . Functional clusters, hubs, and communities in the cortical microconnectome . Cerebral Cortex. October 2014. 25. 10. 3743–57. 10.1093/cercor/bhu252 . 25336598 . 4585513.
15. 1110.2724. Information transfer in social media. Ver Steeg . Greg. Galstyan. Aram . 2012. ACM. Proceedings of the 21st international conference on World Wide Web (WWW '12) . 509–518 . 2011arXiv1110.2724V.
16. Kushwaha . Niraj . Lee . Edward D . July 2023 . Discovering the mesoscale for chains of conflict . PNAS Nexus . 2 . 7 . pgad228 . 10.1093/pnasnexus/pgad228 . 2752-6542 . 10392960 . 37533894.
17. Massey. James. Causality, Feedback And Directed Information. 1990. ISITA. 10.1.1.36.5688.
18. Permuter. Haim Henry. Weissman. Tsachy. Goldsmith. Andrea J.. Finite State Channels With Time-Invariant Deterministic Feedback. IEEE Transactions on Information Theory. February 2009. 55. 2. 644–662. 10.1109/TIT.2008.2009849. cs/0608070. 13178.
19. Kramer. G.. Capacity results for the discrete memoryless network. IEEE Transactions on Information Theory. January 2003. 49. 1. 4–21. 10.1109/TIT.2002.806135.
20. Permuter. Haim H.. Kim. Young-Han. Weissman. Tsachy. Interpretations of Directed Information in Portfolio Theory, Data Compression, and Hypothesis Testing. IEEE Transactions on Information Theory. June 2011. 57. 6. 3248–3259. 10.1109/TIT.2011.2136270. 0912.4872. 11722596.