Statistical validation of mutual information calculations: comparison of alternative numerical algorithms
- PMID: 16089850
- DOI: 10.1103/PhysRevE.71.066208
Statistical validation of mutual information calculations: comparison of alternative numerical algorithms
Abstract
Given two time series X and Y , their mutual information, I (X,Y) = I (Y,X) , is the average number of bits of X that can be predicted by measuring Y and vice versa. In the analysis of observational data, calculation of mutual information occurs in three contexts: identification of nonlinear correlation, determination of an optimal sampling interval, particularly when embedding data, and in the investigation of causal relationships with directed mutual information. In this contribution a minimum description length argument is used to determine the optimal number of elements to use when characterizing the distributions of X and Y . However, even when using partitions of the X and Y axis indicated by minimum description length, mutual information calculations performed with a uniform partition of the XY plane can give misleading results. This motivated the construction of an algorithm for calculating mutual information that uses an adaptive partition. This algorithm also incorporates an explicit test of the statistical independence of X and Y in a calculation that returns an assessment of the corresponding null hypothesis. The previously published Fraser-Swinney algorithm for calculating mutual information includes a sophisticated procedure for local adaptive control of the partitioning process. When the Fraser and Swinney algorithm and the algorithm constructed here are compared, they give very similar numerical results (less than 4% difference in a typical application). Detailed comparisons are possible when X and Y are correlated jointly Gaussian distributed because an analytic expression for I (X,Y) can be derived for that case. Based on these tests, three conclusions can be drawn. First, the algorithm constructed here has an advantage over the Fraser-Swinney algorithm in providing an explicit calculation of the probability of the null hypothesis that X and Y are independent. Second, the Fraser-Swinney algorithm is marginally the more accurate of the two algorithms when large data sets are used. With smaller data sets, however, the Fraser-Swinney algorithm reports structures that disappear when more data are available. Third, the algorithm constructed here requires about 0.5% of the computation time required by the Fraser-Swinney algorithm.
Similar articles
-
Estimating mutual information.Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Jun;69(6 Pt 2):066138. doi: 10.1103/PhysRevE.69.066138. Epub 2004 Jun 23. Phys Rev E Stat Nonlin Soft Matter Phys. 2004. PMID: 15244698
-
Minimax mutual information approach for independent component analysis.Neural Comput. 2004 Jun;16(6):1235-52. doi: 10.1162/089976604773717595. Neural Comput. 2004. PMID: 15130248
-
Optimizing an analytical dose calculation algorithm for fast 2D calculations.Z Med Phys. 2010;20(1):61-7. doi: 10.1016/j.zemedi.2009.12.001. Epub 2010 Jan 6. Z Med Phys. 2010. PMID: 20211425
-
Reduction, integration and emergence in biochemical networks.Biol Cell. 2004 Dec;96(9):719-25. doi: 10.1016/j.biolcel.2004.07.003. Biol Cell. 2004. PMID: 15567527 Review.
-
Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures.Cogn Neurodyn. 2014 Feb;8(1):1-15. doi: 10.1007/s11571-013-9267-8. Epub 2013 Sep 4. Cogn Neurodyn. 2014. PMID: 24465281 Free PMC article. Review.
Cited by
-
Augusta: From RNA-Seq to gene regulatory networks and Boolean models.Comput Struct Biotechnol J. 2024 Jan 20;23:783-790. doi: 10.1016/j.csbj.2024.01.013. eCollection 2024 Dec. Comput Struct Biotechnol J. 2024. PMID: 38312198 Free PMC article.
-
Linear and Nonlinear Effects in Connectedness Structure: Comparison between European Stock Markets.Entropy (Basel). 2022 Feb 21;24(2):303. doi: 10.3390/e24020303. Entropy (Basel). 2022. PMID: 35205597 Free PMC article.
-
Changes in Electroencephalography Activity in Response to Power Mobility Training: A Pilot Project.Physiother Can. 2020 Summer;72(3):260-270. doi: 10.3138/ptc-2018-0092. Physiother Can. 2020. PMID: 35110795 Free PMC article.
-
Information Theory for Agents in Artificial Intelligence, Psychology, and Economics.Entropy (Basel). 2021 Mar 6;23(3):310. doi: 10.3390/e23030310. Entropy (Basel). 2021. PMID: 33800724 Free PMC article. Review.
-
Information Entropy Suggests Stronger Nonlinear Associations between Hydro-Meteorological Variables and ENSO.Entropy (Basel). 2018 Jan 9;20(1):38. doi: 10.3390/e20010038. Entropy (Basel). 2018. PMID: 33265125 Free PMC article.