Empirical dynamic modeling explained

Empirical dynamic modeling (EDM) is a framework for analysis and prediction of nonlinear dynamical systems. Applications include population dynamics,^{[1] [2] [3] [4] [5] [6]} ecosystem service,^[7] medicine,^[8] neuroscience,^{[9] [10] [11]} dynamical systems,^{[12] [13] [14]} geophysics,^{[15] [16] [17]} and human-computer interaction.^[18] EDM was originally developed by Robert May and George Sugihara. It can be considered a methodology for data modeling, predictive analytics, dynamical system analysis, machine learning and time series analysis.

Description

Mathematical models have tremendous power to describe observations of real-world systems. They are routinely used to test hypothesis, explain mechanisms and predict future outcomes. However, real-world systems are often nonlinear and multidimensional, in some instances rendering explicit equation-based modeling problematic. Empirical models, which infer patterns and associations from the data instead of using hypothesized equations, represent a natural and flexible framework for modeling complex dynamics.

Donald DeAngelis and Simeon Yurek illustrated that canonical statistical models are ill-posed when applied to nonlinear dynamical systems.^[19] A hallmark of nonlinear dynamics is state-dependence: system states are related to previous states governing transition from one state to another. EDM operates in this space, the multidimensional state-space of system dynamics rather than on one-dimensional observational time series. EDM does not presume relationships among states, for example, a functional dependence, but projects future states from localised, neighboring states. EDM is thus a state-space, nearest-neighbors paradigm where system dynamics are inferred from states derived from observational time series. This provides a model-free representation of the system naturally encompassing nonlinear dynamics.

A cornerstone of EDM is recognition that time series observed from a dynamical system can be transformed into higher-dimensional state-spaces by time-delay embedding with Takens's theorem. The state-space models are evaluated based on in-sample fidelity to observations, conventionally with Pearson correlation between predictions and observations.

Methods

EDM is continuing to evolve. As of 2022, the main algorithms are Simplex projection,^[20] Sequential locally weighted global linear maps (S-Map) projection,^[21] Multivariate embedding in Simplex or S-Map,^[1] Convergent cross mapping (CCM),^[22] and Multiview Embeding,^[23] described below.

Nomenclature

Parameter	Description
E	embedding dimension
k	number of nearest neighbors
T _{p}	prediction interval
X\in \R	observed time series
y\in \R ^{E}	vector of lagged observations
\theta \geq 0	S-Map localization
X ^{E} _{t} = (X _{t,} X _{t-1} , ... , X _{t-E+1} ) \in \R ^{E}	lagged embedding vectors
\	v \		norm of v
N = \{N _{1,...,N} _{k\}}	list of nearest neighbors

Nearest neighbors are found according to:

NN (y, X, k) = \| X ^{E} _{N _{i}} - y\| \leq \| X ^{E} _{N _{j}} - y\| if 1 \leq i \leq j \leq k

Simplex

Simplex projection^{[20] [24] [25] [26]} is a nearest neighbor projection. It locates the

k nearest neighbors to the location in the state-space from which a prediction is desired. To minimize the number of free parameters k is typically set to E +1 defining an E +1 dimensional simplex in the state-space. The prediction is computed as the average of the weighted phase-space simplex projected Tp points ahead. Each neighbor is weighted proportional to their distance to the projection origin vector in the state-space.

1. Find

k nearest neighbor: N _{k} \gets NN (y, X, k)

1. Define the distance scale:

d \gets \| X ^{E} _{N _{1}} - y\|

1. Compute weights: For :

w _{i} \gets \exp ( - \| X ^{E} _{N _{i}} - y\| / d )

1. Average of state-space simplex:

\hat{y} \gets \sum ^{k} _{i =1} \left(w _{iX} ^{N _{i+T} _{p}} \right) / \sum ^{k} _{i =1} w _{i}

S-Map

S-Map^[21] extends the state-space prediction in Simplex from an average of the

E +1 nearest neighbors to a linear regression fit to all neighbors, but localised with an exponential decay kernel. The exponential localisation function is F(\theta) = exp ( - \theta d/D) , where d is the neighbor distance and D the mean distance. In this way, depending on the value of \theta , neighbors close to the prediction origin point have a higher weight than those further from it, such that a local linear approximation to the nonlinear system is reasonable. This localisation ability allows one to identify an optimal local scale, in-effect quantifying the degree of state dependence, and hence nonlinearity of the system.

Another feature of S-Map is that for a properly fit model, the regression coefficients between variables have been shown to approximate the gradient (directional derivative) of variables along the manifold.^[27] These Jacobians represent the time-varying interaction strengths between system variables.

1. Find

k nearest neighbor: N \gets NN (y, X, k)

1. Sum of distances:

D \gets (1)/(k) \sum ^{k} _{i =1} \| X ^{E} _{N _{i}} - y\|

1. Compute weights: For :

w _{i} \gets \exp ( - \theta \| X ^{E} _{N _{i}} - y\| / D )

1. Reweighting matrix:

W \gets diag (w _{i)}

1. Design matrix:

A \gets \begin{bmatrix} 1 & X _{N _{1}} & X _{N _{1-} 1} & ... & X _{N _{1} - E +1} \\ 1 & X _{N _{2}} & X _{N _{2-} 1} & ... & X _{N _{2} - E +1} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & X _{N _{k}} & X _{N _{k-} 1} & ... & X _{N _{k} - E +1} \end{bmatrix}

1. Weighted design matrix:

A \gets WA

1. Response vector at

Tp : b \gets \begin{bmatrix} X _{N _{1} + T _{p}} \\ X _{N _{2} + T _{p}} \\ \vdots \\ X _{N _{k} + T _{p}} \end{bmatrix}

1. Weighted response vector:

b \gets Wb

1. Least squares solution (SVD):

\hat{c} \gets argmin _{c} \| Ac - b \| ^{2} _{2}

1. Local linear model

\hat{c} is prediction: \hat{y} \gets \hat{c} _{0} + \sum ^{E\hat{c}} _{iy} _{i}

Multivariate Embedding

Multivariate Embedding^{[1] [12] [28]} recognizes that time-delay embeddings are not the only valid state-space construction. In Simplex and S-Map one can generate a state-space from observational vectors, or time-delay embeddings of a single observational time series, or both.

Convergent Cross Mapping

Convergent cross mapping (CCM)^[22] leverages a corollary to the Generalized Takens Theorem^[12] that it should be possible to cross predict or cross map between variables observed from the same system. Suppose that in some dynamical system involving variables

X and Y , X causes Y . Since X and Y belong to the same dynamical system, their reconstructions (via embeddings) M _{x} , and M _{y} , also map to the same system.

The causal variable

X leaves a signature on the affected variable Y , and consequently, the reconstructed states based on Y can be used to cross predict values of X . CCM leverages this property to infer causality by predicting X using the M _{y} library of points (or vice versa for the other direction of causality), while assessing improvements in cross map predictability as larger and larger random samplings of M _{y} are used. If the prediction skill of X increases and saturates as the entire M _{y} is used, this provides evidence that X is casually influencing Y .

Multiview Embedding

Multiview Embedding^[23] is a Dimensionality reduction technique where a large number of state-space time series vectors are combitorially assessed towards maximal model predictability.

Extensions

Extensions to EDM techniques include:

• Generalized Theorems for Nonlinear State Space Reconstruction^[12]
• Extended Convergent Cross Mapping^[13]
• Dynamic stability^[4]
• S-Map regularization^[29]
• Visual analytics with EDM^[30]
• Convergent Cross Sorting^[31]
• Expert system with EDM hybrid^[32]
• Sliding windows based on the extended convergent cross-mapping^[33]
• Empirical Mode Modeling^[17]
• Variable step sizes with bundle embedding^[34]
• Multiview distance regularised S-map^[35]

See also

• System dynamics
• Complex dynamics
• Nonlinear dimensionality reduction

Further reading

• Ecol Res . 2017. Empirical dynamic modeling for beginners. 785 - 796. Chang, CW., Ushio, M. & Hsieh, Ch.. 32. 6. 10.1007/s11284-017-1469-9. 4641225. free. 2017EcoR...32..785C. 2433/235326. free.
• ICES Journal of Marine Science. 2020. Frequently asked questions about nonlinear dynamics and empirical dynamic modelling. 1463 - 1479. Stephan B Munch, Antoine Brias, George Sugihara, Tanya L Rogers. 77. 10.1093/icesjms/fsz209. 4.
• Proceedings of the National Academy of Sciences. 2015. Equation-free modeling unravels the behavior of complex ecological systems. 3856 - 3857. Donald L. DeAngelis, Simeon Yurek. 112. 10.1073/pnas.1503154112. 13. 25829536. 4386356. free.

External links

Animations

Online books or lecture notes

• EDM Introduction. Introduction with video, examples and references.
• Geometrical theory of dynamical systems. Nils Berglund's lecture notes for a course at ETH at the advanced undergraduate level.
• Arxiv preprint server has daily submissions of (non-refereed) manuscripts in dynamical systems.

Research groups

• Sugihara Lab, Scripps Institution of Oceanography, University of California San Diego.

Notes and References

1. https://www.science.org/doi/10.1126/science.283.5407.1528Dixon, P. A., et al. 1999. Episodic fluctuations in larval supply. Science 283:1528–1530
2. https://www.pnas.org/content/112/13/E1569Hao Ye, Richard J. Beamish, Sarah M. Glaser, et al. 2015. Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) E1569-E1576; DOI: 10.1073/pnas.1417063112
3. https://www.pnas.org/content/110/16/6430Ethan R. Deyle, Michael Fogarty, Chih-hao Hsieh, et al. 2013. Proceedings of the National Academy of Sciences Apr 2013, 110 (16) 6430-6435; DOI: 10.1073/pnas.1215506110
4. https://doi.org/10.1038/nature25504Ushio, M., Hsieh, Ch., Masuda, R. et al., 2018. Fluctuating interaction network and time-varying stability of a natural fish community. Nature 554, 360–363
5. http://dx.doi.org/10.1098/rspb.2015.2258Deyle E.R., et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
6. https://onlinelibrary.wiley.com/doi/full/10.1111/ele.13532Tanya L. Rogers, Stephan B. Munch, Simon D. Stewart, Eric P. Palkovacs, Alfredo Giron-Nava, Shin-ichiro S. Matsuzaki, Celia C. Symons. Ecology Letters, 23 (8) August 2020, 1287-1297
7. https://doi.org/10.1371/journal.pone.0248910Park J., et al. 2021. Dynamics of Florida milk production and total phosphate in Lake Okeechobee. PLoS ONE 16(8): e0248910. doi:10.1371/journal.pone.0248910
8. https://www.pnas.org/content/pnas/93/6/2608.full.pdfGeorge Sugihara, Walter Allan, Daniel Sobel, and Kenneth D. Allan, 1996. Nonlinear control of heart rate variability in human infants. Proc. Natl. Acad. Sci. USA. Vol. 93, pp. 2608-2613, March 1996. Medical Sciences
9. https://doi.org/10.1016/j.nicl.2014.12.005McBride, J. C., et al. Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease. Neuroimage-Clinical 7:258–265 (2015)
10. https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004537Tajima S, Yanagawa T, Fujii N, Toyoizumi T (2015) Untangling Brain-Wide Dynamics in Consciousness by Cross-Embedding. PLoS Comput Biol 11(11): e1004537. https://doi.org/10.1371/journal.pcbi.1004537
11. https://ieeexplore.ieee.org/document/9359204W. Watanakeesuntorn et al., "Massively Parallel Causal Inference of Whole Brain Dynamics at Single Neuron Resolution," 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), 2020, pp. 196-205, doi: 10.1109/ICPADS51040.2020.00035
12. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0018295 Deyle ER, Sugihara G (2011) Generalized Theorems for Nonlinear State Space Reconstruction. PLoS ONE 6(3): e18295. doi:10.1371/journal.pone.0018295
13. https://www.nature.com/articles/srep14750Ye, H., Deyle, E., Gilarranz, L. et al., 2015. Distinguishing time-delayed causal interactions using convergent cross mapping. Sci Rep 5, 14750 (2015). doi:10.1038/srep14750
14. https://www.nature.com/articles/s41559-019-0879-1Cenci, S., Saavedra, S. Non-parametric estimation of the structural stability of non-equilibrium community dynamics. Nat Ecol Evol 3, 912–918 (2019). https://doi.org/10.1038/s41559-019-0879-1
15. https://doi.org/10.1073/pnas.1420291112Tsonis A. A., et al. Dynamical evidence for causality between galactic cosmic rays and interannual variation in global temperature. Proc Natl Acad Sci 112(11):3253–3256 (2015).
16. https://www.nature.com/articles/nclimate2568Nes EH Van, et al. Causal feedbacks in climate change. Nat Clim Chang 5(5):445–448 (2015)
17. https://doi.org/10.1007/s11071-022-07311-yPark, J., et al. Empirical mode modeling. Nonlinear Dyn (2022). https://doi.org/10.1007/s11071-022-07311-y
18. van Berkel . Niels . Dennis . Simon . Zyphur . Michael . Li . Jinjing . Heathcote . Andrew . Kostakos . Vassilis . 2021-07-04 . Modeling interaction as a complex system . Human–Computer Interaction . 36 . 4 . 279–305 . 10.1080/07370024.2020.1715221 . 211267275 . 0737-0024. 11343/247884 . free .
19. https://www.pnas.org/content/112/13/3856Donald L. DeAngelis, Simeon Yurek, 2015, Equation-free modeling unravels the behavior of complex ecological systems. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) 3856-3857; DOI: 10.1073/pnas.1503154112
20. https://www.nature.com/articles/344734a0 Sugihara G. and May R., 1990. Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344:734–741
21. https://royalsocietypublishing.org/doi/abs/10.1098/rsta.1994.0106 Sugihara G., 1994. Nonlinear forecasting for the classification of natural time series. Philosophical Transactions: Physical Sciences and Engineering, 348 (1688) : 477–495
22. https://www.science.org/doi/10.1126/science.1227079 Sugihara G., May R., Ye H., et al. 2012. Detecting Causality in Complex Ecosystems. Science 338:496-500
23. https://www.science.org/doi/10.1126/science.aag0863 Ye H., and G. Sugihara, 2016. Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality. Science 353:922–925
24. https://doi.org/10.1007/BFb0091924 Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence (pp. 366–381). Springer.
25. https://doi.org/https://doi.org/10.1016/0167-2789(89)90074-2 Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenomena, 35(3), 335–356.
26. https://doi.org/10.1016/S0167-2789(98)00089-X Judd, K., & Mees, A. (1998). Embedding as a modeling problem. Physica D: Nonlinear Phenomena, 120(3), 273–286.
27. http://dx.doi.org/10.1098/rspb.2015.2258Deyle ER. et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
28. https://doi.org/10.1007/BF01053745 Sauer, T., Yorke, J. A., & Casdagli, M. (1991). Embedology. Journal of Statistical Physics, 65(3), 579–616
29. https://doi.org/10.1111/2041-210X.13150Cenci S, Sugihara G, Saavedra S, 2019. Regularized S-map for inference and forecasting with noisy ecological time series, METHODS IN ECOLOGY AND EVOLUTION, 10 (5), 650-660
30. https://www.computer.org/csdl/journal/tg/2021/02/09216532/1nJsFIg64us Hiroaki Natsukawa, et al. 2021. A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling. IEEE Transactions on Visualization and Computer Graphics. Feb. 2021, 506-516, vol. 27DOI: 10.1109/TVCG.2020.3028956
31. https://doi.org/10.1038/s41598-021-98864-2 Breston, L., Leonardis, E.J., Quinn, L.K. et al. 2021. Convergent cross sorting for estimating dynamic coupling. Sci Rep 11, 20374 (2021). doi:10.1038/s41598-021-98864-2
32. https://doi.org/10.1073/pnas.2102466119 Deyle E. R. et al. A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures. PNAS Vol. 119, No. 26 (2022).
33. https://doi.org/10.1007/s11071-021-06362-x Ge, X., Lin, A. Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping. Nonlinear Dyn 104, 1753–1765 (2021). https://doi.org/10.1007/s11071-021-06362-x
34. https://www.sciencedirect.com/science/article/abs/pii/S0304380022000680 Bethany Johnson, Stephan B. Munch. 2022. An empirical dynamic modeling framework for missing or irregular samples. Ecological Modelling, Volume 468, June 2022, 109948.
35. https://doi.org/10.1111/ele.13897 Chang, C.-W., Miki, T., Ushio, M., et al. (2021) Reconstructing large interaction networks from empirical time series data. Ecology Letters, 24, 2763– 2774. https://doi.org/10.1111/ele.13897