Modeling of dynamical systems through deep learning

This review presents a modern perspective on dynamical systems in the context of current goals and open challenges. In particular, our review focuses on the key challenges of discovering dynamics from data and finding data-driven representations that make nonlinear systems amenable to linear analysis. We explore various challenges in modern dynamical systems, along with emerging techniques in data science and machine learning to tackle them. The two chief challenges are (1) nonlinear dynamics and (2) unknown or partially known dynamics. Machine learning is providing new and powerful techniques for both challenges. Dimensionality reduction methods are used for projecting dynamical methods in reduced form, and these methods perform computational efficiency on real-world data. Data-driven models drive to discover the governing equations and give laws of physics. The identification of dynamical systems through deep learning techniques succeeds in inferring physical systems. Machine learning provides advanced new and powerful algorithms for nonlinear dynamics. Advanced deep learning methods like autoencoders, recurrent neural networks, convolutional neural networks, and reinforcement learning are used in modeling of dynamical systems.     

Towards dynamical system models of language-related brain potentials

Event-related brain potentials (ERP) are important neural correlates of cognitive processes. In the domain of language processing, the N400 and P600 reflect lexical-semantic integration and syntactic processing problems, respectively. We suggest an interpretation of these markers in terms of dynamical system theory and present two nonlinear dynamical models for syntactic computations where different processing strategies correspond to functionally different regions in the system’s phase space.

A discrete time neural network model with spiking neurons

We derive rigorous results describing the asymptotic dynamics of a discrete time model of spiking neurons introduced in Soula et al. (Neural Comput. 18, 1, 2006). Using symbolic dynamic techniques we show how the dynamics of membrane potential has a one to one correspondence with sequences of spikes patterns (“raster plots”). Moreover, though the dynamics is generically periodic, it has a weak form of initial conditions sensitivity due to the presence of a sharp threshold in the model definition. As a consequence, the model exhibits a dynamical regime indistinguishable from chaos in numerical experiments.      

A dynamical systems view of neuroethology: Uncovering stateful computation in natural behaviors

State-dependent computation is key to cognition in both biological and artificial systems. Alan Turing recognized the power of stateful computation when he created the Turing machine with theoretically infinite computational capacity in 1936. Independently, by 1950, ethologists such as Tinbergen and Lorenz also began to implicitly embed rudimentary forms of state-dependent computation to create qualitative models of internal drives and naturally occurring animal behaviors. Here, we reformulate core ethological concepts in explicitly dynamical systems terms for stateful computation. We examine, based on a wealth of recent neural data collected during complex innate behaviors across species, the neural dynamics that determine the temporal structure of internal states. We will also discuss the degree to which the brain can be hierarchically partitioned into nested dynamical systems and the need for a multi-dimensional state-space model of the neuromodulatory system that underlies motivational and affective states.     

Global stability for cholera epidemic models

Cholera is a water and food borne infectious disease caused by the gram-negative bacterium, Vibrio cholerae. Its dynamics are highly complex owing to the coupling among multiple transmission pathways and different factors in pathogen ecology. Although various mathematical models and clinical studies published in recent years have made important contribution to cholera epidemiology, our knowledge of the disease mechanism remains incomplete at present, largely due to the limited understanding of the dynamics of cholera. In this paper, we conduct global stability analysis for several deterministic cholera epidemic models. These models, incorporating both human population and pathogen V. cholerae concentration, constitute four-dimensional non-linear autonomous systems where the classical Poincaré-Bendixson theory is not applicable. We employ three different techniques, including the monotone dynamical systems, the geometric approach, and Lyapunov functions, to investigate the endemic global stability for several biologically important cases. The analysis and results presented in this paper make building blocks towards a comprehensive study and deeper understanding of the fundamental mechanism in cholera dynamics.     

Robert Rosen in the age of systems biology

The widespread use of the term Systems Biology (SB) signals a welcome recognition that organisms must be understood as integrated systems. Although just what this is taken to mean varies from one group to another, it generally implies a focus on biological functions and processes rather than on biological parts and a reliance on mathematical modeling to arrive at an understanding of these biological processes based on biological observations or measurements. SB, thus, falls directly in the line of reflection carried out by Robert Rosen throughout his work. In the present article, we briefly introduce the various currents of SB and then point out several ways Rosen's work can be used to avoid certain pitfalls associated with the use of dynamical systems models for the study of complex systems, as well as to inspire a productive path forward based on loosely organized cooperation among dispersed laboratories.     

Dynamical Systems and Depression: A Framework for Theoretical Perspectives

The theory of dynamical systems allows one to describe the change in a system's macroscopic behavior as a bifurcation in the underlying dynamics. We show here, from the example of depressive syndrome, the existence of a correspondence between clinical and electro-physiological dimensions and the association between clinical remission and brain dynamics reorganization (i.e. bifurcation). On the basis of this experimental study, we discuss the interest of such results concerning the question of normality versus pathology in psychiatry and the relationship between mind and brain.     

On the sensitive dependence on initial conditions of the dynamics of networks of spiking neurons

We have previously formulated an abstract dynamical system for networks of spiking neurons and derived a formal result that identifies the criterion for its dynamics, without inputs, to be “sensitive to initial conditions”. Since formal results are applicable only to the extent to which their assumptions are valid, we begin this article by demonstrating that the assumptions are indeed reasonable for a wide range of networks, particularly those that lack overarching structure. A notable aspect of the criterion is the finding that sensitivity does not necessarily arise from randomness of connectivity or of connection strengths, in networks. The criterion guides us to cases that decouple these aspects: we present two instructive examples of networks, one with random connectivity and connection strengths, yet whose dynamics is insensitive, and another with structured connectivity and connection strengths, yet whose dynamics is sensitive. We then argue based on the criterion and the gross electrophysiology of the cortex that the dynamics of cortical networks ought to be almost surely sensitive under conditions typically found there. We supplement this with two examples of networks modeling cortical columns with widely differing qualitative dynamics, yet with both exhibiting sensitive dependence. Next, we use the criterion to construct a network that undergoes bifurcation from sensitive dynamics to insensitive dynamics when the value of a control parameter is varied. Finally, we extend the formal result to networks driven by stationary input spike trains, deriving a superior criterion than previously reported. Action Editor: John Rinzel     

An ecological framework linking scales across space and time based on self-thinning

Scaling up from measurements made at small spatial and short temporal scales is a central challenge in the ecological and related sciences, where predictions at larger scales and over long time periods are required. It involves two quite distinct aspects: a formulation of a theoretical framework for calculating space-time averages, and an acquisition of data to support that framework. In this paper, we address the theoretical part of the question, and although our primary motivation was an understanding of carbon accounting our formulation is more general. To that end, we adopt a dynamical systems approach, and incorporate a new dynamical formulation of self-thinning. We show how to calculate rates of change for total (and average) plant dry mass, volume, and carbon, in terms of the properties of the individual plants. The results emphasize how local scale statistics (such as, variation in the size of individuals) lead to nonlinear variation at larger scales. Further, we describe how regular and stochastic disturbance can be readily incorporated into this framework. It is shown that stochastic disturbance at patch-scales, results in (to first approximation) regular disturbance at ecosystem scales, and hence can be formulated as such. We conclude that a dynamical formulation of self-thinning can be used as a generic framework for scaling ecological processes in space and time.     

Dynamical phase coexistence: A simple solution to the “savanna problem”

We introduce the concept of dynamical phase coexistence to provide a simple solution for a long-standing problem in theoretical ecology, the so-called “savanna problem”. The challenge is to understand why in savanna ecosystems trees and grasses coexist in a robust way with large spatiotemporal variability. We propose a simple model, a variant of the contact process (CP), which includes two key extra features: varying external (environmental/rainfall) conditions and tree age. The system fluctuates locally between a woodland and a grassland phase, corresponding to the active and absorbing phases of the underlying pure contact process. This leads to a highly variable stable phase characterized by patches of the woodland and grassland phases coexisting dynamically. We show that the mean time to tree extinction under this model increases as a power-law of system size and can be of the order of 10,000,000 years in even moderately sized savannas. Finally, we demonstrate that while local interactions among trees may influence tree spatial distribution and the order of the transition between woodland and grassland phases, they do not affect dynamical coexistence. We expect dynamical coexistence to be relevant in other contexts in physics, biology or the social sciences.     

On the robustness of update schedules in Boolean networks

Deterministic Boolean networks have been used as models of gene regulation and other biological networks. One key element in these models is the update schedule, which indicates the order in which states are to be updated. We study the robustness of the dynamical behavior of a Boolean network with respect to different update schedules (synchronous, block-sequential, sequential), which can provide modelers with a better understanding of the consequences of changes in this aspect of the model. For a given Boolean network, we define equivalence classes of update schedules with the same dynamical behavior, introducing a labeled graph which helps to understand the dependence of the dynamics with respect to the update, and to identify interactions whose timing may be crucial for the presence of a particular attractor of the system. Several other results on the robustness of update schedules and of dynamical cycles with respect to update schedules are presented. Finally, we prove that our equivalence classes generalize those found in sequential dynamical systems.     

Monotone and near-monotone biochemical networks

Monotone subsystems have appealing properties as components of larger networks, since they exhibit robust dynamical stability and predictability of responses to perturbations. This suggests that natural biological systems may have evolved to be, if not monotone, at least close to monotone in the sense of being decomposable into a “small” number of monotone components, In addition, recent research has shown that much insight can be attained from decomposing networks into monotone subsystems and the analysis of the resulting interconnections using tools from control theory. This paper provides an expository introduction to monotone systems and their interconnections, describing the basic concepts and some of the main mathematical results in a largely informal fashion. Supported in part by NSF Grants DMS-0504557 and DMS-0614371.     

Coupled multi-component systems: A simple membrane model

We present initial results regarding the existence, stability and interactionof linear and nonlinear vibrational modes in a system of two coupled, onedimensional lattices with unequal numbers of masses. The effects on thesenonlinear modes of coupling a near continuum system to a discrete systemusing a nonlinear coupling are examined. This numerical model is a firststep towards investigating the dynamical behavior of a flexible sheetcoupled nonlinearly to a semi-rigid support, a system which couldconceivably represent a biological cell membrane with a supporting proteinnetwork. General implications for the dynamical behavior of continuumsystems coupled nonlinearly to discrete systems are introduced.     

Dynamical entropy via entropy of non-random matrices: Application to stability and complexity in modelling ecosystems

In the present paper we have first introduced a measure of dynamical entropy of an ecosystem on the basis of the dynamical model of the system. The dynamical entropy which depends on the eigenvalues of the community matrix of the system leads to a consistent measure of complexity of the ecosystem to characterize the dynamical behaviours such as the stability, instability and periodicity around the stationary states of the system. We have illustrated the theory with some model ecosystems.     

Bilinear state space systems for nonlinear dynamical modelling

Summary We discuss the identification of multiple input, multiple output, discrete-time bilinear state space systems. We consider two identification problems. In the first case, the input to the system is a measurable white noise sequence. We show that it is possible to identify the system by solving a nonlinear optimization problem. The number of parameters in this optimization problem can be reduced by exploiting the principle of separable least squares. A subspace-based algorithm can be used to generate initial estimates for this nonlinear identification procedure. In the second case, the input to the system is not measurable. This makes it a much more difficult identification problem than the case with known inputs. At present, we can only solve this problem for a certain class of single input, single output bilinear state space systems, namely bilinear systems in phase variable form.     

Co-existent activity patterns in inhibitory neuronal networks with short-term synaptic depression

A network of two neurons mutually coupled through inhibitory synapses that display short-term synaptic depression is considered. We show that synaptic depression expands the number of possible activity patterns that the network can display and allows for co-existence of different patterns. Specifically, the network supports different types of n-m anti-phase firing patterns, where one neuron fires n spikes followed by the other neuron firing m spikes. When maximal synaptic conductances are identical, n-n anti-phase firing patterns are obtained and there are conductance intervals over which different pairs of these solutions co-exist. The multitude of n-m anti-phase patterns and their co-existence are not found when the synapses are non-depressing. Geometric singular perturbation methods for dynamical systems are applied to the original eight-dimensional model system to derive a set of one-dimensional conditions for the existence and co-existence of different anti-phase solutions. The generality and validity of these conditions are demonstrated through numerical simulations utilizing the Hodgkin-Huxley and Morris-Lecar neuronal models.     

A dynamical systems perspective on the relationship between symbolic and non-symbolic computation

It has been claimed that connectionist (artificial neural network) models of language processing, which do not appear to employ “rules”, are doing something different in kind from classical symbol processing models, which treat “rules” as atoms (e.g., McClelland and Patterson in Trends Cogn Sci 6(11):465–472, 2002). This claim is hard to assess in the absence of careful, formal comparisons between the two approaches. This paper formally investigates the symbol-processing properties of simple dynamical systems called affine dynamical automata, which are close relatives of several recurrent connectionist models of language processing (e.g., Elman in Cogn Sci 14:179–211, 1990). In line with related work (Moore in Theor Comput Sci 201:99–136, 1998; Siegelmann in Neural networks and analog computation: beyond the Turing limit. Birkhäuser, Boston, 1999), the analysis shows that affine dynamical automata exhibit a range of symbol processing behaviors, some of which can be mirrored by various Turing machine devices, and others of which cannot be. On the assumption that the Turing machine framework is a good way to formalize the “computation” part of our understanding of classical symbol processing, this finding supports the view that there is a fundamental “incompatibility” between connectionist and classical models (see Fodor and Pylyshyn 1988; Smolensky in Behav Brain Sci 11(1):1–74, 1988; beim Graben in Mind Matter 2(2):29--51,2004b). Given the empirical successes of connectionist models, the more general, super-Turing framework is a preferable vantage point from which to consider cognitive phenomena. This vantage may give us insight into ill-formed as well as well-formed language behavior and shed light on important structural properties of learning processes.     

Dynamical systems analysis of arterial blood pressure signals in relation to heart rate fluctuations in chick embryos

We attempted a new approach based on a modern dynamical system theory to reconstruct the arterial blood pressure signals in relation to heart rate fluctuations of developing chick embryos. The dynamical systems approach in general is to model a phenomenon that is presented by a single time series record and approximate the dynamical property (e.g. heart rate fluctuations) of a system based only on information contained in a single-variable (arterial blood pressure) of the system. The time-series data of the arterial blood pressure was reconstructed in 3-dimensional space to draw characteristic orbits. Since the reconstructed orbits of the blood pressure should retain information contained in the pressure signals, we attempted to derive instantaneous heart rate (IHR) from the reconstructed orbits. The derived IHR presenting HR fluctuations coincided well with the IHR obtained conventionally from the peak-to-peak time intervals of the maximum blood pressure. Movements of the reconstructed orbits of the arterial blood pressure in 3-dimensional space reflected HR fluctuations (i.e. transient decelerations and accelerations).