Recruitment learning of boolean functions in sparse random networks

This work presents a new class of neural network models constrained by biological levels of sparsity and weight-precision, and employing only local weight updates. Concept learning is accomplished through the rapid recruitment of existing network knowledge - complex knowledge being realised as a combination of existing basis concepts. Prior network knowledge is here obtained through the random generation of feedforward networks, with the resulting concept library tailored through distributional bias to suit a particular target class. Learning is exclusively local - through supervised Hebbian and Winnow updates - avoiding the necessity for backpropagation of error and allowing remarkably rapid learning. The approach is demonstrated upon concepts of varying difficulty, culminating in the well-known Monks and LED benchmark problems.     

SIR dynamics in random networks with heterogeneous connectivity

Random networks with specified degree distributions have been proposed as realistic models of population structure, yet the problem of dynamically modeling SIR-type epidemics in random networks remains complex. I resolve this dilemma by showing how the SIR dynamics can be modeled with a system of three nonlinear ODE’s. The method makes use of the probability generating function (PGF) formalism for representing the degree distribution of a random network and makes use of network-centric quantities such as the number of edges in a well-defined category rather than node-centric quantities such as the number of infecteds or susceptibles. The PGF provides a simple means of translating between network and node-centric variables and determining the epidemic incidence at any time. The theory also provides a simple means of tracking the evolution of the degree distribution among susceptibles or infecteds. The equations are used to demonstrate the dramatic effects that the degree distribution plays on the final size of an epidemic as well as the speed with which it spreads through the population. Power law degree distributions are observed to generate an almost immediate expansion phase yet have a smaller final size compared to homogeneous degree distributions such as the Poisson. The equations are compared to stochastic simulations, which show good agreement with the theory. Finally, the dynamic equations provide an alternative way of determining the epidemic threshold where large-scale epidemics are expected to occur, and below which epidemic behavior is limited to finite-sized outbreaks.      

Activity dynamics and propagation of synchronous spiking in locally connected random networks

Random network models have been a popular tool for investigating cortical network dynamics. On the scale of roughly a cubic millimeter of cortex, containing about 100,000 neurons, cortical anatomy suggests a more realistic architecture. In this locally connected random network, the connection probability decreases in a Gaussian fashion with the distance between neurons. Here we present three main results from a simulation study of the activity dynamics in such networks. First, for a broad range of parameters these dynamics exhibit a stationary state of asynchronous network activity with irregular single-neuron spiking. This state can be used as a realistic model of ongoing network activity. Parametric dependence of this state and the nature of the network dynamics in other regimes are described. Second, a synchronous excitatory stimulus to a fraction of the neurons results in a strong activity response that easily dominates the network dynamics. And third, due to that activity response an embedding of a divergent-convergent feed-forward subnetwork (as in synfire chains) does not naturally lead to a stable propagation of synchronous activity in the subnetwork; this is in contrast to our earlier findings in isolated subnetworks of that type. Possible mechanisms for stabilizing the interplay of volleys of synchronous spikes and network dynamics by specific learning rules or generalizations of the subnetworks are discussed.     

Bayesian learning of sparse gene regulatory networks

Differential equations (DEs) have been the most widespread formalism for gene regulatory network (GRN) modeling, as they offer natural interpretation of biological processes, easy elucidation of gene relationships, and the capability of using efficient parameter estimation methods. However, an important limitation of DEs is their requirement of O(d(2)) parameters where d is the number of genes modeled, which often causes over-parameterization for large d, leading to the over-fitting of data and dense parameter sets that are hard to interpret. This paper presents the first effort to address the over-parameterization problem by applying the sparse Bayesian learning (SBL) method to sparsify the GRN model of DEs. SBL operates on the parsimony principle, with the objective to reduce the number of effective parameters by driving the redundant parameters to zero. The resulting sparse parameter set offers three important advantages for GRN inference: first, the inferred GRNs are more plausible, since the biological counterparts are known to be sparse; second, gene relationships can be more easily elucidated from sparse sets than from dense sets; and third, the solutions become more optimal and consistent, due to the reduction in the volume of solution space. Experiments are conducted on the yeast Saccharomyces cerevisiae time-series gene expression data, in which known regulatory events related to the cell cycle G1/S phase are reliably reproduced.     

On the sparse reconstruction of gene networks

We discuss a heuristic method for the sparse reconstruction of gene networks. The method is based on iterative greedy algorithms, and uses gene expression data from microarray experiments. Also, we show numerically that the greedy algorithms are able to give good approximative solutions to the sparse reconstruction problem even in the presence of significant levels of noise.     

Augmented sparse reconstruction of protein signaling networks

The problem of reconstructing and identifying intracellular protein signaling and biochemical networks is of critical importance in biology. We propose a mathematical approach called augmented sparse reconstruction for the identification of links among nodes of ordinary differential equation (ODE) networks, given a small set of observed trajectories with various initial conditions. As a test case, the method is applied to the epidermal growth factor receptor (EGFR) driven signaling cascade, a well-studied and clinically important signaling network. Our method builds a system of representation from a collection of trajectory integrals, selectively attenuating blocks of terms in the representation. The system of representation is then augmented with random vectors, and l₁ minimization is used to find sparse representations for the dynamical interactions of each node. After showing the performance of our method on a model of the EGFR protein network, we sketch briefly the potential future therapeutic applications of this approach.     

Connectivity in random networks

Applications of randomly connected networks are reviewed briefly. The connectivity of a random network has been defined in a variety of ways including output connectivity, total or network connectivity, connectance, expected path length and radius. One or more of these definitions may prove more convenient in a given experimental system. Interrelations among these definitions are derived and displayed and asymptotic results provided in the form of two theorems. Computer simulations were used to explore the range of application of these asymptotic approximations. The results were used to determine the output connectivity of the neurons of the brain.     

A note on a paper by Erik Volz: SIR dynamics in random networks

Recent work by Volz (J Math Biol 56:293–310, 2008) has shown how to calculate the growth and eventual decay of an SIR epidemic on a static random network, assuming infection and recovery each happen at constant rates. This calculation allows us to account for effects due to heterogeneity and finiteness of degree that are neglected in the standard mass-action SIR equations. In this note we offer an alternate derivation which arrives at a simpler—though equivalent—system of governing equations to that of Volz. This new derivation is more closely connected to the underlying physical processes, and the resulting equations are of comparable complexity to the mass-action SIR equations. We further show that earlier derivations of the final size of epidemics on networks can be reproduced using the same approach, thereby providing a common framework for calculating both the dynamics and the final size of an epidemic spreading on a random network. Under appropriate assumptions these equations reduce to the standard SIR equations, and we are able to estimate the magnitude of the error introduced by assuming the SIR equations.     

The dynamics of memory retrieval in hierarchical networks

Memory retrieval is of central importance to a wide variety of brain functions. To understand the dynamic nature of memory retrieval and its underlying neurophysiological mechanisms, we develop a biologically plausible spiking neural circuit model, and demonstrate that free memory retrieval of sequences of events naturally arises from the model under the condition of excitation-inhibition (E/I) balance. Using the mean-field model of the spiking circuit, we gain further theoretical insights into how such memory retrieval emerges. We show that the spiking neural circuit model quantitatively reproduces several salient features of free memory retrieval, including its semantic proximity effect and log-normal distributions of inter-retrieval intervals. In addition, we demonstrate that our model can serve as a platform to examine memory retrieval deficits observed in neuropsychiatric diseases such as Parkinson’s and Alzheimer’s diseases. Furthermore, our model allows us to make novel and experimentally testable predictions, such as the prediction that there are long-range correlations in the sequences of retrieved items.     

The circular topology of rhythm in asynchronous random Boolean networks

The analysis of previously evolved rhythmic asynchronous random Boolean networks [Biosystems 59 (2001) 185] reveals common topological characteristics indicating that rhythm originates from a circular functional structure. The rhythm generating core of the network has the form of a closed ring which operates as a synchronisation substrate by supporting a travelling wave of state change; the size of the ring corresponds well with the period of oscillation. The remaining nodes in the network are either stationary or follow the activity of the ring without feeding back into it so as to form a coherent whole. Rings are typically formed early on in the evolutionary search process. Alternatively, long chains of nodes are favoured before they close upon themselves to stabilize. Analysis of asynchronous networks with de-correlated (non-rhythmic, non-stationary) attractors reveals no such common topological characteristics. These results have been obtained using statistical analysis and a specifically developed bottom-up pruning algorithm. This algorithm works from local interactions to global configuration by eliminating redundant links. The suitability of the algorithm has been confirmed by both numerical and single lesion analysis. The ring topology solution for the generation of rhythm implies that it will be harder to evolve rhythmic networks for big sizes and small periods and for bigger number of connections per node. These trends are confirmed empirically. Finally, the identified mechanisms are utilised to handcraft rhythmic networks of different periods showing that a low number of connections suffices for a large variety of rhythms. Random asynchronous update forces the evolved solutions to be highly robust maintaining their performance in the presence of intrinsic noise. The biological implications of such robust designs for molecular clocks are discussed.     

Effects of Hebbian learning on the dynamics and structure of random networks with inhibitory and excitatory neurons

The aim of the present paper is to study the effects of Hebbian learning in random recurrent neural networks with biological connectivity, i.e. sparse connections and separate populations of excitatory and inhibitory neurons. We furthermore consider that the neuron dynamics may occur at a (shorter) time scale than synaptic plasticity and consider the possibility of learning rules with passive forgetting. We show that the application of such Hebbian learning leads to drastic changes in the network dynamics and structure. In particular, the learning rule contracts the norm of the weight matrix and yields a rapid decay of the dynamics complexity and entropy. In other words, the network is rewired by Hebbian learning into a new synaptic structure that emerges with learning on the basis of the correlations that progressively build up between neurons. We also observe that, within this emerging structure, the strongest synapses organize as a small-world network. The second effect of the decay of the weight matrix spectral radius consists in a rapid contraction of the spectral radius of the Jacobian matrix. This drives the system through the "edge of chaos" where sensitivity to the input pattern is maximal. Taken together, this scenario is remarkably predicted by theoretical arguments derived from dynamical systems and graph theory.     

Unraveling spurious properties of interaction networks with tailored random networks

We investigate interaction networks that we derive from multivariate time series with methods frequently employed in diverse scientific fields such as biology, quantitative finance, physics, earth and climate sciences, and the neurosciences. Mimicking experimental situations, we generate time series with finite length and varying frequency content but from independent stochastic processes. Using the correlation coefficient and the maximum cross-correlation, we estimate interdependencies between these time series. With clustering coefficient and average shortest path length, we observe unweighted interaction networks, derived via thresholding the values of interdependence, to possess non-trivial topologies as compared to Erd?s-Rényi networks, which would indicate small-world characteristics. These topologies reflect the mostly unavoidable finiteness of the data, which limits the reliability of typically used estimators of signal interdependence. We propose random networks that are tailored to the way interaction networks are derived from empirical data. Through an exemplary investigation of multichannel electroencephalographic recordings of epileptic seizures--known for their complex spatial and temporal dynamics--we show that such random networks help to distinguish network properties of interdependence structures related to seizure dynamics from those spuriously induced by the applied methods of analysis.     

Efficient computation via sparse coding in electrosensory neural networks

The electric sense combines spatial aspects of vision and touch with temporal features of audition. Its accessible neural architecture shares similarities with mammalian sensory systems and allows for recordings from successive brain areas to test hypotheses about neural coding. Further, electrosensory stimuli encountered during prey capture, navigation, and communication, can be readily synthesized in the laboratory. These features enable analyses of the neural circuitry that reveal general principles of encoding and decoding, such as segregation of information into separate streams and neural response sparsification. A systems level understanding arises via linkage between cellular differentiation and network architecture, revealed by in vitro and in vivo analyses, while computational modeling reveals how single cell dynamics and connectivity shape the sparsification process.     

On the dynamics of random Boolean networks subject to noise: Attractors, ergodic sets and cell types

The asymptotic dynamics of random Boolean networks subject to random fluctuations is investigated. Under the influence of noise, the system can escape from the attractors of the deterministic model, and a thorough study of these transitions is presented. We show that the dynamics is more properly described by sets of attractors rather than single ones. We generalize here a previous notion of ergodic sets, and we show that the Threshold Ergodic Sets so defined are robust with respect to noise and, at the same time, that they do not suffer from a major drawback of ergodic sets. The system jumps from one attractor to another of the same Threshold Ergodic Set under the influence of noise, never leaving it. By interpreting random Boolean networks as models of genetic regulatory networks, we also propose to associate cell types to Threshold Ergodic Sets rather than to deterministic attractors or to ergodic sets, as it had been previously suggested. We also propose to associate cell differentiation to the process whereby a Threshold Ergodic Set composed by several attractors gives rise to another one composed by a smaller number of attractors. We show that this approach accounts for several interesting experimental facts about cell differentiation, including the possibility to obtain an induced pluripotent stem cell from a fully differentiated one by overexpressing some of its genes.     

Guiding the self-organization of random Boolean networks

Random Boolean networks (RBNs) are models of genetic regulatory networks. It is useful to describe RBNs as self-organizing systems to study how changes in the nodes and connections affect the global network dynamics. This article reviews eight different methods for guiding the self-organization of RBNs. In particular, the article is focused on guiding RBNs toward the critical dynamical regime, which is near the phase transition between the ordered and dynamical phases. The properties and advantages of the critical regime for life, computation, adaptability, evolvability, and robustness are reviewed. The guidance methods of RBNs can be used for engineering systems with the features of the critical regime, as well as for studying how natural selection evolved living systems, which are also critical.     

Biologically-informed neural networks guide mechanistic modeling from sparse experimental data

Biologically-informed neural networks (BINNs), an extension of physics-informed neural networks [1], are introduced and used to discover the underlying dynamics of biological systems from sparse experimental data. In the present work, BINNs are trained in a supervised learning framework to approximate in vitro cell biology assay experiments while respecting a generalized form of the governing reaction-diffusion partial differential equation (PDE). By allowing the diffusion and reaction terms to be multilayer perceptrons (MLPs), the nonlinear forms of these terms can be learned while simultaneously converging to the solution of the governing PDE. Further, the trained MLPs are used to guide the selection of biologically interpretable mechanistic forms of the PDE terms which provides new insights into the biological and physical mechanisms that govern the dynamics of the observed system. The method is evaluated on sparse real-world data from wound healing assays with varying initial cell densities [2].     

Identification of dynamic mass-action biochemical reaction networks using sparse Bayesian methods

Identifying the reactions that govern a dynamical biological system is a crucial but challenging task in systems biology. In this work, we present a data-driven method to infer the underlying biochemical reaction system governing a set of observed species concentrations over time. We formulate the problem as a regression over a large, but limited, mass-action constrained reaction space and utilize sparse Bayesian inference via the regularized horseshoe prior to produce robust, interpretable biochemical reaction networks, along with uncertainty estimates of parameters. The resulting systems of chemical reactions and posteriors inform the biologist of potentially several reaction systems that can be further investigated. We demonstrate the method on two examples of recovering the dynamics of an unknown reaction system, to illustrate the benefits of improved accuracy and information obtained.