Extremely Dilute Modular Neuronal Networks: Neocortical Memory Retrieval Dynamics

A model of columnar networks of neocortical association areas is studied. The neuronal network is composed of many Hebbian autoassociators, or modules, each of which interacts with a relatively small number of the others, randomly chosen. Any module encodes and stores a number of elementary percepts, or features. Memory items, or patterns, are peculiar combinations of features sparsely distributed over the multi-modular network. Any feature stored in any module can be involved in several of the stored patterns; feature-sharing is in fact source of local ambiguities and, consequently, a potential cause of erroneous memory retrieval spreading through the model network in pattern completion tasks.The memory retrieval dynamics of the large modular autoassociator is investigated by combining mathematical analysis and numerical simulations. An oscillatory retrieval process is proposed that is very efficient in overcoming feature-sharing drawbacks; it requires a mechanism that modulates the robustness of local attractors to noise, and neuronal activity sparseness such that quiescent and active modules are about equally noisy to any post-synaptic module.Moreover, it is shown that statistical correlation between 'kinds' of features across the set of memory patterns can be exploited to obtain a more efficient achievement of memory retrieval capabilities.It is also shown that some spots of the network cannot be reached by retrieval activity spread if they are not directly cued by the stimulus. The locations of these activity isles depend on the pattern to retrieve, while their extension only depends (in large networks) on statistics of inter-modular connections and stored patterns. The existence of activity isles determines an upper-bound to retrieval quality that does not depend on the specific retrieval dynamics adopted, nor on whether feature-sharing is permitted. The oscillatory retrieval process nearly saturates this bound.     

Cluster associative memory formation in a three-layer network of phase oscillators

A three-layer network model of oscillatory associative memory is proposed. The network is capable of storing binary images, which can be retrieved upon presenting an appropriate stimulus. Binary images are encoded in the form of the spatial distribution of oscillatory phase clusters in-phase and anti-phase relative to a reference periodic signal. The information is loaded into the network using a set of interlayer connection weights. A condition for error-free pattern retrieval is formulated, delimiting the maximal number of patterns to be stored in the memory (storage capacity). It is shown that the capacity can be significantly increased by generating an optimal alphabet (basis pattern set). The number of stored patterns can reach values of the network size (the number of oscillators in each layer), which is significantly higher than the capacity of conventional oscillatory memory models. The dynamical and information characteristics of the retrieval process based on the optimal alphabet, including the size of “attraction basins“ and the input pattern distortion admissible for error-free retrieval, are investigated.     

Chaotic synchronization using a network of neural oscillators

Synchronization of chaotic low-dimensional systems has been a topic of much recent research. Such systems have found applications for secure communications. In this work we show how synchronization can be achieved in a high-dimensional chaotic neural network. The network used in our studies is an extension of the Hopfield Network, known as the Complex Hopfield Network (CHN). The CHN, also an associative memory, has both fixed point and limit cycle or oscillatory behavior. In the oscillatory mode, the network wanders chaotically from one stored pattern to another. We show how a pair of identical high-dimensional CHNs can be synchronized by communicating only a subset of state vector components. The synchronizability of such a system is characterized through simulations.     

An associative memory that can form hypotheses: a phase-coded neural network

 Nonlinear associative memories as realized, e.g., by Hopfield nets are characterized by attractor-type dynamics. When fed with a starting pattern, they converge to exactly one of the stored patterns which is supposed to be most similar. These systems cannot render hypotheses of classification, i.e., render several possible answers to a given classification problem. Inspired by von der Malsburg’s correlation theory of brain function, we extend conventional neural network architectures by introducing additional dynamical variables. Assuming an oscillatory time structure of neural firing, i.e., the existence of neural clocks, we assign a so-called phase to each formal neuron. The phases explicitly describe detailed correlations of neural activities neglected in conventional neural network architectures. Implementing this extension into a simple self-organizing network based on a feature map, we present an associative memory that actually is capable of forming hypotheses of classification. Received: 6 December 1993/Accepted in revised form: 14 July 1994     

Izhikevich神经元网络的同步与联想记忆

联想记忆是人脑的一项重要功能。以Izhikevich神经元模型为节点,构建神经网络,神经元之间采用全连结的方式;以神经元群体的时空编码（spatio-temporal coding）理论研究所构建神经网络的联想记忆功能。在加入高斯白噪声的情况下,调节网络中神经元之间的连接强度的大小,当连接强度和噪声强度达到一个阈值时网络中部分神经元同步放电,实现了存储模式的联想记忆与恢复。仿真结果表明,神经元之间的连接强度在联想记忆的过程中发挥了重要的作用,噪声可以促使神经元间的同步放电,有助于神经网络实现存储模式的联想记忆与恢复。     

Synchronization of neural activity is a promising mechanism of memory information processing in networks of columns

Synchronization of the oscillatory discharge of cortical neurons could be a part of the mechanism that is involved in cortical information processing. On the assumption that the basic functional unit is the column composed of local excitatory and inhibitory cells and generating oscillatory neural activity, a network model that attains associative memory function is proposed. The synchronization of oscillation in the model is studied analytically using a sublattice analysis. In particular, the retrieval of a single memory pattern can be studied in the system, which can be derived from the original network model of interacting columns and is formally equivalent to a system of an isolated column. The network model simulated numerically shows a remarkable performance in which retrieval is achieved simultaneously for more than one memory pattern. The manifestations of this simultaneous retrieval in the network dynamics are successive transitions of the network state from a synchronized oscillation for a memory pattern to that for another memory pattern.     

An artificial hysteresis binary neuron: a model suppressing the oscillatory behaviors of neural dynamics

A hysteresis binary McCulloch-Pitts neuron model is proposed in order to suppress the complicated oscillatory behaviors of neural dynamics. The artificial hysteresis binary neural network is used for scheduling time-multiplex crossbar switches in order to demonstrate the effects of hysteresis. Time-multiplex crossbar switching systems must control traffic on demand such that packet blocking probability and packet waiting time are minimized. The system using n×n processing elements solves an n×n crossbar-control problem with O(1) time, while the best existing parallel algorithm requires O(n) time. The hysteresis binary neural network maximizes the throughput of packets through a crossbar switch. The solution quality of our system does not degrade with the problem size.     

Library-based numerical reduction of the Hodgkin–Huxley neuron for network simulation

We present an efficient library-based numerical method for simulating the Hodgkin–Huxley (HH) neuronal networks. The key components in our numerical method involve (i) a pre-computed high resolution data library which contains typical neuronal trajectories (i.e., the time-courses of membrane potential and gating variables) during the interval of an action potential (spike), thus allowing us to avoid resolving the spikes in detail and to use large numerical time steps for evolving the HH neuron equations; (ii) an algorithm of spike-spike corrections within the groups of strongly coupled neurons to account for spike-spike interactions in a single large time step. By using the library method, we can evolve the HH networks using time steps one order of magnitude larger than the typical time steps used for resolving the trajectories without the library, while achieving comparable resolution in statistical quantifications of the network activity, such as average firing rate, interspike interval distribution, power spectra of voltage traces. Moreover, our large time steps using the library method can break the stability requirement of standard methods (such as Runge–Kutta (RK) methods) for the original dynamics. We compare our library-based method with RK methods, and find that our method can capture very well phase-locked, synchronous, and chaotic dynamics of HH neuronal networks. It is important to point out that, in essence, our library-based HH neuron solver can be viewed as a numerical reduction of the HH neuron to an integrate-and-fire (I&F) neuronal representation that does not sacrifice the gating dynamics (as normally done in the analytical reduction to an I&F neuron).     

A model cortical circuit for the storage of temporal sequences

Despite the fact that temporal information processing is of particular significance in biological memory systems, not much has yet been explored about how these systems manage to store temporal information involved in sequences of stimuli. A neural network model capable of learning and recalling temporal sequences is proposed, based on a neural mechanism in which the sequences are expanded into a series of periodic rectangular oscillations. Thus, the mathematical framework underlying the model, to some extent, is concerned with the Walsh function series. The oscillatory activities generated by the interplay between excitatory and inhibitory neuron pools are transmitted to another neuron pool whose role in learning and retrieval is to modify the rhythms and phases of the rectangular oscillations. Thus, a basic functional neural circuit involves three different neuron pools. The modifiability of rhythms and phases is incorporated into the model with the aim of improving the quality of the retrieval. Numerical simulations were conducted to show the characteristic features of the learning as well as the performance of the model in memory recall.     

Generating oscillatory bursts from a network of regular spiking neurons without inhibition

Avian nucleus isthmi pars parvocellularis (Ipc) neurons are reciprocally connected with the layer 10 (L10) neurons in the optic tectum and respond with oscillatory bursts to visual stimulation. Our in vitro experiments show that both neuron types respond with regular spiking to somatic current injection and that the feedforward and feedback synaptic connections are excitatory, but of different strength and time course. To elucidate mechanisms of oscillatory bursting in this network of regularly spiking neurons, we investigated an experimentally constrained model of coupled leaky integrate-and-fire neurons with spike-rate adaptation. The model reproduces the observed Ipc oscillatory bursting in response to simulated visual stimulation. A scan through the model parameter volume reveals that Ipc oscillatory burst generation can be caused by strong and brief feedforward synaptic conductance changes. The mechanism is sensitive to the parameter values of spike-rate adaptation. In conclusion, we show that a network of regular-spiking neurons with feedforward excitation and spike-rate adaptation can generate oscillatory bursting in response to a constant input.     

Activation-Inhibition dynamics of the oscillatory bursts of the human EEG during resting state. The macroscopic temporal range of few seconds

The ubiquitous brain oscillations occur in bursts of oscillatory activity. The present report tries to define the statistical characteristics of electroencephalographical (EEG) bursts of oscillatory activity during resting state in humans to define (i) the statistical properties of amplitude and duration of oscillatory bursts, (ii) its possible correlation, (iii) its frequency content, and (iv) the presence or not of a fixed threshold to trigger an oscillatory burst. The open eyes EEG recordings of five subjects with no artifacts were selected from a sample of 40 subjects. The recordings were filtered in frequency ranges of 2 Hz wide from 1 to 99 Hz. The analytic Hilbert transform was computed to obtain the amplitude envelopes of oscillatory bursts. The criteria of thresholding and a minimum of three cycles to define an oscillatory burst were imposed. Amplitude and duration parameters were extracted and they showed durations between hundreds of milliseconds and a few seconds, and peak amplitudes showed a unimodal distribution. Both parameters were positively correlated and the oscillatory burst durations were explained by a linear model with the terms peak amplitude and peak amplitude of amplitude envelope time derivative. The frequency content of the amplitude envelope was contained in the 0–2 Hz range. The results suggest the presence of amplitude modulated continuous oscillations in the human EEG during the resting conditions in a broad frequency range, with durations in the range of few seconds and modulated positively by amplitude and negatively by the time derivative of the amplitude envelope suggesting activation-inhibition dynamics. This macroscopic oscillatory network behavior is less pronounced in the low-frequency range (1–3 Hz).     

Emergent mechanisms in multiple pattern generation of the lobster pyloric network

By using a hard-wired oscillator network, multiple pattern generation of the lobster pyloric network is simulated. The network model is constructed using a relaxation oscillator representing an oscillatory or quiescent (i.e. steady-state) neuron. Modulatory inputs to the network are hypothesized to cause changes in the dynamical properties of each pyloric neuron: the oscillatory frequency, the postinhibitory rebound property, and the resting membrane potential. Changes in each of these properties are induced by changing appropriate parameters of the oscillator. By changing seven parameters of the network as a whole, modulatory input-dependent patterns are successfully simulated. Received: 13 July 1999 / Accepted in revised form: 17 December 1999     

Mathematical aspects of outer-product asynchronous content-addressable memories

Conclusions We have given evidence by mathematical analysis and by example that the construction of CAM's as quasineural networks based on the twin principles of an outer-product weight matrix and a random asynchronous single-neuron dynamics encounters two obstacles to good performance which appear to be inherent. It is desirable to have the stored memory vectors of a CAM as mutually far apart as possible in order to have unambiguous retrieval with as large a fraction of initial errors (minrad) as possible. For a given number,m, of memory vectors to be stored this requires that their dimension,n, be larger thanm and that the memory vectors be nearly mutually orthogonal (except for complementary pairs). In HOCAM's, it does not seem possible to have both a large minrad and an efficient ratiom/n. Attempts to increasem/n are likely to introduce extraneous fixed-points which reduce minrad appreciably. We have demonstrated this phenomenon in several cases for a particular mode of constructing CAM's of arbitrary size which have a desirable spacing between memory vectors. We conjecture that it is present also in HOCAM's having a random selection of memory vectors. (A mathematical proof of this conjecture now seems possible.) This may account for the rather pessimistic results on capacity obtained by mathematical analysis here and in Cottrell (1988), by a probabilistic analysis in Posner (1987) and by simulation in Hopfield (1982). Further, in Cottrell (1988) there is evidence that outer-product weights are near optimal with respect to minrad, so that otherW may not improve matters.We have left to another paper a study of other approaches to content-addressable memories of which we are aware, but which are not focused on asynchronous dynamics; e.g. computer CAM's as in Kohonen (1977) and biological memory models as in Little (1974); Palm (1980) and Little and Shaw (1978). We have not considered the learning, or adaptive, aspects of CAM's. However, insofar as learning is Hebbian and leads to outer-product weights, our analysis has implications for the effectiveness of learned weights, as may be inferred from our results on ambiguous retrieval.This research was partially supported by NSF Grant CCR-87121192 and AFOSR Grant 88-0245     

From synapse to network: models of information storage and retrieval in neural circuits

The mechanisms of information storage and retrieval in brain circuits are still the subject of debate. It is widely believed that information is stored at least in part through changes in synaptic connectivity in networks that encode this information and that these changes lead in turn to modifications of network dynamics, such that the stored information can be retrieved at a later time. Here, we review recent progress in deriving synaptic plasticity rules from experimental data and in understanding how plasticity rules affect the dynamics of recurrent networks. We show that the dynamics generated by such networks exhibit a large degree of diversity, depending on parameters, similar to experimental observations in vivo during delayed response tasks.     

Spontaneous Local Gamma Oscillation Selectively Enhances Neural Network Responsiveness

Synchronized oscillation is very commonly observed in many neuronal systems and might play an important role in the response properties of the system. We have studied how the spontaneous oscillatory activity affects the responsiveness of a neuronal network, using a neural network model of the visual cortex built from Hodgkin-Huxley type excitatory (E-) and inhibitory (I-) neurons. When the isotropic local E-I and I-E synaptic connections were sufficiently strong, the network commonly generated gamma frequency oscillatory firing patterns in response to random feed-forward (FF) input spikes. This spontaneous oscillatory network activity injects a periodic local current that could amplify a weak synaptic input and enhance the network's responsiveness. When E-E connections were added, we found that the strength of oscillation can be modulated by varying the FF input strength without any changes in single neuron properties or interneuron connectivity. The response modulation is proportional to the oscillation strength, which leads to self-regulation such that the cortical network selectively amplifies various FF inputs according to its strength, without requiring any adaptation mechanism. We show that this selective cortical amplification is controlled by E-E cell interactions. We also found that this response amplification is spatially localized, which suggests that the responsiveness modulation may also be spatially selective. This suggests a generalized mechanism by which neural oscillatory activity can enhance the selectivity of a neural network to FF inputs.     

Network Events on Multiple Space and Time Scales in Cultured Neural Networks and in a Stochastic Rate Model

Cortical networks, in-vitro as well as in-vivo, can spontaneously generate a variety of collective dynamical events such as network spikes, UP and DOWN states, global oscillations, and avalanches. Though each of them has been variously recognized in previous works as expression of the excitability of the cortical tissue and the associated nonlinear dynamics, a unified picture of the determinant factors (dynamical and architectural) is desirable and not yet available. Progress has also been partially hindered by the use of a variety of statistical measures to define the network events of interest. We propose here a common probabilistic definition of network events that, applied to the firing activity of cultured neural networks, highlights the co-occurrence of network spikes, power-law distributed avalanches, and exponentially distributed ‘quasi-orbits’, which offer a third type of collective behavior. A rate model, including synaptic excitation and inhibition with no imposed topology, synaptic short-term depression, and finite-size noise, accounts for all these different, coexisting phenomena. We find that their emergence is largely regulated by the proximity to an oscillatory instability of the dynamics, where the non-linear excitable behavior leads to a self-amplification of activity fluctuations over a wide range of scales in space and time. In this sense, the cultured network dynamics is compatible with an excitation-inhibition balance corresponding to a slightly sub-critical regime. Finally, we propose and test a method to infer the characteristic time of the fatigue process, from the observed time course of the network’s firing rate. Unlike the model, possessing a single fatigue mechanism, the cultured network appears to show multiple time scales, signalling the possible coexistence of different fatigue mechanisms.     

A maximum neural network approach for N-queens problems

 A novel neural network approach using the maximum neuron model is presented for N-queens problems. The goal of the N-queens problem is to find a set of locations of N queens on an N×N chessboard such that no pair of queens commands each other. The maximum neuron model proposed by Takefuji et al. has been applied to two optimization problems where the optimization of objective functions is requested without constraints. This paper demonstrates the effectiveness of the maximum neuron model for constraint satisfaction problems through the N-queens problem. The performance is verified through simulations in up to 500-queens problems on the sequential mode, the N-parallel mode, and the N ²-parallel mode, where our maximum neural network shows the far better performance than the existing neural networks. Received: 4 June 1996/Accepted in revised form: 13 November 1996     

Stochastic resonance of localized activity driven by common noise

We study the influence of spatially correlated noise on the transient dynamics of a recurrent network with Mexican-Hat-type connectivity. We derive the closed form of the order parameter functional in the thermodynamical limit of neuron number N. Our analysis shows that network dynamics is qualitatively changed by the presence of common noise. Network dynamics driven by common noise obtains the global level of fluctuation, which is not observed in a network driven by independent noise only. We show that the optimal level of global fluctuation enhances the transition from non-localized firing states to spatially localized firing states, and also enhances the rotation speed of localized activity.     

Synchronous and asynchronous bursting states: role of intrinsic neural dynamics

Brain signals such as local field potentials often display gamma-band oscillations (30-70 Hz) in a variety of cognitive tasks. These oscillatory activities possibly reflect synchronization of cell assemblies that are engaged in a cognitive function. A type of pyramidal neurons, i.e., chattering neurons, show fast rhythmic bursting (FRB) in the gamma frequency range, and may play an active role in generating the gamma-band oscillations in the cerebral cortex. Our previous phase response analyses have revealed that the synchronization between the coupled bursting neurons significantly depends on the bursting mode that is defined as the number of spikes in each burst. Namely, a network of neurons bursting through a Ca(2+)-dependent mechanism exhibited sharp transitions between synchronous and asynchronous firing states when the neurons exchanged the bursting mode between singlet, doublet and so on. However, whether a broad class of bursting neuron models commonly show such a network behavior remains unclear. Here, we analyze the mechanism underlying this network behavior using a mathematically tractable neuron model. Then we extend our results to a multi-compartment version of the NaP current-based neuron model and prove a similar tight relationship between the bursting mode changes and the network state changes in this model. Thus, the synchronization behavior couples tightly to the bursting mode in a wide class of networks of bursting neurons.