Cross-talk theory of memory capacity in neural networks

The present paper presents a theory for the mechanics of cross-talk among constituent neurons in networks in which multiple memory traces have been embedded, and develops criteria for memory capacity based on the disruptive influences of this cross-talk. The theory is based on interconnection patterns defined by the sequential configuration model of dynamic firing patterns. The theory accurately predicts the memory capacities observed in computer simulated nets, and predicts that cortical-like modules should be able to store up to about 300–900 selectively retrievable memory traces before disruption by cross-talk is likely. It also predicts that the cortex may has designed itself for modules of 30,000 neurons to at least in part to optimize memory capacity.     

Working Memory Cells' Behavior May Be Explained by Cross-Regional Networks with Synaptic Facilitation

Neurons in the cortex exhibit a number of patterns that correlate with working memory. Specifically, averaged across trials of working memory tasks, neurons exhibit different firing rate patterns during the delay of those tasks. These patterns include: 1) persistent fixed-frequency elevated rates above baseline, 2) elevated rates that decay throughout the tasks memory period, 3) rates that accelerate throughout the delay, and 4) patterns of inhibited firing (below baseline) analogous to each of the preceding excitatory patterns. Persistent elevated rate patterns are believed to be the neural correlate of working memory retention and preparation for execution of behavioral/motor responses as required in working memory tasks. Models have proposed that such activity corresponds to stable attractors in cortical neural networks with fixed synaptic weights. However, the variability in patterned behavior and the firing statistics of real neurons across the entire range of those behaviors across and within trials of working memory tasks are typical not reproduced. Here we examine the effect of dynamic synapses and network architectures with multiple cortical areas on the states and dynamics of working memory networks. The analysis indicates that the multiple pattern types exhibited by cells in working memory networks are inherent in networks with dynamic synapses, and that the variability and firing statistics in such networks with distributed architectures agree with that observed in the cortex.     

Distributed neural networks of tactile working memory

Microelectrode recordings of cortical activity in primates performing working memory tasks reveal some cortical neurons exhibiting sustained or graded persistent elevations in firing rate during the period in which sensory information is actively maintained in short-term memory. These neurons are called “memory cells”. Imaging and transcranial magnetic stimulation studies indicate that memory cells may arise from distributed cortical networks. Depending on the sensory modality of the memorandum in working memory tasks, neurons exhibiting memory-correlated patterns of firing have been detected in different association cortices including prefrontal cortex, and primary sensory cortices as well.Here we elaborate on neurophysiological experiments that lead to our understanding of the neuromechanisms of working memory, and mainly discuss findings on widely distributed cortical networks involved in tactile working memory.     

Physical models of neural networks

The state of art in computer modelling of neural networks with associative memory is reviewed. The available experimental data are considered on learning and memory of small neural systems, on isolated synapses and on molecular level. Computer simulations demonstrate that realistic models of neural ensembles exhibit properties which can be interpreted as image recognition, categorization, learning, prototype forming, etc. A bilayer model of associative neural network is proposed. One layer corresponds to the short-term memory, the other one to the long-term memory. Patterns are stored in terms of the synaptic strength matrix. We have studied the relaxational dynamics of neurons firing and suppression within the short-term memory layer under the influence of the long-term memory layer. The interaction among the layers has found to create a number of novel stable states which are not the learning patterns. These synthetic patterns may consist of elements belonging to different non-intersecting learning patterns. Within the framework of a hypothesis of selective and definite coding of images in brain one can interpret the observed effect as the "idea? generating" process.     

Acetylcholine-gated current translates wake neuronal firing rate information into a spike timing-based code in Non-REM sleep,stabilizing neural network dynamics during memory consolidation

Sleep is critical for memory consolidation, although the exact mechanisms mediating this process are unknown. Combining reduced network models and analysis of in vivo recordings, we tested the hypothesis that neuromodulatory changes in acetylcholine (ACh) levels during non-rapid eye movement (NREM) sleep mediate stabilization of network-wide firing patterns, with temporal order of neurons’ firing dependent on their mean firing rate during wake. In both reduced models and in vivo recordings from mouse hippocampus, we find that the relative order of firing among neurons during NREM sleep reflects their relative firing rates during prior wake. Our modeling results show that this remapping of wake-associated, firing frequency-based representations is based on NREM-associated changes in neuronal excitability mediated by ACh-gated potassium current. We also show that learning-dependent reordering of sequential firing during NREM sleep, together with spike timing-dependent plasticity (STDP), reconfigures neuronal firing rates across the network. This rescaling of firing rates has been reported in multiple brain circuits across periods of sleep. Our model and experimental data both suggest that this effect is amplified in neural circuits following learning. Together our data suggest that sleep may bias neural networks from firing rate-based towards phase-based information encoding to consolidate memories.     

Characterization,scaling, and partial representation of diffuse and discrete input junctions to CA3 hippocampus

This paper applies a general mathematical system for characterizing and scaling functional connectivity and information flow across the diffuse (EC) and discrete (DG) input junctions to the CA3 hippocampus. Both gross connectivity and coordinated multiunit informational firing patterns are quantitatively characterized in terms of 32 defining parameters interrelated by 17 equations, and then scaled down according to rules for uniformly proportional scaling and for partial representation. The diffuse EC-CA3 junction is shown to be uniformly scalable with realistic representation of both essential spatiotemporal cooperativity and coordinated firing patterns down to populations of a few hundred neurons. Scaling of the discrete DG-CA3 junction can be effected with a two-step process, which necessarily deviates from uniform proportionality but nonetheless produces a valuable and readily interpretable reduced model, also utilizing a few hundred neurons in the receiving population. Partial representation produces a reduced model of only a portion of the full network where each model neuron corresponds directly to a biological neuron. The mathematical analysis illustrated here shows that although omissions and distortions are inescapable in such an application, satisfactorily complete and accurate models the size of pattern modules are possible. Finally, the mathematical characterization of these junctions generates a theory which sees the DG as a definer of the fine structure of embedded traces in the hippocampus and entire coordinated patterns of sequences of 14-cell links in CA3 as triggered by the firing of sequences of individual neurons in DG.     

Trace feature map: a model of episodic associative memory

An approach to episodic associative memory is presented, which has several desirable properties as a human memory model. The design is based on topological feature map representation of data. An ordinary feature map is a classifier, mapping an input vector onto a topologically meaningful location on the map. A trace feature map, in addition, creates a memory trace on that location. The traces can be stored episodically in a single presentation, and retrieved with a partial cue. Nearby traces overlap, which results in plausible memory interference behavior. Performance degrades gracefully as the memory is overloaded. More recent traces are easier to recall as are traces that are unique in the memory.This research was supported in part by an ITA Foundation grant and by fellowships from the Academy of Finland, the Emil Aaltonen Foundation and the Foundation for the Advancement of Technology (Finland) when the author was at UCLA     

The functional role of different neural activation profiles during precision grip: An artificial neural network approach

A dynamic and recurrent artificial neural network was used to investigate the functional properties of firing patterns observed in the primary motor (M1) and the primary somatosensory (S1) cortex of the behaving monkey during control of precision grip force. In the behaving monkey it was found that neurons in M1 and in S1 increase their firing activity with increasing grip force, as do the intrinsic and extrinsic hand muscles implicated in the task. However, some neurons also decreased their activity as a function of increasing force. The functional implication of these latter neurons is not clear and has not been elucidated so far. In order to explore their functional implication, we therefore simulated patterns of neural activity in artificial neural networks that represent cortical, spinal and afferent neural populations and tested whether particular activity profiles would emerge as a function of the input and of the connectivity of these networks. The functional implication of units with emergent or imposed decreasing activity was then explored.Decreasing patterns of activity in M1 units did not emerge from the networks. However, the same networks generated decreasing activity if imposed as target patterns. As indicated by the emerging weight space, M1 projection units with decreasing patterns are functionally less involved in driving alpha motoneurons than units with increasing profiles. Furthermore, these units did not provide significant fusimotor drive, whereas those with increasing profiles did. Fusimotor drive was a function of the (imposed) form of muscle spindle afferent activity: with gamma (fusimotor) drive, muscle spindle afferents provided signals other than muscle length (as observed experimentally). The network solutions thus predict a functional dichotomy between increasing and decreasing M1 neurons: the former primarily drive alpha and gamma motoneurons, the latter only weakly alpha motoneurons.     

Modelling relative recency discrimination tasks using a stochastic working memory model

Relative recency discrimination (RD) task is typically used to assess the temporal organization function of the prefrontal cortex (PFC). Subjects look at a series of cards (with words or drawings on them) and on seeing a test card determine which of the two items was seen more recently. Results show that patients with damage to the prefrontal cortex are severely impaired on this task. We propose a memory trace-priming mechanism, based on automatic time-marking process hypothesis (Schacter, D.L., 1987. Memory, amnesia, and frontal lobe dysfunction: a critique and interpretation. Psychobiology 15, 21-36), to offer a computational account of the results. In this model, successive words seen by subjects leave decaying memory traces in PFC, which subsequently prime the representations in higher sensory areas such as inferior temporal Cortex (IT) during discrimination judgements. The paper focuses on the evaluation of a probabilistic pre-frontal trace mechanism using a pool of clusters of neurons with self-sustained firing that ends at a random time. The results show that the probabilistic behavior of subjects can be accounted for by the stochasticity of the trace model. A good fit to experimental data is obtained with a PFC memory persistence probability with a decay time constant of tau = approximately 30 s. The model allows for a distributed representation in IT and PFC, but the best fit suggests a sparse representation. It is concluded that further data are needed on representations in IT and PFC, on the connectivity between these two areas, and on the statistical and dynamic properties of memory neurons in PFC.     

Cortical network modeling: analytical methods for firing rates and some properties of networks of LIF neurons.

The circuitry of cortical networks involves interacting populations of excitatory (E) and inhibitory (I) neurons whose relationships are now known to a large extent. Inputs to E- and I-cells may have their origins in remote or local cortical areas. We consider a rudimentary model involving E- and I-cells. One of our goals is to test an analytic approach to finding firing rates in neural networks without using a diffusion approximation and to this end we consider in detail networks of excitatory neurons with leaky integrate-and-fire (LIF) dynamics. A simple measure of synchronization, denoted by S(q), where q is between 0 and 100 is introduced. Fully connected E-networks have a large tendency to become dominated by synchronously firing groups of cells, except when inputs are relatively weak. We observed random or asynchronous firing in such networks with diverse sets of parameter values. When such firing patterns were found, the analytical approach was often able to accurately predict average neuronal firing rates. We also considered several properties of E-E networks, distinguishing several kinds of firing pattern. Included were those with silences before or after periods of intense activity or with periodic synchronization. We investigated the occurrence of synchronized firing with respect to changes in the internal excitatory postsynaptic potential (EPSP) magnitude in a network of 100 neurons with fixed values of the remaining parameters. When the internal EPSP size was less than a certain value, synchronization was absent. The amount of synchronization then increased slowly as the EPSP amplitude increased until at a particular EPSP size the amount of synchronization abruptly increased, with S(5) attaining the maximum value of 100%. We also found network frequency transfer characteristics for various network sizes and found a linear dependence of firing frequency over wide ranges of the external afferent frequency, with non-linear effects at lower input frequencies. The theory may also be applied to sparsely connected networks, whose firing behaviour was found to change abruptly as the probability of a connection passed through a critical value. The analytical method was also found to be useful for a feed-forward excitatory network and a network of excitatory and inhibitory neurons.     

Mechanism of graded persistent cellular activity of entorhinal cortex layer v neurons

Working memory is an emergent property of neuronal networks, but its cellular basis remains elusive. Recent data show that principal neurons of the entorhinal cortex display persistent firing at graded firing rates that can be shifted up or down in response to brief excitatory or inhibitory stimuli. Here, we present a model of a potential mechanism for graded firing. Our multicompartmental model provides stable plateau firing generated by a nonspecific calcium-sensitive cationic (CAN) current. Sustained firing is insensitive to small variations in Ca2+ concentration in a neutral zone. However, both high and low Ca2+ levels alter firing rates. Specifically, increases in persistent firing rate are triggered only during high levels of calcium, while decreases in rate occur in the presence of low levels of calcium. The model is consistent with detailed experimental observations and provides a mechanism for maintenance of memory-related activity in individual neurons.     

The chronotron: a neuron that learns to fire temporally precise spike patterns

In many cases, neurons process information carried by the precise timings of spikes. Here we show how neurons can learn to generate specific temporally precise output spikes in response to input patterns of spikes having precise timings, thus processing and memorizing information that is entirely temporally coded, both as input and as output. We introduce two new supervised learning rules for spiking neurons with temporal coding of information (chronotrons), one that provides high memory capacity (E-learning), and one that has a higher biological plausibility (I-learning). With I-learning, the neuron learns to fire the target spike trains through synaptic changes that are proportional to the synaptic currents at the timings of real and target output spikes. We study these learning rules in computer simulations where we train integrate-and-fire neurons. Both learning rules allow neurons to fire at the desired timings, with sub-millisecond precision. We show how chronotrons can learn to classify their inputs, by firing identical, temporally precise spike trains for different inputs belonging to the same class. When the input is noisy, the classification also leads to noise reduction. We compute lower bounds for the memory capacity of chronotrons and explore the influence of various parameters on chronotrons' performance. The chronotrons can model neurons that encode information in the time of the first spike relative to the onset of salient stimuli or neurons in oscillatory networks that encode information in the phases of spikes relative to the background oscillation. Our results show that firing one spike per cycle optimizes memory capacity in neurons encoding information in the phase of firing relative to a background rhythm.     

Nonlinear sequence-dependent structure of nigral dopamine neuron interspike interval firing patterns.

Firing patterns of 15 dopamine neurons in the rat substantia nigra were studied. These cells alternated between two firing modes, single-spike and bursting, which interwove to produce irregular, aperiodic interspike interval (ISI) patterns. When examined by linear autocorrelation analysis, these patterns appeared to reflect a primarily stochastic or random process. However, dynamical analysis revealed that the sequential behavior of a majority of these cells expressed "higher-dimensional" nonlinear deterministic structure. Dimensionality refers to the number of degrees of freedom or complexity of a time series. Bursting was statistically associated with some aspects of nonlinear ISI sequence dependence. Controlling for the effects of nonstationarity substantially increased overall predictability of ISI sequences. We hypothesize that the nonlinear deterministic structure of ISI firing patterns reflects the neuron's response to coordinated synaptic inputs emerging from neural circuit interactions.     

Memory Capacity of Networks with Stochastic Binary Synapses

In standard attractor neural network models, specific patterns of activity are stored in the synaptic matrix, so that they become fixed point attractors of the network dynamics. The storage capacity of such networks has been quantified in two ways: the maximal number of patterns that can be stored, and the stored information measured in bits per synapse. In this paper, we compute both quantities in fully connected networks of N binary neurons with binary synapses, storing patterns with coding level , in the large and sparse coding limits (). We also derive finite-size corrections that accurately reproduce the results of simulations in networks of tens of thousands of neurons. These methods are applied to three different scenarios: (1) the classic Willshaw model, (2) networks with stochastic learning in which patterns are shown only once (one shot learning), (3) networks with stochastic learning in which patterns are shown multiple times. The storage capacities are optimized over network parameters, which allows us to compare the performance of the different models. We show that finite-size effects strongly reduce the capacity, even for networks of realistic sizes. We discuss the implications of these results for memory storage in the hippocampus and cerebral cortex.     

Hippocampal ripples and memory consolidation

During slow wave sleep and quiet wakefulness, the hippocampus generates high frequency field oscillations (ripples) during which pyramidal neurons replay previous waking activity in a temporally compressed manner. As a result, reactivated firing patterns occur within shorter time windows propitious for synaptic plasticity within the hippocampal network and in downstream neocortical structures. This is consistent with the long-held view that ripples participate in strengthening and reorganizing memory traces, possibly by mediating information transfer to neocortical areas. Recent studies have confirmed that ripples and associated neuronal reactivations play a causal role in memory consolidation during sleep and rest. However, further research will be necessary to better understand the neurophysiological mechanisms of memory consolidation, in particular the selection of reactivated assemblies, and the functional specificity of awake ripples.     

Mechanisms Explaining Transitions between Tonic and Phasic Firing in Neuronal Populations as Predicted by a Low Dimensional Firing Rate Model

Several firing patterns experimentally observed in neural populations have been successfully correlated to animal behavior. Population bursting, hereby regarded as a period of high firing rate followed by a period of quiescence, is typically observed in groups of neurons during behavior. Biophysical membrane-potential models of single cell bursting involve at least three equations. Extending such models to study the collective behavior of neural populations involves thousands of equations and can be very expensive computationally. For this reason, low dimensional population models that capture biophysical aspects of networks are needed. The present paper uses a firing-rate model to study mechanisms that trigger and stop transitions between tonic and phasic population firing. These mechanisms are captured through a two-dimensional system, which can potentially be extended to include interactions between different areas of the nervous system with a small number of equations. The typical behavior of midbrain dopaminergic neurons in the rodent is used as an example to illustrate and interpret our results. The model presented here can be used as a building block to study interactions between networks of neurons. This theoretical approach may help contextualize and understand the factors involved in regulating burst firing in populations and how it may modulate distinct aspects of behavior.     

Bistable,Irregular Firing and Population Oscillations in a Modular Attractor Memory Network

Attractor neural networks are thought to underlie working memory functions in the cerebral cortex. Several such models have been proposed that successfully reproduce firing properties of neurons recorded from monkeys performing working memory tasks. However, the regular temporal structure of spike trains in these models is often incompatible with experimental data. Here, we show that the in vivo observations of bistable activity with irregular firing at the single cell level can be achieved in a large-scale network model with a modular structure in terms of several connected hypercolumns. Despite high irregularity of individual spike trains, the model shows population oscillations in the beta and gamma band in ground and active states, respectively. Irregular firing typically emerges in a high-conductance regime of balanced excitation and inhibition. Population oscillations can produce such a regime, but in previous models only a non-coding ground state was oscillatory. Due to the modular structure of our network, the oscillatory and irregular firing was maintained also in the active state without fine-tuning. Our model provides a novel mechanistic view of how irregular firing emerges in cortical populations as they go from beta to gamma oscillations during memory retrieval.     

Hodgkin–Huxley type modelling and parameter estimation of GnRH neurons

In this paper a simple one compartment Hodgkin–Huxley type electrophysiological model of GnRH neurons is presented, that is able to reasonably reproduce the most important qualitative features of the firing pattern, such as baseline potential, depolarization amplitudes, sub-baseline hyperpolarization phenomenon and average firing frequency in response to excitatory current. In addition, the same model provides an acceptable numerical fit of voltage clamp (VC) measurement results. The parameters of the model have been estimated using averaged VC traces, and characteristic values of measured current clamp traces originating from GnRH neurons in hypothalamic slices. The resulting parameter values show a good agreement with literature data in most of the cases. Applying parametric changes, which lead to the increase of baseline potential and enhance cell excitability, the model becomes capable of bursting. The effects of various parameters to burst length have been analyzed by simulation.     

Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep

Human dreaming occurs during rapid eye movement (REM) sleep. To investigate the structure of neural activity during REM sleep, we simultaneously recorded the activity of multiple neurons in the rat hippocampus during both sleep and awake behavior. We show that temporally sequenced ensemble firing rate patterns reflecting tens of seconds to minutes of behavioral experience are reproduced during REM episodes at an equivalent timescale. Furthermore, within such REM episodes behavior-dependent modulation of the subcortically driven theta rhythm is also reproduced. These results demonstrate that long temporal sequences of patterned multineuronal activity suggestive of episodic memory traces are reactivated during REM sleep. Such reactivation may be important for memory processing and provides a basis for the electrophysiological examination of the content of dream states.