Improving recall from an associative memory

The associative net as a model of biological associative memory is investigated. Calculating the output pattern retrieved from a partially connected associative net presented with noisy input cues involves several computations. This is complicated by variations in the dendritic sums of the output units due to errors in the cue and differences in input activity and unit usage. The possible implementation of these computations by biological neural machinery is unclear. We demonstrate that a relatively simple transformation can reduce variation in the dendritic sums. This leads to a winners-take-all type of strategy that produces increased recall performance which is equivalent to the more complicated optimal strategy proposed by others. We describe in detail the possible biological implications of our strategies, the novel feature of which ascribes a role to the NMDA and non-NMDA channels found in hippocampal pyramidal cells. Received: 13 April 1994 / Accepted: 25 October 1994     

A cognitive and associative memory

By introducing a physiological constraint in the auto-correlation matrix memory, the system is found to acquire an ability in cognition i.e. the ability to identify and input pattern by its proximity to any one of the stored memories. The physiological constraint here is that the attribute of a given synapse (i.e. excitatory or inhibitory) is uniquely determined by the neuron it belongs. Thus the synaptic coupling is generally not symmetric. Analytical and numerical analyses revealed that the present model retrieves a memory if an input pattern is close to the pattern of the stored memories; if not, it gives a clear response by going into a special mode where almost all neurons are in the same state in each time step. This uniform mode may be stationary or periodic, depending on whether or not the number of the excitatory neurons exceeds the number of inhibitory neurons.     

Efficient connection strategies in 1D and 2D associative memory models with and without displaced connectivity

This study examines the performance of sparsely connected associative memory models built using a number of different connection strategies, applied to one- and two-dimensional topologies. Efficient patterns of connectivity are identified which yield high performance at relatively low wiring costs in both topologies. Networks with displaced connectivity are seen to perform particularly well. It is found that two-dimensional models are more tolerant of variations in connection strategy than their one-dimensional counterparts; though networks built with both topologies become less so as their connection density is decreased.     

Stability and attractivity in associative memory networks

We focus on stable and attractive states in a network having two-state neuron-like elements. We calculate the connection matrix which guarantees the stability and the strongest attractivity of p memorized patterns. We present an analytical evaluation of the patterns' attractivity. These results are illustrated by some computer simulations.     

Mixed state on a sparsely encoded associative memory model

 We analyzed symmetric mixed states corresponding to the so-called concept formation on a sparsely encoded associative memory model with 0–1 neurons. Three types of mixed states – OR, AND, and a majority decision – are described as typical examples. Each element of the OR mixed state is composed of corresponding memory pattern elements by means of the OR operation. The other two types are similarly defined. By analyzing their equilibrium properties through self-consistent signal-to-noise analysis and computer simulation, we found that the storage capacity of the OR mixed state diverges in a sparse limit, but that the other states do not diverge. In addition, we found that the optimal threshold values, which maximize the storage capacity for the memory pattern and the OR mixed state, coincide with each other in the spare limit. We conclude that the OR mixed state is a reasonable representative of mixed state in the sparse limit. Received: 10 November 1999 / Accepted in revised form: 5 April 2001     

Dynamics control of semantic processes in a hierarchical associative memory

 A neural mechanism for control of dynamics and function of associative processes in a hierarchical memory system is demonstrated. For the representation and processing of abstract knowledge, the semantic declarative memory system of the human brain is considered. The dynamics control mechanism is based on the influence of neuronal adaptation on the complexity of neural network dynamics. Different dynamical modes correspond to different levels of the ultrametric structure of the hierarchical memory being invoked during an associative process. The mechanism is deterministic but may also underlie free associative thought processes. The formulation of an abstract neural network model of hierarchical associative memory utilizes a recent approach to incorporate neuronal adaptation. It includes a generalized neuronal activation function recently derived by a Hodgkin-Huxley-type model. It is shown that the extent to which a hierarchically organized memory structure is searched is controlled by the neuronal adaptability, i.e. the strength of coupling between neuronal activity and excitability. In the brain, the concentration of various neuromodulators in turn can regulate the adaptability. An autonomously controlled sequence of bifurcations, from an initial exploratory to a final retrieval phase, of an associative process is shown to result from an activity-dependent release of neuromodulators. The dynamics control mechanism may be important in the context of various disorders of the brain and may also extend the range of applications of artificial neural networks. Received: 19 April 1995/Accepted in revised form: 8 August 1995     

Dynamics control of semantic processes in a hierarchical associative memory

A neural mechanism for control of dynamics and function of associative processes in a hierarchical memory system is demonstrated. For the representation and processing of abstract knowledge, the semantic declarative memory system of the human brain is considered. The dynamics control mechanism is based on the influence of neuronal adaptation on the complexity of neural network dynamics. Different dynamical modes correspond to different levels of the ultrametric structure of the hierarchical memory being invoked during an associative process. The mechanism is deterministic but may also underlie free associative thought processes. The formulation of an abstract neural network model of hierarchical associative memory utilizes a recent approach to incorporate neuronal adaptation. It includes a generalized neuronal activation function recently derived by a Hodgkin-Huxley-type model. It is shown that the extent to which a hierarchically organized memory structure is searched is controlled by the neuronal adaptability, i.e. the strength of coupling between neuronal activity and excitability. In the brain, the concentration of various neuromodulators in turn can regulate the adaptability. An autonomously controlled sequence of bifurcations, from an initial exploratory to a final retrieval phase, of an associative process is shown to result from an activity-dependent release of neuromodulators. The dynamics control mechanism may be important in the context of various disorders of the brain and may also extend the range of applications of artificial neural networks.     

Acetylcholine and associative memory in the piriform cortex

The significance of cholinergic modulation for associative memory performance in the piriform cortex was examined in a study combining cellular neurophysiology in brain slices with realistic biophysical network simulations. Three different physiological effects of acetylcholine were identified at the single-cell level: suppression of neuronal adaptation, suppression of synaptic transmission in the intrinsic fibers layer, and activity-dependent increase in synaptic strength. Biophysical simulations show how these three effects are joined together to enhance learning and recall performance of the cortical network. Furthermore, our data suggest that activity-dependent synaptic decay during learning is a crucial factor in determining learning capability of the cortical network. Accordingly, it is predicted that acetylcholine should also enhance long-term depression in the piriform cortex.     

Trace feature map: a model of episodic associative memory

An approach to episodic associative memory is presented, which has several desirable properties as a human memory model. The design is based on topological feature map representation of data. An ordinary feature map is a classifier, mapping an input vector onto a topologically meaningful location on the map. A trace feature map, in addition, creates a memory trace on that location. The traces can be stored episodically in a single presentation, and retrieved with a partial cue. Nearby traces overlap, which results in plausible memory interference behavior. Performance degrades gracefully as the memory is overloaded. More recent traces are easier to recall as are traces that are unique in the memory.This research was supported in part by an ITA Foundation grant and by fellowships from the Academy of Finland, the Emil Aaltonen Foundation and the Foundation for the Advancement of Technology (Finland) when the author was at UCLA     

Visual associative memory and the orientation-contingent color after-effect

The traditional explanation of the McCollough effect (ME) by selective adaptation of single detectors selective to color and orientation suffers from a number of inconsistencies: 1) the ME lasts much longer (from several days up to 3 months) than the ordinary adaptation, the decay of the effect being completely arrested by night sleep or occluding the eye for a long time; 2) the strength of the ME practically does not depend on the intensity of adapting light; and 3) a set of related pattern-contingent after-effects discovered later required for such an explanation new detectors, specific for other patterns. These properties can be explained, however, in the framework of associative memory and novelty filters. A computational model has been developed, which consists of 1) an input layer of two (left and right eyes) square matrices with two analog receptors (red and green) in each pixel, 2) an isomorphic associative neural layer, each analog neuron being synaptically connected with all receptors of both eyes, and 3) an output layer (novelty filter). The modification of synaptic efficacies conforms to the Hebb learning rule. The function of the model was examined by simulation. After a few presentations of colored gratings, the model displays the ME that is slowly destroyed by subsequent presentations of random pictures. With a sufficiently large receptor matrix, the effect lasts a thousand times longer than the period of adaptation. Continuous darkness does not change the strength of the effect. Like in real ME, the model does not display interocular transfer. The model can account for different pattern-contingent color after-effects without assuming any predetermined specific detectors. Such detectors are constructed in the course of adaptation to specific stimuli (gratings).     

Pattern recognition and associative memory as dynamical processes in a synergetic system

We consider a model for associative memory and pattern recognition which was devised by Haken (1987b). This model treats the activity of the neurons as continuous variables and exploits an analogy with pattern formation in synergetic systems. The capability of such a system to act as associative memory is demonstrated by the reconstruction of faces which are partially offered to the system, and which are restored by the corresponding dynamical process. We demonstrate how this model can be cast into a form which is translation invariant and how partially hidden faces in scenes can be recognized by means of the control of attention parameters of specific patterns.     

Pattern recognition and associative memory as dynamical processes in a synergetic system

We first present numerical results for the decomposition procedure for complex scenes described in Part I of these papers. Part II then mainly deals with a formalism that allows a formulation of our approach to pattern recognition and associative memory that is simultaneously invariant against translation, rotation, and scaling, Part II thus contains an explicit elaboration of ideas of Part I.     

Persistent sodium current is a nonsynaptic substrate for long-term associative memory

Although synaptic plasticity is widely regarded as the primary mechanism of memory [1], forms of nonsynaptic plasticity, such as increased somal or dendritic excitability or membrane potential depolarization, also have been implicated in learning in both vertebrate and invertebrate experimental systems [2], [3], [4], [5], [6] and [7]. Compared to synaptic plasticity, however, there is much less information available on the mechanisms of specific types of nonsynaptic plasticity involved in well-defined examples of behavioral memory. Recently, we have shown that learning-induced somal depolarization of an identified modulatory cell type (the cerebral giant cells, CGCs) of the snail Lymnaea stagnalis encodes information that enables the expression of long-term associative memory [8]. The Lymnaea CGCs therefore provide a highly suitable experimental system for investigating the ionic mechanisms of nonsynaptic plasticity that can be linked to behavioral learning. Based on a combined behavioral, electrophysiological, immunohistochemical, and computer simulation approach, here we show that an increase of a persistent sodium current of this neuron underlies its delayed and persistent depolarization after behavioral single-trial classical conditioning. Our findings provide new insights into how learning-induced membrane level changes are translated into a form of long-lasting neuronal plasticity already known to contribute to maintained adaptive modifications at the network and behavioral level [8].     

Coexistence of memory patterns and mixed states in a sparsely encoded associative memory model storing ultrametric patterns

Analyzing the coexistence of memory patterns and mixed states gives important information for constructing a model for the face responsive neurons of the monkey inferior-temporal cortex. We analyzed whether the memory patterns coexist with mixed states when the sparse coding scheme is used for the associative memory model storing ultrametric patterns. For memory patterns and mixed states to coexist, there must be sufficient capacity for storing them and their threshold values must be the same. We determined that the storage capacities for all mixed states composed of correlated memory patterns diverge as 1/|flogf| (where f is the firing rate) even when the correlation of the memory patterns is infinitesimally small. We also determined that the memory patterns and the mixed states can become the equilibrium state of the model in the same threshold value. These results mean that they can coexist in this model. These findings should contribute to research on face responsive neurons in the monkey inferior-temporal cortex.     

Cluster associative memory formation in a three-layer network of phase oscillators

A three-layer network model of oscillatory associative memory is proposed. The network is capable of storing binary images, which can be retrieved upon presenting an appropriate stimulus. Binary images are encoded in the form of the spatial distribution of oscillatory phase clusters in-phase and anti-phase relative to a reference periodic signal. The information is loaded into the network using a set of interlayer connection weights. A condition for error-free pattern retrieval is formulated, delimiting the maximal number of patterns to be stored in the memory (storage capacity). It is shown that the capacity can be significantly increased by generating an optimal alphabet (basis pattern set). The number of stored patterns can reach values of the network size (the number of oscillators in each layer), which is significantly higher than the capacity of conventional oscillatory memory models. The dynamical and information characteristics of the retrieval process based on the optimal alphabet, including the size of “attraction basins“ and the input pattern distortion admissible for error-free retrieval, are investigated.     

An associative memory that can form hypotheses: a phase-coded neural network

 Nonlinear associative memories as realized, e.g., by Hopfield nets are characterized by attractor-type dynamics. When fed with a starting pattern, they converge to exactly one of the stored patterns which is supposed to be most similar. These systems cannot render hypotheses of classification, i.e., render several possible answers to a given classification problem. Inspired by von der Malsburg’s correlation theory of brain function, we extend conventional neural network architectures by introducing additional dynamical variables. Assuming an oscillatory time structure of neural firing, i.e., the existence of neural clocks, we assign a so-called phase to each formal neuron. The phases explicitly describe detailed correlations of neural activities neglected in conventional neural network architectures. Implementing this extension into a simple self-organizing network based on a feature map, we present an associative memory that actually is capable of forming hypotheses of classification. Received: 6 December 1993/Accepted in revised form: 14 July 1994     

Interfering with theories of sleep and memory: sleep, declarative memory, and associative interference

Mounting behavioral evidence in humans supports the claim that sleep leads to improvements in recently acquired, nondeclarative memories. Examples include motor-sequence learning; visual-discrimination learning; and perceptual learning of a synthetic language. In contrast, there are limited human data supporting a benefit of sleep for declarative (hippocampus-mediated) memory in humans (for review, see). This is particularly surprising given that animal models (e.g.,) and neuroimaging studies (e.g.,) predict that sleep facilitates hippocampus-based memory consolidation. We hypothesized that we could unmask the benefits of sleep by challenging the declarative memory system with competing information (interference). This is the first study to demonstrate that sleep protects declarative memories from subsequent associative interference, and it has important implications for understanding the neurobiology of memory consolidation.     

Construction of an associative memory using unstable periodic orbits of a chaotic attractor

Unstable periodic orbits are the skeleton of a chaotic attractor. We constructed an associative memory based on the chaotic attractor of an artificial neural network, which associates input patterns to unstable periodic orbits. By processing an input, the system is driven out of the ground state to one of the pre-defined disjunctive areas of the attractor. Each of these areas is associated with a different unstable periodic orbit. We call an input pattern learned if the control mechanism keeps the system on the unstable periodic orbit during the response. Otherwise, the system relaxes back to the ground state on a chaotic trajectory. The major benefits of this memory device are its high capacity and low-energy consumption. In addition, new information can be simply added by linking a new input to a new unstable periodic orbit.