A hierarchical neural network model for associative memory

A hierarchical neural network model with feedback interconnections, which has the function of associative memory and the ability to recognize patterns, is proposed. The model consists of a hierarchical multi-layered network to which efferent connections are added, so as to make positive feedback loops in pairs with afferent connections. The cell-layer at the initial stage of the network is the input layer which receives the stimulus input and at the same time works as an output layer for associative recall. The deepest layer is the output layer for pattern-recognition. Pattern-recognition is performed hierarchically by integrating information by converging afferent paths in the network. For the purpose of associative recall, the integrated information is again distributed to lower-order cells by diverging efferent paths. These two operations progress simultaneously in the network. If a fragment of a training pattern is presented to the network which has completed its self-organization, the entire pattern will gradually be recalled in the initial layer. If a stimulus consisting of a number of training patterns superposed is presented, one pattern gradually becomes predominant in the recalled output after competition between the patterns, and the others disappear. At about the same time when the recalled pattern reaches a steady state in he initial layer, in the deepest layer of the network, a response is elicited from the cell corresponding to the category of the finally-recalled pattern. Once a steady state has been reached, the response of the network is automatically extinguished by inhibitory signals from a steadiness-detecting cell. If the same stimulus is still presented after inhibition, a response for another pattern, formerly suppressed, will now appear, because the cells of the network have adaptation characteristics which makes the same response unlikely to recur. Since inhibition occurs repeatedly, the superposed input patterns are recalled one by one in turn.     

A self-organizing neural network with a function of associative memory: Feedback-type cognitron

Previously, one of the authors proposed a new hypothesis on the organization of synaptic connections, and constructed a model of self-organizing multi-layered neural network cognitron (Fukushima, 1975). the cognitron consists of a number of neural layers with similar structure connected in a cascade one after another. We have modified the structure of the cognitron, and have developed a new network having an ability of associative memory. The new network, named a feedback-type cognitron, has not only the feedforward connections as in the conventional cognitron, but also modifiable feedback connections from the last-layer cells to the front-layer ones. This network has been simulated on a digital computer. If several stimulus patterns are repeatedly presented to the network, the interconnections between the cells are gradually organized. The feedback connections, as well as the conventional feedforward ones, are self-organized depending on the characteristies of the externally presented stimulus patterns. After adequate number of stimulus presentations, each cell usually acquires the selective responsiveness to one of the stimulus patterns which have been frequently given. That is, every different stimulus pattern becomes to elicit an individual response to the network. After the completion of the self-organization, several stimulus patterns are presented to the network, and the responses are observed. Once a stimulus is given to the network, the signal keeps circulating in the network even after cutting off the stimulus, and the response gradually changes. Even though an imperfect or an ambiguous pattern is presented, the response usually converges to one of the patterns which have been frequently given during the process of self-organization. In some cases, however, a new pattern which has never been presented before, emerges. It is seen that this feedback-type cognitron has characteristics quite similar to some functions of the brain, such as the associative recall of memory, or the creation of a new idea by intuition.     

A new neural network architecture with associative memory, pruning and order-sensitive learning

A new paradigm of neural network architecture is proposed that works as associative memory along with capabilities of pruning and order-sensitive learning. The network has a composite structure wherein each node of the network is a Hopfield network by itself. The Hopfield network employs an order-sensitive learning technique and converges to user-specified stable states without having any spurious states. This is based on geometrical structure of the network and of the energy function. The network is so designed that it allows pruning in binary order as it progressively carries out associative memory retrieval. The capacity of the network is 2n, where n is the number of basic nodes in the network. The capabilities of the network are demonstrated by experimenting on three different application areas, namely a Library Database, a Protein Structure Database and Natural Language Understanding.     

Associative search network: A reinforcement learning associative memory

An associative memory system is presented which does not require a teacher to provide the desired associations. For each input key it conducts a search for the output pattern which optimizes an external payoff or reinforcement signal. The associative search network (ASN) combines pattern recognition and function optimization capabilities in a simple and effective way. We define the associative search problem, discuss conditions under which the associative search network is capable of solving it, and present results from computer simulations. The synthesis of sensory-motor control surfaces is discussed as an example of the associative search problem.     

Cluster associative memory formation in a three-layer network of phase oscillators

A three-layer network model of oscillatory associative memory is proposed. The network is capable of storing binary images, which can be retrieved upon presenting an appropriate stimulus. Binary images are encoded in the form of the spatial distribution of oscillatory phase clusters in-phase and anti-phase relative to a reference periodic signal. The information is loaded into the network using a set of interlayer connection weights. A condition for error-free pattern retrieval is formulated, delimiting the maximal number of patterns to be stored in the memory (storage capacity). It is shown that the capacity can be significantly increased by generating an optimal alphabet (basis pattern set). The number of stored patterns can reach values of the network size (the number of oscillators in each layer), which is significantly higher than the capacity of conventional oscillatory memory models. The dynamical and information characteristics of the retrieval process based on the optimal alphabet, including the size of “attraction basins“ and the input pattern distortion admissible for error-free retrieval, are investigated.     

Towards understanding of the cortical network underlying associative memory

Declarative knowledge and experiences are represented in the association cortex and are recalled by reactivation of the neural representation. Electrophysiological experiments have revealed that associations between semantically linked visual objects are formed in neural representations in the temporal and limbic cortices. Memory traces are created by the reorganization of neural circuits. These regions are reactivated during retrieval and contribute to the contents of a memory. Two different types of retrieval signals are suggested as follows: automatic and active. One flows backward from the medial temporal lobe during the automatic retrieval process, whereas the other is conveyed as a top-down signal from the prefrontal cortex to the temporal cortex during the active retrieval process. By sending the top-down signal, the prefrontal cortex manipulates and organizes to-be-remembered information, devises strategies for retrieval and monitors the outcome. To further understand the neural mechanism of memory, the following two complementary views are needed: how the multiple cortical areas in the brain-wide network interact to orchestrate cognitive functions and how the properties of single neurons and their synaptic connections with neighbouring neurons combine to form local circuits and to exhibit the function of each cortical area. We will discuss some new methodological innovations that tackle these challenges.     

Simulated model of auto-associative memory in the form of a neural network with a low level of activity

Information parameters of the neuron net performing the functions of autoassociative memory were investigated by imitation modelling method. The fully connected network with Hebb gradual synapses was studied. Its information capacity was shown to increase significantly with a decrease of the activity level of the stored patterns. This evidence agrees well with the analytical result obtained earlier by the replica method.     

An oscillatory neural network model of sparse distributed memory and novelty detection

A model of sparse distributed memory is developed that is based on phase relations between the incoming signals and an oscillatory mechanism for information processing. This includes phase-frequency encoding of input information, natural frequency adaptation among the network oscillators for storage of input signals, and a resonance amplification mechanism that responds to familiar stimuli. Simulations of this model show different types of dynamics in response to new and familiar stimuli. The application of the model to hippocampal working memory is discussed.     

Parameter estimation of a monod-type model. Part II: Use of a neural network with associative memories

Summary On the basis of a previous study related with parameter identifiability and sensitivity analysis of a Monod-type model, a parameter estimation method based on Artificial Neural Networks (ANN) with Associative Memories (AMs) is presented. A combination of an iterative procedure and a convergence index given by AMs allows to confirm the nature of relations existing between state variables and parameters which were found in the first part of the study. The convergence criterion is particularly well adapted to showing various influences of state variables on parameter estimation of such a model.     

Transitory memory retrieval in a biologically plausible neural network model

A number of memory models have been proposed. These all have the basic structure that excitatory neurons are reciprocally connected by recurrent connections together with the connections with inhibitory neurons, which yields associative memory (i.e., pattern completion) and successive retrieval of memory. In most of the models, a simple mathematical model for a neuron in the form of a discrete map is adopted. It has not, however, been clarified whether behaviors like associative memory and successive retrieval of memory appear when a biologically plausible neuron model is used. In this paper, we propose a network model for associative memory and successive retrieval of memory based on Pinsky-Rinzel neurons. The state of pattern completion in associative memory can be observed with an appropriate balance of excitatory and inhibitory connection strengths. Increasing of the connection strength of inhibitory interneurons changes the state of memory retrieval from associative memory to successive retrieval of memory. We investigate this transition.     

A neural network model for visual motion detection that can explain psychophysical and neurophysiological phenomena

This paper proposes a new neural network model for visual motion detection. The model can well explain both psychophysical findings (the changes of displacement thresholds with stimulus velocity and the perception of apparent motion) and neurophysiological findings (the selectivity for the direction and the velocity of a moving stimulus). To confirm the behavior of the model, numerical examinations were conducted. The results were consistent with both psychophysical and neurophysiological findings.     

An algebraic model of an associative noise-like coding memory

A mathematical model of an associative memory is presented, sharing with the optical holography memory systems the properties which establish an analogy with biological memory. This memory system-developed from Gabor's model of memoryis based on a noise-like coding of the information by which it realizes a distributed, damage-tolerant, equipotential storage through simultaneous state changes of discrete substratum elements. Each two associated items being stored are coded by each other by means of two noise-like patterns obtained from them through a randomizing preprocessing. The algebraic braic transformations operating the information storage and retrieval are matrix-vector products involving Toeplitz type matrices. Several noise-like coded memory traces are superimposed on a common substratum without crosstalk interference; moreover, extraneous noise added to these memory traces does not injure the stored information. The main performances shown by this memory model are: i) the selective, complete recovering of stored information from incomplete keys, both mixed with extraneous information and translated from the position learnt; ii) a dynamic recollection where the information just recovered acts as a new key for a sequential retrieval process; iii) context-dependent responses. The hypothesis that the information is stored in the nervous system through a noise-like coding is suggested. The model has been simulated on a digital computer using bidimensional images.     

Associative memory in quaternionic Hopfield neural network

Associative memory networks based on quaternionic Hopfield neural network are investigated in this paper. These networks are composed of quaternionic neurons, and input, output, threshold, and connection weights are represented in quaternions, which is a class of hypercomplex number systems. The energy function of the network and the Hebbian rule for embedding patterns are introduced. The stable states and their basins are explored for the networks with three neurons and four neurons. It is clarified that there exist at most 16 stable states, called multiplet components, as the degenerated stored patterns, and each of these states has its basin in the quaternionic networks.     

Stability of dead zone bidirectional associative memory neural networks involving time delays

A mathematical model describing the dynamical interactions of bidirectional associative memory networks involving transmission delays is considered. The influence of a dead zone or a zone of noactivation on the global stability is investigated and various easily verifiable sets of sufficient conditions are established. The asymptotic nature of solutions when the given system of equations does not possess an equilibrium pattern is discussed.     

An oscillatory neural network model that demonstrates the benefits of multisensory learning

Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.     

Trace feature map: a model of episodic associative memory

An approach to episodic associative memory is presented, which has several desirable properties as a human memory model. The design is based on topological feature map representation of data. An ordinary feature map is a classifier, mapping an input vector onto a topologically meaningful location on the map. A trace feature map, in addition, creates a memory trace on that location. The traces can be stored episodically in a single presentation, and retrieved with a partial cue. Nearby traces overlap, which results in plausible memory interference behavior. Performance degrades gracefully as the memory is overloaded. More recent traces are easier to recall as are traces that are unique in the memory.This research was supported in part by an ITA Foundation grant and by fellowships from the Academy of Finland, the Emil Aaltonen Foundation and the Foundation for the Advancement of Technology (Finland) when the author was at UCLA     

Synaptic plasticity, memory and the hippocampus: a neural network approach to causality

Two facts about the hippocampus have been common currency among neuroscientists for several decades. First, lesions of the hippocampus in humans prevent the acquisition of new episodic memories; second, activity-dependent synaptic plasticity is a prominent feature of hippocampal synapses. Given this background, the hypothesis that hippocampus-dependent memory is mediated, at least in part, by hippocampal synaptic plasticity has seemed as cogent in theory as it has been difficult to prove in practice. Here we argue that the recent development of transgenic molecular devices will encourage a shift from mechanistic investigations of synaptic plasticity in single neurons towards an analysis of how networks of neurons encode and represent memory, and we suggest ways in which this might be achieved. In the process, the hypothesis that synaptic plasticity is necessary and sufficient for information storage in the brain may finally be validated.     

Hippocampus: cognitive processes and neural representations that underlie declarative memory

The hippocampus serves a critical role in declarative memory--our capacity to recall everyday facts and events. Recent studies using functional brain imaging in humans and neuropsychological analyses of humans and animals with hippocampal damage have revealed some of the elemental cognitive processes mediated by the hippocampus. In addition, recent characterizations of neuronal firing patterns in behaving animals and humans have suggested how neural representations in the hippocampus underlie those elemental cognitive processes in the service of declarative memory.