A holographic model for associative memory chains

Holographic brain models are well suited to describe specific brain functions. Central nervous systems and holographic systems both show parallel information processing and non-localized storage in common. To process information both systems use correlation functions suggesting to develop cybernetical brain models in terms of holography. Associative holographic storage is done with two simultaneously existing patterns. They may reconstruct each other mutually. Time-sequentially existing patterns are connected to associative chains, if every two succeeding patterns do exist within a common period of time in order to be stored in pairs. Read out (recall) of associative chains—reconstructing coupled patterns which didn't exist simultaneously—requires advanced holographic techniques. Three different methods are described and tested experimentally. The underlying principles are feedback mechanisms, nonlinearities of the storage material and tridimensional architecture of the voluminous recording medium. Those principles evidently occur in neural storage systems supporting analogous information processing in neural- and holographic systems.     

A self-organizing neural network with a function of associative memory: Feedback-type cognitron

Previously, one of the authors proposed a new hypothesis on the organization of synaptic connections, and constructed a model of self-organizing multi-layered neural network cognitron (Fukushima, 1975). the cognitron consists of a number of neural layers with similar structure connected in a cascade one after another. We have modified the structure of the cognitron, and have developed a new network having an ability of associative memory. The new network, named a feedback-type cognitron, has not only the feedforward connections as in the conventional cognitron, but also modifiable feedback connections from the last-layer cells to the front-layer ones. This network has been simulated on a digital computer. If several stimulus patterns are repeatedly presented to the network, the interconnections between the cells are gradually organized. The feedback connections, as well as the conventional feedforward ones, are self-organized depending on the characteristies of the externally presented stimulus patterns. After adequate number of stimulus presentations, each cell usually acquires the selective responsiveness to one of the stimulus patterns which have been frequently given. That is, every different stimulus pattern becomes to elicit an individual response to the network. After the completion of the self-organization, several stimulus patterns are presented to the network, and the responses are observed. Once a stimulus is given to the network, the signal keeps circulating in the network even after cutting off the stimulus, and the response gradually changes. Even though an imperfect or an ambiguous pattern is presented, the response usually converges to one of the patterns which have been frequently given during the process of self-organization. In some cases, however, a new pattern which has never been presented before, emerges. It is seen that this feedback-type cognitron has characteristics quite similar to some functions of the brain, such as the associative recall of memory, or the creation of a new idea by intuition.     

A new neural network architecture with associative memory, pruning and order-sensitive learning

A new paradigm of neural network architecture is proposed that works as associative memory along with capabilities of pruning and order-sensitive learning. The network has a composite structure wherein each node of the network is a Hopfield network by itself. The Hopfield network employs an order-sensitive learning technique and converges to user-specified stable states without having any spurious states. This is based on geometrical structure of the network and of the energy function. The network is so designed that it allows pruning in binary order as it progressively carries out associative memory retrieval. The capacity of the network is 2n, where n is the number of basic nodes in the network. The capabilities of the network are demonstrated by experimenting on three different application areas, namely a Library Database, a Protein Structure Database and Natural Language Understanding.     

An associative memory that can form hypotheses: a phase-coded neural network

 Nonlinear associative memories as realized, e.g., by Hopfield nets are characterized by attractor-type dynamics. When fed with a starting pattern, they converge to exactly one of the stored patterns which is supposed to be most similar. These systems cannot render hypotheses of classification, i.e., render several possible answers to a given classification problem. Inspired by von der Malsburg’s correlation theory of brain function, we extend conventional neural network architectures by introducing additional dynamical variables. Assuming an oscillatory time structure of neural firing, i.e., the existence of neural clocks, we assign a so-called phase to each formal neuron. The phases explicitly describe detailed correlations of neural activities neglected in conventional neural network architectures. Implementing this extension into a simple self-organizing network based on a feature map, we present an associative memory that actually is capable of forming hypotheses of classification. Received: 6 December 1993/Accepted in revised form: 14 July 1994     

Associative search network: A reinforcement learning associative memory

An associative memory system is presented which does not require a teacher to provide the desired associations. For each input key it conducts a search for the output pattern which optimizes an external payoff or reinforcement signal. The associative search network (ASN) combines pattern recognition and function optimization capabilities in a simple and effective way. We define the associative search problem, discuss conditions under which the associative search network is capable of solving it, and present results from computer simulations. The synthesis of sensory-motor control surfaces is discussed as an example of the associative search problem.     

A model of associative memory in the brain

A model of associative memory for time varying spatial patterns is proposed and simulated on a digital computer. This is a network composed of many neuron-like elements, and shows an ability for associative memory similar to that of the brain.Suppose a number of sequences of spatial patterns are presented to this network, for example, 12345, ABC, and so on. Then, these patterns are memorized in the network. After that, if any part of one of these sequences, say 23, is presented to the circuit, the rest of the sequence, 45, is recalled following to it. It resembles to such a situation — if we hear a part of a melody which we have memorized in the past, the rest of the melody is recalled even after it is stopped half-way. Although the recalled patterns are not always 100% correct, they are not completely destroyed even if the presented patterns are imperfect.     

Dynamics control of semantic processes in a hierarchical associative memory

 A neural mechanism for control of dynamics and function of associative processes in a hierarchical memory system is demonstrated. For the representation and processing of abstract knowledge, the semantic declarative memory system of the human brain is considered. The dynamics control mechanism is based on the influence of neuronal adaptation on the complexity of neural network dynamics. Different dynamical modes correspond to different levels of the ultrametric structure of the hierarchical memory being invoked during an associative process. The mechanism is deterministic but may also underlie free associative thought processes. The formulation of an abstract neural network model of hierarchical associative memory utilizes a recent approach to incorporate neuronal adaptation. It includes a generalized neuronal activation function recently derived by a Hodgkin-Huxley-type model. It is shown that the extent to which a hierarchically organized memory structure is searched is controlled by the neuronal adaptability, i.e. the strength of coupling between neuronal activity and excitability. In the brain, the concentration of various neuromodulators in turn can regulate the adaptability. An autonomously controlled sequence of bifurcations, from an initial exploratory to a final retrieval phase, of an associative process is shown to result from an activity-dependent release of neuromodulators. The dynamics control mechanism may be important in the context of various disorders of the brain and may also extend the range of applications of artificial neural networks. Received: 19 April 1995/Accepted in revised form: 8 August 1995     

Dynamics control of semantic processes in a hierarchical associative memory

A neural mechanism for control of dynamics and function of associative processes in a hierarchical memory system is demonstrated. For the representation and processing of abstract knowledge, the semantic declarative memory system of the human brain is considered. The dynamics control mechanism is based on the influence of neuronal adaptation on the complexity of neural network dynamics. Different dynamical modes correspond to different levels of the ultrametric structure of the hierarchical memory being invoked during an associative process. The mechanism is deterministic but may also underlie free associative thought processes. The formulation of an abstract neural network model of hierarchical associative memory utilizes a recent approach to incorporate neuronal adaptation. It includes a generalized neuronal activation function recently derived by a Hodgkin-Huxley-type model. It is shown that the extent to which a hierarchically organized memory structure is searched is controlled by the neuronal adaptability, i.e. the strength of coupling between neuronal activity and excitability. In the brain, the concentration of various neuromodulators in turn can regulate the adaptability. An autonomously controlled sequence of bifurcations, from an initial exploratory to a final retrieval phase, of an associative process is shown to result from an activity-dependent release of neuromodulators. The dynamics control mechanism may be important in the context of various disorders of the brain and may also extend the range of applications of artificial neural networks.     

A hierarchical unsupervised growing neural network for clustering gene expression patterns

MOTIVATION: We describe a new approach to the analysis of gene expression data coming from DNA array experiments, using an unsupervised neural network. DNA array technologies allow monitoring thousands of genes rapidly and efficiently. One of the interests of these studies is the search for correlated gene expression patterns, and this is usually achieved by clustering them. The Self-Organising Tree Algorithm, (SOTA) (Dopazo,J. and Carazo,J.M. (1997) J. Mol. Evol., 44, 226-233), is a neural network that grows adopting the topology of a binary tree. The result of the algorithm is a hierarchical cluster obtained with the accuracy and robustness of a neural network. RESULTS: SOTA clustering confers several advantages over classical hierarchical clustering methods. SOTA is a divisive method: the clustering process is performed from top to bottom, i.e. the highest hierarchical levels are resolved before going to the details of the lowest levels. The growing can be stopped at the desired hierarchical level. Moreover, a criterion to stop the growing of the tree, based on the approximate distribution of probability obtained by randomisation of the original data set, is provided. By means of this criterion, a statistical support for the definition of clusters is proposed. In addition, obtaining average gene expression patterns is a built-in feature of the algorithm. Different neurons defining the different hierarchical levels represent the averages of the gene expression patterns contained in the clusters. Since SOTA runtimes are approximately linear with the number of items to be classified, it is especially suitable for dealing with huge amounts of data. The method proposed is very general and applies to any data providing that they can be coded as a series of numbers and that a computable measure of similarity between data items can be used. AVAILABILITY: A server running the program can be found at: http://bioinfo.cnio.es/sotarray.     

A neural network model for trace conditioning

We studied the dynamics of a neural network that has both recurrent excitatory and random inhibitory connections. Neurons started to become active when a relatively weak transient excitatory signal was presented and the activity was sustained due to the recurrent excitatory connections. The sustained activity stopped when a strong transient signal was presented or when neurons were disinhibited. The random inhibitory connections modulated the activity patterns of neurons so that the patterns evolved without recurrence with time. Hence, a time passage between the onsets of the two transient signals was represented by the sequence of activity patterns. We then applied this model to represent the trace eye blink conditioning, which is mediated by the hippocampus. We assumed this model as CA3 of the hippocampus and considered an output neuron corresponding to a neuron in CA1. The activity pattern of the output neuron was similar to that of CA1 neurons during trace eye blink conditioning, which was experimentally observed.     

A neural net model for multiple memory domains

Previous studies with neural nets constructed of discrete populations of formal neurons have assumed that all neurons have the same probability of connection with any other neuron in the net. However, in this new study we incorporate the behavior of the neural systems in which the neural connections can be set up by means of chemical markers carried by the individual cells. With this new approach we studied the dynamics of isolated neural nets again as well as the dynamics of neural nets with sustained inputs. Results obtained with this approach show simple and multiple hysteresis phenomena. Such hysteresis loops may be considered to represent the basis for short-term memory.     

A neural network model for cognitive activity

A consideration of the storage of information as an energized neuronal state leads to the development of a new type of neural network model which is capable of pattern recognition, concept formation and recognition of patterns of events in time. The network consists of several layers of cells, each cell representing by connections from the lower levels some combination of features or concepts. Information travels toward higher layers by such connections during an association phase, and then reverses during a recognition phase, where higher-order concepts can redirect the flow to more appropriate elements, revising the perception of the environment. This permits a more efficient method of distinguishing closely-related patterns and also permits the formation of negative associations, which is a likely requirement for formation of "abstract" concepts.     

A neural network model for texture discrimination

A model of texture discrimination in visual cortex was built using a feedforward network with lateral interactions among relatively realistic spiking neural elements. The elements have various membrane currents, equilibrium potentials and time constants, with action potentials and synapses. The model is derived from the modified programs of MacGregor (1987). Gabor-like filters are applied to overlapping regions in the original image; the neural network with lateral excitatory and inhibitory interactions then compares and adjusts the Gabor amplitudes in order to produce the actual texture discrimination. Finally, a combination layer selects and groups various representations in the output of the network to form the final transformed image material. We show that both texture segmentation and detection of texture boundaries can be represented in the firing activity of such a network for a wide variety of synthetic to natural images. Performance details depend most strongly on the global balance of strengths of the excitatory and inhibitory lateral interconnections. The spatial distribution of lateral connective strengths has relatively little effect. Detailed temporal firing activities of single elements in the lateral connected network were examined under various stimulus conditions. Results show (as in area 17 of cortex) that a single element's response to image features local to its receptive field can be altered by changes in the global context.     

A neural network model of semantic memory linking feature-based object representation and words

Recent theories in cognitive neuroscience suggest that semantic memory is a distributed process, which involves many cortical areas and is based on a multimodal representation of objects. The aim of this work is to extend a previous model of object representation to realize a semantic memory, in which sensory-motor representations of objects are linked with words. The model assumes that each object is described as a collection of features, coded in different cortical areas via a topological organization. Features in different objects are segmented via γ-band synchronization of neural oscillators. The feature areas are further connected with a lexical area, devoted to the representation of words. Synapses among the feature areas, and among the lexical area and the feature areas are trained via a time-dependent Hebbian rule, during a period in which individual objects are presented together with the corresponding words. Simulation results demonstrate that, during the retrieval phase, the network can deal with the simultaneous presence of objects (from sensory-motor inputs) and words (from acoustic inputs), can correctly associate objects with words and segment objects even in the presence of incomplete information. Moreover, the network can realize some semantic links among words representing objects with shared features. These results support the idea that semantic memory can be described as an integrated process, whose content is retrieved by the co-activation of different multimodal regions. In perspective, extended versions of this model may be used to test conceptual theories, and to provide a quantitative assessment of existing data (for instance concerning patients with neural deficits).     

Transitory memory retrieval in a biologically plausible neural network model

A number of memory models have been proposed. These all have the basic structure that excitatory neurons are reciprocally connected by recurrent connections together with the connections with inhibitory neurons, which yields associative memory (i.e., pattern completion) and successive retrieval of memory. In most of the models, a simple mathematical model for a neuron in the form of a discrete map is adopted. It has not, however, been clarified whether behaviors like associative memory and successive retrieval of memory appear when a biologically plausible neuron model is used. In this paper, we propose a network model for associative memory and successive retrieval of memory based on Pinsky-Rinzel neurons. The state of pattern completion in associative memory can be observed with an appropriate balance of excitatory and inhibitory connection strengths. Increasing of the connection strength of inhibitory interneurons changes the state of memory retrieval from associative memory to successive retrieval of memory. We investigate this transition.     

A discrete fully recurrent network of max product units for associative memory and classification

This paper defines the truncated normalized max product operation for the transformation of states of a network and provides a method for solving a set of equations based on this operation. The operation serves as the transformation for the set of fully connected units in a recurrent network that otherwise might consist of linear threshold units. Component values of the state vector and outputs of the units take on the values in the set [0, 0.1,..., 0.9, 1]. The result is a much larger state space given a particular number of units and size of connection matrix than for a network based on threshold units. Since the operation defined here can form the basis of transformations in a recurrent network with a finite number of states, fixed points or cycles are possible and the network based on this operation for transformations can be used as an associative memory or pattern classifier with fixed points taking on the role of prototypes. Discrete fully recurrent networks have proven themselves to be very useful as associative memories and as classifiers. However they are often based on units that have binary states. The effect of this is that the data to be processed consisting of vectors in R(n) have to be converted to vectors in [0, 1]m with m much larger than n since binary encoding based on positional notation is not feasible. This implies a large increase in the number of components. The effect can be lessened by allowing more states for each unit in our network. The network proposed demonstrates those properties that are desirable in an associative memory very well as the simulations show.     

A neural network model for limb trajectory formation

This paper deals with the problem of representing and generating unconstrained aiming movements of a limb by means of a neural network architecture. The network produced time trajectories of a limb from a starting posture toward targets specified by sensory stimuli. Thus the network performed a sensory-motor transformation. The experimenters trained the network using a bell-shaped velocity profile on the trajectories. This type of profile is characteristic of most movements performed by biological systems. We investigated the generalization capabilities of the network as well as its internal organization. Experiments performed during learning and on the trained network showed that: (i) the task could be learned by a three-layer sequential network; (ii) the network successfully generalized in trajectory space and adjusted the velocity profiles properly; (iii) the same task could not be learned by a linear network; (iv) after learning, the internal connections became organized into inhibitory and excitatory zones and encoded the main features of the training set; (v) the model was robust to noise on the input signals; (vi) the network exhibited attractor-dynamics properties; (vii) the network was able to solve the motorequivalence problem. A key feature of this work is the fact that the neural network was coupled to a mechanical model of a limb in which muscles are represented as springs. With this representation the model solved the problem of motor redundancy.     

A neural network model for stroboscopic alternative motion

A neural network which models multistable perception is presented. The network consists of sensor and inner neurons. The dynamics is established by a stochastic neuronal dynamics, a formal Hebb-type coupling dynamics and a resource mechanism that corresponds to saturation effects in perception. From this a system of coupled differential equations is derived and analyzed. Single stimuli are bound to exactly one percept, even in ambiguous situations where multistability occurs. The network exhibits discontinuous as well as continuous phase transitions and models various empirical findings, including the percepts of succession, alternative motion and simultaneity; the percept of oscillation is explained by oscillating percepts at a continuous phase transition. Received: 13 September 1995 / Accepted: 3 June 1996