On the behavior of some associative neural networks

Since Hopfield published his work on an associative memory model, a large number of works have studied the model from several angles and showed in particular its weaknesses, and presented ways to overcome them. Most of the proposed solutions seem to us however not biologically plausible. In this paper we present a simple statistical analysis of two networks similar to the Hopfield net, and show that the usage of positive feedback enhances the net recognizing capability without jeopardizing the stability. We also describe a layered parallel network composed of modules, each module being a modified Hopfield net. We finally present computer simulation results to support our analytical findings. The most important principles of this network are supported by data from the world of neurobiology.     

A new neural network architecture with associative memory, pruning and order-sensitive learning

A new paradigm of neural network architecture is proposed that works as associative memory along with capabilities of pruning and order-sensitive learning. The network has a composite structure wherein each node of the network is a Hopfield network by itself. The Hopfield network employs an order-sensitive learning technique and converges to user-specified stable states without having any spurious states. This is based on geometrical structure of the network and of the energy function. The network is so designed that it allows pruning in binary order as it progressively carries out associative memory retrieval. The capacity of the network is 2n, where n is the number of basic nodes in the network. The capabilities of the network are demonstrated by experimenting on three different application areas, namely a Library Database, a Protein Structure Database and Natural Language Understanding.     

Internal representations for associative memory

We describe a class of feed forward neural network models for associative content addressable memory (ACAM) which utilize sparse internal representations for stored data. In addition to the input and output layers, our networks incorporate an intermediate processing layer which serves to label each stored memory and to perform error correction and association. We study two classes of internal label representations: the unary representation and various sparse, distributed representations. Finally, we consider storage of sparse data and sparsification of data. These models are found to have advantages in terms of storage capacity, hardware efficiency, and recall reliability when compared to the Hopfield model, and to possess analogies to both biological neural networks and standard digital computer memories.     

Chaotic synchronization using a network of neural oscillators

Synchronization of chaotic low-dimensional systems has been a topic of much recent research. Such systems have found applications for secure communications. In this work we show how synchronization can be achieved in a high-dimensional chaotic neural network. The network used in our studies is an extension of the Hopfield Network, known as the Complex Hopfield Network (CHN). The CHN, also an associative memory, has both fixed point and limit cycle or oscillatory behavior. In the oscillatory mode, the network wanders chaotically from one stored pattern to another. We show how a pair of identical high-dimensional CHNs can be synchronized by communicating only a subset of state vector components. The synchronizability of such a system is characterized through simulations.     

Article for analog vector algebra computation

We introduce the concept of an analog neural network represented by chemical operations performed on strands of DNA. This new type of DNA computing has the advantage that it should be fault tolerant and thus more immune to DNA hybridization errors than a Boolean DNA computer. We describe a particular set of DNA operations to effect the interconversion of electrical and DNA data and to represent the Hopfield associative memory and the feed-forward neural network of Rumelhart et al. We speculate that networks containing as many as 10(9) neurons might be feasible.     

Modelling studies on the computational function of fast temporal structure in cortical circuit activity

The interplay between modelling and experimental studies can support the exploration of the function of neuronal circuits in the cortex. We exemplify such an approach with a study on the role of spike timing and gamma-oscillations in associative memory in strongly connected circuits of cortical neurones. It is demonstrated how associative memory studies on different levels of abstraction can specify the functionality to be expected in real cortical neuronal circuits. In our model overlapping random configurations of sparse cell populations correspond to memory items that are stored by simple Hebbian coincidence learning. This associative memory task will be implemented with biophysically well tested compartmental neurones developed by Pinsky and Rinzel . We ran simulation experiments to study memory recall in two network architectures: one interconnected pool of cells, and two reciprocally connected pools. When recalling a memory by stimulating a spatially overlapping set of cells, the completed pattern is coded by an event of synchronized single spikes occurring after 25-60 ms. These fast associations are performed even at a memory load corresponding to the memory capacity of optimally tuned formal associative networks (>0.1 bit/synapse). With tonic stimulation or feedback loops in the network the neurones fire periodically in the gamma-frequency range (20-80 Hz). With fast changing inputs memory recall can be switched between items within a single gamma cycle. Thus, oscillation is not a primary coding feature necessary for associative memory. However, it accompanies reverberatory feedback providing an improved iterative memory recall completed after a few gamma cycles (60-260 ms). In the bidirectional architecture reverberations do not express in a rigid phase locking between the pools. For small stimulation sets bursting occurred in these cells acting as a supportive mechanism for associative memory.     

选择方向强化学习的神经网络模型

提出了神经网络模型的一种选择方向强化学习规则,定义并导出了新模型与Hopfield模型两种不同的筛选曲线,由此表明新模型对相关图样的分辨力优于Hopfield模型。在微机上模拟了由100个神经元构成的网络,结果显示新模型具有重复记忆这一神经生理学特点。定义并分行了记忆强度因子,模拟结果表明记忆强度因子愈大的记忆态,联想性能愈好,学习周期愈短。     

An after-shannon measure of the storage capacity of an associative noise-like coding memory

The maximum amount of information that can be stored, on the average, in each storage element, according to an associative scheme, has been measured for the memory model proposed by the author (Bottini 1980). In this model, the (binary) items being stored are coded by noise-like keys and the memory traces formed in this way are superimposed, by algebraic addition, on the same many-level storage elements. It is shown that the problem of measuring the information retrieved from the memory in a single recall and the problem — concerning the data-communication field —of measuring the information transmitted over a noisy channel are formally similar. In particular, the Shannon noisy-channel coding theorem can find an application also in our case of an associative memory. Finally, it is evidenced that the so-called matrix model of an associative memory has the same storage capacity as the model studied here.     

Molecular associative memory

An example of a kinetic system with address-bearing molecules and directed interactions is investigated. We show that by introducing exchange between Ising spins via the address-bearing messengers, the Hopfield model of associative memory can be made local.     

轻度认知损伤患者的联系记忆损伤特征

遗忘型轻度认知损伤患者(aMCI)在项目记忆和联系记忆上都有损伤．本文通过临床记忆量表中的项目记忆和联系记忆测验,研究aMCI的联系记忆是否比项目记忆有更显著的损伤．另外,通过分析配对联想学习测验,进一步研究aMCI联系记忆损伤的特点．25名aMCI和28名健康老人参与了两个联系记忆测验(配对联想学习测验和联想回忆测验)和两个项目记忆测验(图像自由回忆和无意义图形再认),aMCI患者在联系记忆测验上表现出了更显著的损伤,即使控制了项目记忆的损伤,aMCI的联系记忆仍然比健康老人显著降低．另外,ROC分析表明联系记忆测验比项目记忆测验对aMCI病人有更高的区分度．对配对联想学习测验的分析表明,相对于健康老人,aMCI患者在记忆有语言联系的词对要比记忆无语义联系的词对更为困难．本研究进一步表明aMCI患者的联系记忆比项目记忆有更大的损伤．相对于健康老人,aMCI患者不仅难以在两个无关项目间创建记忆连接,而且在有效利用项目间本身的语义联系方面存在更大的损伤．联系记忆测验比项目记忆测验对aMCI患者有更高的区分度．在神经心理评估中增加联系记忆测验,能更加有效地识别aMCI患者．     

Acetylcholine and associative memory in the piriform cortex

The significance of cholinergic modulation for associative memory performance in the piriform cortex was examined in a study combining cellular neurophysiology in brain slices with realistic biophysical network simulations. Three different physiological effects of acetylcholine were identified at the single-cell level: suppression of neuronal adaptation, suppression of synaptic transmission in the intrinsic fibers layer, and activity-dependent increase in synaptic strength. Biophysical simulations show how these three effects are joined together to enhance learning and recall performance of the cortical network. Furthermore, our data suggest that activity-dependent synaptic decay during learning is a crucial factor in determining learning capability of the cortical network. Accordingly, it is predicted that acetylcholine should also enhance long-term depression in the piriform cortex.     

条件联想神经网络研究

在人脑的某些功能和神经系统中的突前抑制机制启发下,本文提出一个新型的神经网络模型——条件联想神经网络.模型是一个有突触前抑制的联想记忆神经网络.通过初步分析和计算机模拟,证明本模型具有一般联想记忆模型所未有的一些新的特性,如可以在不同条件下,对同一输入有不同的反应.对同一输入,在不同的条件下,又可以有相同的反应.这些特点将有助于人们对神经系统中信息处理过程的了解.此外,文中也指出可能实现本模型的神经结构.     

Associative Memory and Segmentation in an Oscillatory Neural Model of the Olfactory Bulb

We discuss the first few stages of olfactory processing in the framework of a layered neural network. Its central component is an oscillatory associative memory, describing the external plexiform layer, that consists of inhibitory and excitatory neurons with dendrodendritic interactions. We explore the computational properties of this neural network and point out its possible functional role in the olfactory bulb. When receiving a complex input that is composed of several odors, the network segments it into its components. This is done in two stages. First, multiple odor input is preprocessed in the glomerular layer via a decorrelation mechanism that relies on temporal independence of odor sources. Second, as the recall process of a pattern consists of associative convergence to an oscillatory attractor, multiple inputs are identified by alternate dominance of memory patterns during different sniff cycles. This could explain how quick analysis of mixed odors is subserved by the rapid sniffing behavior of highly olfactory animals. When one of the odors is much stronger than the rest, the network converges onto it, thus displaying odor masking.     

An artificial modular neural network and its basic dynamical characteristics

This work contains a proposition of an artificial modular neural network (MNN) in which every module network exchanges input/output information with others simultaneously. It further studies the basic dynamical characteristics of this network through both computer simulations and analytical considerations. A notable feature of this model is that it has generic representation with regard to the number of composed modules, network topologies, and classes of introduced interactions. The information processing of the MNN is described as the minimization of a total-energy function that consists of partial-energy functions for modules and their interactions, and the activity and weight dynamics are derived from the total-energy function under the Lyapunov stability condition. This concept was realized by Cross-Coupled Hopfield Nets (CCHN) that one of the authors proposed. In this paper, in order to investigate the basic dynamical properties of CCHN, we offer a representative model called Cross-Coupled Hopfield Nets with Local And Global Interactions (CCHN-LAGI) to which two distinct classes of interactions – local and global interactions – are introduced. Through a conventional test for associative memories, it is confirmed that our energy-function-based approach gives us proper dynamics of CCHN-LAGI even if the networks have different modularity. We also discuss the contribution of a single interaction and the joint contribution of the two distinct interactions through the eigenvalue analysis of connection matrices. Received: 18 July 1995 / Accepted in revised form: 2 October 1997     

On the dynamic organization of memory

The interaction of memory structures and retrieval dynamics is discussed. A mathematical model for associative free recall is presented to support the view that the organization of simple processing units plays an important role in the retrieval of memory traces. Computer simulations show that "flexibility" and "fidelity" of the dynamics strongly depend on the network structure, the amplification and decay parameters, and the noise term.     

Neuronal Mechanisms Encoding Global-to-Fine Information in Inferior-Temporal Cortex

Sugase et al. found that global information is represented at the initial transient firing of a single face-responsive neuron in inferior-temporal (IT) cortex, and that finer information is represented at the subsequent sustained firing. A feed-forward model and an attractor network are conceivable models to reproduce this dynamics. The attractor network, specifically an associative memory model, is employed to elucidate the neuronal mechanisms producing the dynamics. The results obtained by computer simulations show that a state of neuronal population initially approaches to a mean state of similar memory patterns, and that it finally converges to a memory pattern. This dynamics qualitatively coincides with that of face-responsive neurons. The dynamics of a single neuron in the model also coincides with that of a single face-responsive neuron. Furthermore, we propose two physiological experiments and predict the results from our model. Both predicted results are not explainable by the feed-forward model. Therefore, if the results obtained by actual physiological experiments coincide with our predicted results, the attractor network might be the neuronal mechanisms producing the dynamics of face-responsive neurons.     

Improving recall from an associative memory

The associative net as a model of biological associative memory is investigated. Calculating the output pattern retrieved from a partially connected associative net presented with noisy input cues involves several computations. This is complicated by variations in the dendritic sums of the output units due to errors in the cue and differences in input activity and unit usage. The possible implementation of these computations by biological neural machinery is unclear. We demonstrate that a relatively simple transformation can reduce variation in the dendritic sums. This leads to a winners-take-all type of strategy that produces increased recall performance which is equivalent to the more complicated optimal strategy proposed by others. We describe in detail the possible biological implications of our strategies, the novel feature of which ascribes a role to the NMDA and non-NMDA channels found in hippocampal pyramidal cells. Received: 13 April 1994 / Accepted: 25 October 1994     

Neural Population-Level Memory Traces in the Mouse Hippocampus

One of the fundamental goals in neurosciences is to elucidate the formation and retrieval of brain''s associative memory traces in real-time. Here, we describe real-time neural ensemble transient dynamics in the mouse hippocampal CA1 region and demonstrate their relationships with behavioral performances during both learning and recall. We employed the classic trace fear conditioning paradigm involving a neutral tone followed by a mild foot-shock 20 seconds later. Our large-scale recording and decoding methods revealed that conditioned tone responses and tone-shock association patterns were not present in CA1 during the first pairing, but emerged quickly after multiple pairings. These encoding patterns showed increased immediate-replay, correlating tightly with increased immediate-freezing during learning. Moreover, during contextual recall, these patterns reappeared in tandem six-to-fourteen times per minute, again correlating tightly with behavioral recall. Upon traced tone recall, while various fear memories were retrieved, the shock traces exhibited a unique recall-peak around the 20-second trace interval, further signifying the memory of time for the expected shock. Therefore, our study has revealed various real-time associative memory traces during learning and recall in CA1, and demonstrates that real-time memory traces can be decoded on a moment-to-moment basis over any single trial.     

Associative learning in a network model of Hermissenda crassicornis

A companion paper in a previous issue of this journal presented a resistance-capacitance circuit computer model of the four-neuron visual-vestibular network of the invertebrate marine mollusk Hermissenda crassicornis. In the present paper, we demonstrate that changes in the model's output in response to simulated associative training is quantitatively similar to behavioral and electrophysiological changes in response to associative training of Hermissenda crassicornis. Specifically, the model demonstrates many characteristics of conditioning: sensitivity to stimulus contingency, stimulus specificity, extinction, and savings. The model's learning features also are shown to be devoid of non-associative components. Thus, this computational model is an excellent tool for examining the information flow and dynamics of biological associative learning and for uncovering insights concerning associative learning, memory, and recall that can be applied to the development of artificial neural networks.     

A hierarchical neural network model for associative memory

A hierarchical neural network model with feedback interconnections, which has the function of associative memory and the ability to recognize patterns, is proposed. The model consists of a hierarchical multi-layered network to which efferent connections are added, so as to make positive feedback loops in pairs with afferent connections. The cell-layer at the initial stage of the network is the input layer which receives the stimulus input and at the same time works as an output layer for associative recall. The deepest layer is the output layer for pattern-recognition. Pattern-recognition is performed hierarchically by integrating information by converging afferent paths in the network. For the purpose of associative recall, the integrated information is again distributed to lower-order cells by diverging efferent paths. These two operations progress simultaneously in the network. If a fragment of a training pattern is presented to the network which has completed its self-organization, the entire pattern will gradually be recalled in the initial layer. If a stimulus consisting of a number of training patterns superposed is presented, one pattern gradually becomes predominant in the recalled output after competition between the patterns, and the others disappear. At about the same time when the recalled pattern reaches a steady state in he initial layer, in the deepest layer of the network, a response is elicited from the cell corresponding to the category of the finally-recalled pattern. Once a steady state has been reached, the response of the network is automatically extinguished by inhibitory signals from a steadiness-detecting cell. If the same stimulus is still presented after inhibition, a response for another pattern, formerly suppressed, will now appear, because the cells of the network have adaptation characteristics which makes the same response unlikely to recur. Since inhibition occurs repeatedly, the superposed input patterns are recalled one by one in turn.