A self-organizing neural network with a function of associative memory: Feedback-type cognitron

Previously, one of the authors proposed a new hypothesis on the organization of synaptic connections, and constructed a model of self-organizing multi-layered neural network cognitron (Fukushima, 1975). the cognitron consists of a number of neural layers with similar structure connected in a cascade one after another. We have modified the structure of the cognitron, and have developed a new network having an ability of associative memory. The new network, named a feedback-type cognitron, has not only the feedforward connections as in the conventional cognitron, but also modifiable feedback connections from the last-layer cells to the front-layer ones. This network has been simulated on a digital computer. If several stimulus patterns are repeatedly presented to the network, the interconnections between the cells are gradually organized. The feedback connections, as well as the conventional feedforward ones, are self-organized depending on the characteristies of the externally presented stimulus patterns. After adequate number of stimulus presentations, each cell usually acquires the selective responsiveness to one of the stimulus patterns which have been frequently given. That is, every different stimulus pattern becomes to elicit an individual response to the network. After the completion of the self-organization, several stimulus patterns are presented to the network, and the responses are observed. Once a stimulus is given to the network, the signal keeps circulating in the network even after cutting off the stimulus, and the response gradually changes. Even though an imperfect or an ambiguous pattern is presented, the response usually converges to one of the patterns which have been frequently given during the process of self-organization. In some cases, however, a new pattern which has never been presented before, emerges. It is seen that this feedback-type cognitron has characteristics quite similar to some functions of the brain, such as the associative recall of memory, or the creation of a new idea by intuition.     

Specificity of cortical synaptic connectivity: Emphasis on perspectives gained from quantitative electron microscopy

This report traces the historical development of concepts regarding the specificity of synaptic connectivity in the cerebral cortex as viewed primarily from the perspective of electron microscopy. The occurrence of stereotypical patterns of connection (e.g., contrasting synaptic patterns on the surfaces of spiny vs. non-spiny neurons, the general consistency with which axonal pathways impinge on and originate within specific cortical areas and layers, triadic synaptic relationships) implies that cortical connectivity is highly structured. The high degree of order characterizing many aspects of cortical organization is mirrored by an equally ordered arrangement of synaptic connections between specific types of neurons. This observation is based on quantitative electron microscopic studies of synapses between identified neurons and from the results of correlative anatomical/electrophysiological investigations. The recognition of recurring synaptic patterns and responses between specific neurons has generated increased support for the notion of specificity of synaptic connections at the expense of randomness, but the role of specificity in cortical function is an unresolved question. At the core of cortical processing lie myriad possibilities for computation provided by the wealth of synaptic connections involving each cortical neuron. Specificity, by limiting possibilities for connection, can impose an order on synaptic interactions even as processes of dynamic selection or synaptic remodeling ensure the constant formation and dissolution of cortical circuits. These operations make maximal use of the richness of cortical synaptic connections to produce a highly flexible system, irrespective of the degree of randomness or specificity that obtains for cortical wiring at any particular time.     

The immune system as a neural network: a multi-epitope approach.

The term "neural network" has been applied to arrays of simple activation units linked by weighted connections. If the connections are modified according to a defined learning algorithm, such networks can be trained to store and retrieve patterned information. Memories are distributed throughout the network, allowing the network to recall complete patterns from incomplete input (pattern completion). The major biological application of neural network theory to date has been in the neurosciences, but the immune system may represent an alternative organ system in which to search for neural network architecture. Previous applications of parallel distributed processing to idiotype network theory have focused upon the recognition of individual epitopes. We argue here that this approach may be too restrictive, underestimating the power of neural network architecture. We propose that the network stores and retrieves large, complex patterns consisting of multiple epitopes separated in time and space. Such a network would be capable of perceiving an entire bacterium, and of storing the time course of a viral infection. While recognition of solitary epitopes occurs at the cellular level in this model, recognition of structures larger than the width of an antibody binding site takes place at the organ level, via network architecture integration of, i.e. individual epitope responses. The Oudin-Cazenave enigma, the sharing of idiotypic determinants by antibodies directed against distinct regions of the same antigen, suggests that some network level of integration of the individual clonal responses to large antigens does occur. The role of cytokines in prior neural network models of the immune system is unclear. We speculate that cytokines may influence the temperature of the network, such that changes in the cytokine milieu serve to "anneal" the network, allowing it to achieve the optimum steady-state in the shortest period of time.     

Recognition of general patterns using neural networks

The Hopfield model of neural network stores memory in its symmetric synaptic connections and can only learn to recognize sets of nearly orthogonal patterns. A new algorithm is put forth to permit the recognition of general (non-orthogonal) patterns. The algorithm specifies the construction of the new network's memory matrix T _ij, which is, in general, asymmetrical and contains the Hopfield neural network (Hopfield 1982) as a special case. We find further that in addition to this new algorithm for general pattern recognition, there exists in fact a large class of T _ij memory matrices which permit the recognition of non-orthogonal patterns. The general form of this class of T _ij memory matrix is presented, and the projection matrix neural network (Personnaz et al. 1985) is found as a special case of this general form. This general form of memory matrix extends the library of memory matrices which allow a neural network to recognize non-orthogonal patterns. A neural network which followed this general form of memory matrix was modeled on a computer and successfully recognized a set of non-orthogonal patterns. The new network also showed a tolerance for altered and incomplete data. Through this new method, general patterns may be taught to the neural network.     

植物与病原微生物互作分子基础的研究进展

植物在与病原微生物共同进化过程中形成了复杂的免疫防卫体系。植物的先天免疫系统可大致分为两个层面。第一个层面的免疫基于细胞表面的模式识别受体对病原物相关分子模式的识别,该免疫过程被称为病原物相关分子模式触发的免疫(PAMP-triggered immunity,PTI),能帮助植物抵抗大部分病原微生物;第二个层面的免疫起始于细胞内部,主要依靠抗病基因编码的蛋白产物直接或间接识别病原微生物分泌的效应子并且激发防卫反应,来抵抗那些能够利用效应子抑制第一层面免疫的病原微生物,这一过程被称为效应子触发的免疫(Effector-triggered immunity,ETI)。这两个层面的免疫都是基于植物对"自我"及"非我"的识别,依靠MAPK级联等信号网络,将识别结果传递到细胞核内,调控相应基因的表达,做出适当的免疫应答。本文着重阐述了植物与病原微生物互作过程中不同层面的免疫反应所发生主要事件的分子基础及研究进展。     

A recurrent neural network for partitioning of hand drawn characters into strokes of different orientations.

A neural network is described which is intended to extract orientation features that should be used for recognition of hand drawn characters. The network partitions the input hand drawn characters into separate line segments (strokes) according to their orientations. The network consists of several neural layers; each layer serves for extracting strokes of a certain orientation. Every neural layer has one-to-one correspondence with an input screen. The network uses an iterative update procedure which includes interactions of neurons inside each layer through oriented excitatory connections and inhibitory interrelations between the corresponding neurons of different layers. Computer simulation of the network was performed. Experiments showed that the network efficiently classifies all pixels of any hand drawn characters according to the orientations of the strokes constituting these characters and performs, as a result of that, a reasonable segmentation of characters.     

A self-organizing neural network sharing features of the mammalian visual system

This paper describes a neural network model whose structure is designed to closely fit neuroanatomical and-physiological data, and not to be most suitable for rigorous mathematical analysis.It is shown by computer simulation that a process of self-organization that departs from a fixed retinotopic order at peripheral layers and includes hebbian modifications of synaptic connectivity at higher processing levels leads to a system that is capable of mimicking various functions of visual systems:In the initial state the overall structure of the network is preset, individual connections at higher levels are randomly selected and their strength is initialized with random numbers.For this model the outcome of the self-organization process is determined by the stimulation during the developmental phase. Depending on the type of stimuli used the model can either develop towards a featureselective preprocessor stage in a complex vision system or towards a subsystem for associative recall of abstract patterns.This flexibility supports the hypothesis that the principles embodied are rather universal and can account for the development of various nervous system structures.Presented at teh 9th Cybernetics-Congress, Göttingen, March 1986     

A neural network model for the mechanism of feature-extraction

We propose a new multilayered neural network model which has the ability of rapid self-organization. This model is a modified version of the cognitron (Fukushima, 1975). It has modifiable inhibitory feedback connections, as well as conventional modifiable excitatory feedforward connections, between the cells of adjoining layers. If a feature-extracting cell in the network is excited by a stimulus which is already familiar to the network, the cell immediately feeds back inhibitory signals to its presynaptic cells in the preceding layer, which suppresses their response. On the other hand, the feature-extracting cell does not respond to an unfamiliar feature, and the responses from its presynaptic cells are therefore not suppressed because they do not receive any feedback inhibition. Modifiable synapses in the new network are reinforced in a way similar to those in the cognitron, and synaptic connections from cells yielding a large sustained output are reinforced. Since familiar stimulus features do not elicit a sustained response from the cells of the network, only circuits which detect novel stimulus features develop. The network therefore quickly acquires favorable pattern-selectivity by the mere repetitive presentation of set of learning patterns.     

Columnar Connectivity and Laminar Processing in Cat Primary Auditory Cortex

Background

Radial intra- and interlaminar connections form a basic microcircuit in primary auditory cortex (AI) that extracts acoustic information and distributes it to cortical and subcortical networks. Though the structure of this microcircuit is known, we do not know how the functional connectivity between layers relates to laminar processing.

Methodology/Principal Findings

We studied the relationships between functional connectivity and receptive field properties in this columnar microcircuit by simultaneously recording from single neurons in cat AI in response to broadband dynamic moving ripple stimuli. We used spectrotemporal receptive fields (STRFs) to estimate the relationship between receptive field parameters and the functional connectivity between pairs of neurons. Interlaminar connectivity obtained through cross-covariance analysis reflected a consistent pattern of information flow from thalamic input layers to cortical output layers. Connection strength and STRF similarity were greatest for intralaminar neuron pairs and in supragranular layers and weaker for interlaminar projections. Interlaminar connection strength co-varied with several STRF parameters: feature selectivity, phase locking to the stimulus envelope, best temporal modulation frequency, and best spectral modulation frequency. Connectivity properties and receptive field relationships differed for vertical and horizontal connections.

Conclusions/Significance

Thus, the mode of local processing in supragranular layers differs from that in infragranular layers. Therefore, specific connectivity patterns in the auditory cortex shape the flow of information and constrain how spectrotemporal processing transformations progress in the canonical columnar auditory microcircuit.     

Competitive interactions between retinal ganglion cells during prenatal development

In the mammalian visual system, retinal ganglion cell axons terminate within the LGN in a series of alternating eye-specific layers. These layers are not present initially during development. In the cat they emerge secondarily following a prenatal period in which originally intermixed inputs from the two eyes gradually segregate from each other to give rise to the characteristic set of layers by birth. Many lines of evidence suggest that activity-dependent competitive interactions between ganglion cell axons from the two eyes for LGN neurons play an important role in the final patterning of retinogeniculate connections. Studies of the branching patterns of individual ganglion cell axons suggest that during the period when inputs from the two eyes are intermixed, axons from one eye send side branches into territory later occupied exclusively by axons from the other eye. Ultrastructural studies indicate that these branches in fact are sites of synaptic contacts, which are later eliminated since the side branches disappear as axons form their mature terminal arbors in appropriate territory. In vitro microelectrode recordings from LGN neurons indicate that they can receive convergent synaptic excitation from electrical stimulation of the optic nerves before but not after the eye-specific layers form, suggesting that at least some of the synaptic contacts seen at the ultrastructural level are functonal. Finally, experiments in which tetrodotoxin was infused intracranially during the two week period during which the eye-specific layers normally form demonstrate that it is possible to prevent, or at least delay, the formation of the layers. Accordingly, individual axons fail to develop their restricted terminal arbor branching pattern and instead branch widely throughout the LGN. These results indicate that all of the machinery necessary for synaptic function and competition is present during fetal life. Moreover, it is highly likely that neuronal activity is required for the formation of the eye-specific layers. If so, then activity would have to be present in the form of spontaneously generated action potentials, since vision is not possible at these early ages. Thus, the functioning of the retinogeniculate system many weeks before it is put to the use for which it is ultimately designed may contribute to the final patterning of connections present in the adult.     

Modeling the Seasonal Adaptation of Circadian Clocks by Changes in the Network Structure of the Suprachiasmatic Nucleus

The dynamics of circadian rhythms needs to be adapted to day length changes between summer and winter. It has been observed experimentally, however, that the dynamics of individual neurons of the suprachiasmatic nucleus (SCN) does not change as the seasons change. Rather, the seasonal adaptation of the circadian clock is hypothesized to be a consequence of changes in the intercellular dynamics, which leads to a phase distribution of electrical activity of SCN neurons that is narrower in winter and broader during summer. Yet to understand this complex intercellular dynamics, a more thorough understanding of the impact of the network structure formed by the SCN neurons is needed. To that effect, we propose a mathematical model for the dynamics of the SCN neuronal architecture in which the structure of the network plays a pivotal role. Using our model we show that the fraction of long-range cell-to-cell connections and the seasonal changes in the daily rhythms may be tightly related. In particular, simulations of the proposed mathematical model indicate that the fraction of long-range connections between the cells adjusts the phase distribution and consequently the length of the behavioral activity as follows: dense long-range connections during winter lead to a narrow activity phase, while rare long-range connections during summer lead to a broad activity phase. Our model is also able to account for the experimental observations indicating a larger light-induced phase-shift of the circadian clock during winter, which we show to be a consequence of higher synchronization between neurons. Our model thus provides evidence that the variations in the seasonal dynamics of circadian clocks can in part also be understood and regulated by the plasticity of the SCN network structure.     

Homeostatic control of synaptic rewiring in recurrent networks induces the formation of stable memory engrams

Brain networks store new memories using functional and structural synaptic plasticity. Memory formation is generally attributed to Hebbian plasticity, while homeostatic plasticity is thought to have an ancillary role in stabilizing network dynamics. Here we report that homeostatic plasticity alone can also lead to the formation of stable memories. We analyze this phenomenon using a new theory of network remodeling, combined with numerical simulations of recurrent spiking neural networks that exhibit structural plasticity based on firing rate homeostasis. These networks are able to store repeatedly presented patterns and recall them upon the presentation of incomplete cues. Storage is fast, governed by the homeostatic drift. In contrast, forgetting is slow, driven by a diffusion process. Joint stimulation of neurons induces the growth of associative connections between them, leading to the formation of memory engrams. These memories are stored in a distributed fashion throughout connectivity matrix, and individual synaptic connections have only a small influence. Although memory-specific connections are increased in number, the total number of inputs and outputs of neurons undergo only small changes during stimulation. We find that homeostatic structural plasticity induces a specific type of “silent memories”, different from conventional attractor states.     

Neuronal Pathfinding and Recognition: Roles of Cell Adhesion Molecules

Neurons contact their neighbors through a diverse array of cell adhesion and other surface molecules. These molecules can exhibit highly regulated patterns of expression, underscoring their multiple roles in establishing specific interactions between neurons and their environment. Recent studies are beginning to ask how these membrane-bound neural recognition molecules interact with each other and intracellular signaling pathways within an individual neuronal growth cone, and direct the formation of neural connections during development.     

Seasonality and heterogeneity of live fish movements in Scottish fish farms

Movement of live animals is a key contributor to disease spread. Farmed Atlantic salmon Salmo salar, rainbow trout Onchorynchus mykiss and brown/sea trout Salmo trutta are initially raised in freshwater (FW) farms; all the salmon and some of the trout are subsequently moved to seawater (SW) farms. Frequently, fish are moved between farms during their FW stage and sometimes during their SW stage. Seasonality and differences in contact patterns across production phases have been shown to influence the course of an epidemic in livestock; however, these parameters have not been included in previous network models studying disease transmission in salmonids. In Scotland, farmers are required to register fish movements onto and off their farms; these records were used in the present study to investigate seasonality and heterogeneity of movements for each production phase separately for farmed salmon, rainbow trout and brown/sea trout. Salmon FW-FW and FW-SW movements showed a higher degree of heterogeneity in number of contacts and different seasonal patterns compared with SW-SW movements. FW-FW movements peaked from May to July and FW-SW movements peaked from March to April and from October to November. Salmon SW-SW movements occurred more consistently over the year and showed fewer connections and number of repeated connections between farms. Therefore, the salmon SW-SW network might be treated as homogeneous regarding the number of connections between farms and without seasonality. However, seasonality and production phase should be included in simulation models concerning FW-FW and FW-SW movements specifically. The number of rainbow trout FW-FW and brown/sea trout FW-FW movements were different from random. However, movements from other production phases were too low to discern a seasonal pattern or differences in contact pattern.     

Stabilization of bimanual coordination due to active interhemispheric inhibition: a dynamical account

Based on recent brain-imaging data and congruent theoretical insights, a dynamical model is derived to account for the patterns of brain activity observed during stable performance of bimanual multifrequency patterns, as well as during behavioral instabilities in the form of phase transitions between such patterns. The model incorporates four dynamical processes, defined over both motor and premotor cortices, which are coupled through inhibitory and excitatory inter- and intrahemispheric connections. In particular, the model underscores the crucial role of interhemispheric inhibition in reducing the interference between disparate frequencies during stable performance, as well as the failure of this reduction during behavioral transitions. As an aside, the model also accounts for in- and antiphase preferences during isofrequency movements. The viability of the proposed model is illustrated by magnetoencephalographic signals that were recorded from an experienced subject performing a polyrhythmic tapping task that was designed to induce transitions between multifrequency patterns. Consistent with the models dynamics, contra- and ipsilateral cortical areas of activation were frequency- and phase-locked, while their activation strength changed markedly in the vicinity of transitions in coordination.     

Spike-timing-dependent plasticity leads to gamma band responses in a neural network

Early gamma band responses of the human electroencephalogram have been identified as an early interface linking top-down and bottom-up processing. This was based on findings that observed strong sensitivity of this signal to stimulus size and at the same time, to processes of attention and memory. Here, we simulate these findings in a simple random network of biologically plausible spiking neurons. During a learning phase, different stimuli were presented to the network and the synaptic connections were modified according to a spike-timing-dependent plasticity learning rule. In a subsequent test phase, we stimulated the network with (i) patterns of different sizes to simulate bottom-up effects and (ii) with patterns that were or were not presented during the learning phase. The network displayed qualitatively similar behavior as early gamma band responses measured from the scalp of human subjects: there was a general increase in response strength with increasing stimulus size and stronger responses for learned stimuli. We demonstrated that within one neural architecture early gamma band responses can be modulated both by bottom-up factors and by basal learning mechanisms mediated via spike-timing-dependent plasticity.     

Constructive Autoassociative Neural Network for Facial Recognition

Autoassociative artificial neural networks have been used in many different computer vision applications. However, it is difficult to define the most suitable neural network architecture because this definition is based on previous knowledge and depends on the problem domain. To address this problem, we propose a constructive autoassociative neural network called CANet (Constructive Autoassociative Neural Network). CANet integrates the concepts of receptive fields and autoassociative memory in a dynamic architecture that changes the configuration of the receptive fields by adding new neurons in the hidden layer, while a pruning algorithm removes neurons from the output layer. Neurons in the CANet output layer present lateral inhibitory connections that improve the recognition rate. Experiments in face recognition and facial expression recognition show that the CANet outperforms other methods presented in the literature.     

Resting-State Temporal Synchronization Networks Emerge from Connectivity Topology and Heterogeneity

Spatial patterns of coherent activity across different brain areas have been identified during the resting-state fluctuations of the brain. However, recent studies indicate that resting-state activity is not stationary, but shows complex temporal dynamics. We were interested in the spatiotemporal dynamics of the phase interactions among resting-state fMRI BOLD signals from human subjects. We found that the global phase synchrony of the BOLD signals evolves on a characteristic ultra-slow (<0.01Hz) time scale, and that its temporal variations reflect the transient formation and dissolution of multiple communities of synchronized brain regions. Synchronized communities reoccurred intermittently in time and across scanning sessions. We found that the synchronization communities relate to previously defined functional networks known to be engaged in sensory-motor or cognitive function, called resting-state networks (RSNs), including the default mode network, the somato-motor network, the visual network, the auditory network, the cognitive control networks, the self-referential network, and combinations of these and other RSNs. We studied the mechanism originating the observed spatiotemporal synchronization dynamics by using a network model of phase oscillators connected through the brain’s anatomical connectivity estimated using diffusion imaging human data. The model consistently approximates the temporal and spatial synchronization patterns of the empirical data, and reveals that multiple clusters that transiently synchronize and desynchronize emerge from the complex topology of anatomical connections, provided that oscillators are heterogeneous.     

Emergence of preferred firing sequences in large spiking neural networks during simulated neuronal development

Two main processes concurrently refine the nervous system over the course of development: cell death and selective synaptic pruning. We simulated large spiking neural networks (100 x 100 neurons "at birth") characterized by an early developmental phase with cell death due to excessive firing rate, followed by the onset of spike timing dependent synaptic plasticity (STDP), driven by spatiotemporal patterns of stimulation. The cell death affected the inhibitory units more than the excitatory units during the early developmental phase. The network activity showed the appearance of recurrent spatiotemporal firing patterns along the STDP phase, thus suggesting the emergence of cell assemblies from the initially randomly connected networks. Some of these patterns were detected throughout the simulation despite the activity-driven network modifications while others disappeared.     

Conductance-based refractory density model of primary visual cortex

A layered continual population model of primary visual cortex has been constructed, which reproduces a set of experimental data, including postsynaptic responses of single neurons on extracellular electric stimulation and spatially distributed activity patterns in response to visual stimulation. In the model, synaptically interacting excitatory and inhibitory neuronal populations are described by a conductance-based refractory density approach. Populations of two-compartment excitatory and inhibitory neurons in cortical layers 2/3 and 4 are distributed in the 2-d cortical space and connected by AMPA, NMDA and GABA type synapses. The external connections are pinwheel-like, according to the orientation of a stimulus. Intracortical connections are isotropic local and patchy between neurons with similar orientations. The model proposes better temporal resolution and more detailed elaboration than conventional mean-field models. In comparison to large network simulations, it excludes a posteriori statistical data manipulation and provides better computational efficiency and minimal parametrization.