A neural network model for selective attention in visual pattern recognition

A neural network model of the mechanism of selective attention in visual pattern recognition is proposed and simulated on a digital computer.When a complex figure consisting of two patterns or more is presented to the model, it is segmented into individual patterns, and each pattern is recognized separately. Even if one of the patterns to which the model is paying selective attention is affected by noise or defects, the model can recall the complete pattern from which the noise has been eliminated and the defects corrected. It is not necessary for perfect recall that the stimulus pattern should be identical in shape to the training pattern. Even though the pattern is distorted in shape or changed in size, it can be correctly recognized and the missing portions restored.The model consists of a hierarchical neural network which has efferent as well as afferent connections between cells. The afferent and the efferent signals interact with each other in the network: the efferent signals, that is, the signals for selective attention, have a facilitating effect on the afferent ones, and, at the same time, the afferent signals gate efferent signal flow. When some feature in the stimulus is not extracted in the afferent paths, the threshold for detection of that feature is automatically lowered by decreasing the efficiency of inhibition, and the model tries to extract even vague traces of the undetected feature.     

Doubly modifiable synapses: a model of short and long term auto-associative memory

Synapses that can be strengthened in temporary and persistent manners by two separate mechanisms are shown to have powerful advantages in neural networks that perform auto-associative recall and recognition. A multiplicative relation between the two weights allows the same set of connections to be used in a closely interactive way for short-term and long-term memory. Algorithms and simulations are described for the storage, consolidation and recall of patterns that have been presented only once to a network. With double modifiability, the short-term performance is dramatically improved, becoming almost independent of the amount of long-term experience. The high quality of short-term recall allows consolidation to take place, with benefits from the selection and optimization of long term engrams to take account of relations between stored patterns. Long-term capacity is greater than short-term capacity, with little or no deficit compared with that obtained with singly modifiable synapses. Long-term recall requires special, simply implemented, procedures for increasing the temporary weights of the synapses being used to initiate recall. A consolidation algorithm is described for improving long-term recall when there is overlap between patterns. Confusional errors are reduced by strengthening the associations between non-overlapping elements in the patterns, in a two-stage process that has several of the characteristics of sleep.     

A neural network model for cognitive activity

A consideration of the storage of information as an energized neuronal state leads to the development of a new type of neural network model which is capable of pattern recognition, concept formation and recognition of patterns of events in time. The network consists of several layers of cells, each cell representing by connections from the lower levels some combination of features or concepts. Information travels toward higher layers by such connections during an association phase, and then reverses during a recognition phase, where higher-order concepts can redirect the flow to more appropriate elements, revising the perception of the environment. This permits a more efficient method of distinguishing closely-related patterns and also permits the formation of negative associations, which is a likely requirement for formation of "abstract" concepts.     

Automatic target recognition based on cross-plot

Automatic target recognition that relies on rapid feature extraction of real-time target from photo-realistic imaging will enable efficient identification of target patterns. To achieve this objective, Cross-plots of binary patterns are explored as potential signatures for the observed target by high-speed capture of the crucial spatial features using minimal computational resources. Target recognition was implemented based on the proposed pattern recognition concept and tested rigorously for its precision and recall performance. We conclude that Cross-plotting is able to produce a digital fingerprint of a target that correlates efficiently and effectively to signatures of patterns having its identity in a target repository.     

Storage of a temporal pattern sequence in a network

Learning of single patterns and a temporal pattern sequence in a network when the coupling coefficients between the network elements change their values according to a definite coupling function is described. In contrast to technical systems (e.g. film, tape) where temporal sequences are often encoded in the storage location, the network stores information only by changing the values of the coupling coefficients. A network of 100 elements was simulated on an UNIVAC 1100/80 computer. Eight single patterns and a sequence of these patterns were offered at the input of the network. After the learning process the network reproduces every stored pattern as an output signal when only parts of it are fed in. The activity, that is the sum of all output signals, is regulated by an external control signal. By setting that control signal to a suitable value the network is able to reproduce the stored pattern sequence starting from any arbitrary pattern. Lowering the external control signal during that process causes the network to hold the last presented pattern until the external control signal is changed again. It is speculated that the coupling function implemented in the simulation may be anaogous to a characteristic describing the chemical process of cooperative binding.Supported by DFG (Ha 381/9 and Ha 381/11)     

The binding of visual patterns in bumblebees

Bees navigating between their nests and foraging sites rely on their ability to learn and to recall many complex visual patterns [1-4]. How are the elements that make up one of these patterns bound together so that the whole pattern can be recalled when it is required? Consider the sentence: 'Dons nod off.' The words in it can be distinguished by the pattern of elements or letters that they contain. Words may contain the same elements arranged in different orders (don, nod), or contain elements of different types, or vary in both these respects (nod, off). We show here that bumblebees (Bombus terrestris) can learn to group the elements of a pattern together, such that different identifiable patterns contain the same elements in different combinations--analogous to the grouping of letters found in words. Our results suggest that pattern binding in bees is achieved in part by linking pattern elements directly together and in part by associating the elements with cues that are related to the context in which the pattern is seen.     

Temporal classification of Drosophila segmentation gene expression patterns by the multi-valued neural recognition method

In order to reconstruct the establishment of the body pattern over time in Drosophila embryos, we have developed automated methods for detecting the age of an embryo on the basis of knowledge about its gene expression patterns. In this paper we perform temporal classification of confocal images of expression patterns of genes controlling segmentation by means of a neural network based on multi-valued neurons (MVN). MVN are artificial neural processing elements with complex-valued weights and high functionality, which proved to be efficient for solving the image recognition problems. The results obtained by this method confirm its efficiency for image recognition and indicate that the method can detect characteristic features of expression patterns which mark their development over time.     

A hierarchical neural network model for associative memory

A hierarchical neural network model with feedback interconnections, which has the function of associative memory and the ability to recognize patterns, is proposed. The model consists of a hierarchical multi-layered network to which efferent connections are added, so as to make positive feedback loops in pairs with afferent connections. The cell-layer at the initial stage of the network is the input layer which receives the stimulus input and at the same time works as an output layer for associative recall. The deepest layer is the output layer for pattern-recognition. Pattern-recognition is performed hierarchically by integrating information by converging afferent paths in the network. For the purpose of associative recall, the integrated information is again distributed to lower-order cells by diverging efferent paths. These two operations progress simultaneously in the network. If a fragment of a training pattern is presented to the network which has completed its self-organization, the entire pattern will gradually be recalled in the initial layer. If a stimulus consisting of a number of training patterns superposed is presented, one pattern gradually becomes predominant in the recalled output after competition between the patterns, and the others disappear. At about the same time when the recalled pattern reaches a steady state in he initial layer, in the deepest layer of the network, a response is elicited from the cell corresponding to the category of the finally-recalled pattern. Once a steady state has been reached, the response of the network is automatically extinguished by inhibitory signals from a steadiness-detecting cell. If the same stimulus is still presented after inhibition, a response for another pattern, formerly suppressed, will now appear, because the cells of the network have adaptation characteristics which makes the same response unlikely to recur. Since inhibition occurs repeatedly, the superposed input patterns are recalled one by one in turn.     

The immune system as a neural network: a multi-epitope approach.

The term "neural network" has been applied to arrays of simple activation units linked by weighted connections. If the connections are modified according to a defined learning algorithm, such networks can be trained to store and retrieve patterned information. Memories are distributed throughout the network, allowing the network to recall complete patterns from incomplete input (pattern completion). The major biological application of neural network theory to date has been in the neurosciences, but the immune system may represent an alternative organ system in which to search for neural network architecture. Previous applications of parallel distributed processing to idiotype network theory have focused upon the recognition of individual epitopes. We argue here that this approach may be too restrictive, underestimating the power of neural network architecture. We propose that the network stores and retrieves large, complex patterns consisting of multiple epitopes separated in time and space. Such a network would be capable of perceiving an entire bacterium, and of storing the time course of a viral infection. While recognition of solitary epitopes occurs at the cellular level in this model, recognition of structures larger than the width of an antibody binding site takes place at the organ level, via network architecture integration of, i.e. individual epitope responses. The Oudin-Cazenave enigma, the sharing of idiotypic determinants by antibodies directed against distinct regions of the same antigen, suggests that some network level of integration of the individual clonal responses to large antigens does occur. The role of cytokines in prior neural network models of the immune system is unclear. We speculate that cytokines may influence the temperature of the network, such that changes in the cytokine milieu serve to "anneal" the network, allowing it to achieve the optimum steady-state in the shortest period of time.     

Effects of dilation and translation on a perceptron-type learning rule for higher order Hopfield neural networks

In this paper, we will take a further look at a generalized perceptron-like learning rule which uses dilation and translation parameters in order to enhance the recall performance of higher order Hopfield neural networks without significantly increasing their complexity. We will practically study the influence of these parameters on the perceptron learning and recall process, using a generalized version of the Hebbian learning rule for initialization. Our analysis will be based on a pattern recognition problem with random patterns. We will see that in case of a highly correlated set of patterns, there can be gained some improvements concerning the learning and recall performance. On the other hand, we will show that the dilation and translation parameters have to be chosen carefully for a positive result.     

Learning and recall in a dynamic theory of coordination patterns

A dynamic theory of learning and recall of coordination patterns is developed in the context of relative timing skills. Characterizing the coordination patterns in such skills by the collective variable, relative phase, we choose a model system in which the intrinsic pattern dynamics as well as the influence of environmental and memorized information are well understood from previous experimental and theoretical work. To describe learning we endow memorized information with dynamics which is determined by a phenomenological strategy. Similarly, additional degrees of freedom must be introduced to understand recall. As such recall variables we choose the relative strengths with which each memorized pattern acts on the pattern dynamics and model their dynamics phenomenologically. The resulting dynamical system that resembles models used in pattern recognition theory is shown to adequately describe the learning and recall processes. Moreover, due to the operational character of the theory, several predictions emerge that are open to experimental test. In particular, we show under which conditions phase transitions occur in the dynamics of the coordination patterns during learning and during recall. Considering different time scales and their relations we demonstrate how these phase transitions can be identified and observed. Other predictions include the influence of the intrinsic pattern dynamics on the recall process and the existence of history and hysteresis effects in recall. We discuss different forms of forgetting and differentiation of memorized information. The results show how a new theoretical view of learning and recall as change of behavioral dynamics can lead to a different understanding of these processes by providing testable predictions.     

Pattern recall in networks of chaotic neurons

This research investigates the potential utility of chaotic dynamics in neural information processing. A novel chaotic spiking neural network model is presented which is composed of non-linear dynamic state (NDS) neurons. The activity of each NDS neuron is driven by a set of non-linear equations coupled with a threshold based spike output mechanism. If time-delayed self-connections are enabled then the network stabilises to a periodic pattern of activation. Previous publications of this work have demonstrated that the chaotic dynamics which drive the network activity ensure that an extremely large number of such periodic patterns can be generated by this network. This paper presents a major extension to this model which enables the network to recall a pattern of activity from a selection of previously stabilised patterns.     

A Signature of Attractor Dynamics in the CA3 Region of the Hippocampus

The notion of attractor networks is the leading hypothesis for how associative memories are stored and recalled. A defining anatomical feature of such networks is excitatory recurrent connections. These “attract” the firing pattern of the network to a stored pattern, even when the external input is incomplete (pattern completion). The CA3 region of the hippocampus has been postulated to be such an attractor network; however, the experimental evidence has been ambiguous, leading to the suggestion that CA3 is not an attractor network. In order to resolve this controversy and to better understand how CA3 functions, we simulated CA3 and its input structures. In our simulation, we could reproduce critical experimental results and establish the criteria for identifying attractor properties. Notably, under conditions in which there is continuous input, the output should be “attracted” to a stored pattern. However, contrary to previous expectations, as a pattern is gradually “morphed” from one stored pattern to another, a sharp transition between output patterns is not expected. The observed firing patterns of CA3 meet these criteria and can be quantitatively accounted for by our model. Notably, as morphing proceeds, the activity pattern in the dentate gyrus changes; in contrast, the activity pattern in the downstream CA3 network is attracted to a stored pattern and thus undergoes little change. We furthermore show that other aspects of the observed firing patterns can be explained by learning that occurs during behavioral testing. The CA3 thus displays both the learning and recall signatures of an attractor network. These observations, taken together with existing anatomical and behavioral evidence, make the strong case that CA3 constructs associative memories based on attractor dynamics.     

Balanced dopamine is critical for pattern completion during associative memory recall

Pattern completion, the ability to retrieve complete memories initiated by partial cues, is a critical feature of the memory process. However, little is known regarding the molecular and cellular mechanisms underlying this process. To study the role of dopamine in memory recall, we have analyzed dopamine transporter heterozygous knockout mice (DAT(+/-)), and found that while these mice possess normal learning, consolidation, and memory recall under full cue conditions, they exhibit specific deficits in pattern completion under partial cue condition. This form of memory recall deficit in the dopamine transporter heterozygous knockout mice can be reversed by a low dose of the dopamine antagonist haloperidol, further confirming that the inability to retrieve memory patterns is a result of dopamine imbalance. Therefore, our results reveal that a delicate control of the brain's dopamine level is critical for pattern completion during associative memory recall.     

Pattern recognition in neural networks with competing dynamics: coexistence of fixed-point and cyclic attractors

We study the properties of the dynamical phase transition occurring in neural network models in which a competition between associative memory and sequential pattern recognition exists. This competition occurs through a weighted mixture of the symmetric and asymmetric parts of the synaptic matrix. Through a generating functional formalism, we determine the structure of the parameter space at non-zero temperature and near saturation (i.e., when the number of stored patterns scales with the size of the network), identifying the regions of high and weak pattern correlations, the spin-glass solutions, and the order-disorder transition between these regions. This analysis reveals that, when associative memory is dominant, smooth transitions appear between high correlated regions and spurious states. In contrast when sequential pattern recognition is stronger than associative memory, the transitions are always discontinuous. Additionally, when the symmetric and asymmetric parts of the synaptic matrix are defined in terms of the same set of patterns, there is a discontinuous transition between associative memory and sequential pattern recognition. In contrast, when the symmetric and asymmetric parts of the synaptic matrix are defined in terms of independent sets of patterns, the network is able to perform both associative memory and sequential pattern recognition for a wide range of parameter values.     

Automated,High Accuracy Classification of Parkinsonian Disorders: A Pattern Recognition Approach

Progressive supranuclear palsy (PSP), multiple system atrophy (MSA) and idiopathic Parkinson’s disease (IPD) can be clinically indistinguishable, especially in the early stages, despite distinct patterns of molecular pathology. Structural neuroimaging holds promise for providing objective biomarkers for discriminating these diseases at the single subject level but all studies to date have reported incomplete separation of disease groups. In this study, we employed multi-class pattern recognition to assess the value of anatomical patterns derived from a widely available structural neuroimaging sequence for automated classification of these disorders. To achieve this, 17 patients with PSP, 14 with IPD and 19 with MSA were scanned using structural MRI along with 19 healthy controls (HCs). An advanced probabilistic pattern recognition approach was employed to evaluate the diagnostic value of several pre-defined anatomical patterns for discriminating the disorders, including: (i) a subcortical motor network; (ii) each of its component regions and (iii) the whole brain. All disease groups could be discriminated simultaneously with high accuracy using the subcortical motor network. The region providing the most accurate predictions overall was the midbrain/brainstem, which discriminated all disease groups from one another and from HCs. The subcortical network also produced more accurate predictions than the whole brain and all of its constituent regions. PSP was accurately predicted from the midbrain/brainstem, cerebellum and all basal ganglia compartments; MSA from the midbrain/brainstem and cerebellum and IPD from the midbrain/brainstem only. This study demonstrates that automated analysis of structural MRI can accurately predict diagnosis in individual patients with Parkinsonian disorders, and identifies distinct patterns of regional atrophy particularly useful for this process.     

The significance of wing pattern diversity in the Lycaenidae: mate discrimination by two recently diverged species

Closely related species of lycaenid butterflies are determinable, in part, by subtle differences in wing pattern. We found that female wing patterns can act as an effective mate‐recognition signal in some populations of two recently diverged species. In field experiments, we observed that males from a Lycaeides idas population and an alpine population of L. melissa preferentially initiate courtship with conspecific females. A morphometric study indicated that at least two wing pattern elements were important for distinguishing the two species: hindwing spots and orange crescent‐shaped pattern elements called aurorae. We deceived male L. idas into initiating courtship with computer generated paper models of heterospecific females when these pattern elements were manipulated, indicating that the wing pattern elements that define the diversity of this group can be effective mate recognition signals.     

Pattern storage in a sparsely coded neural network with cyclic activation

We investigate an artificial neural network model with a modified Hebb rule. It is an auto-associative neural network similar to the Hopfield model and to the Willshaw model. It has properties of both of these models. Another property is that the patterns are sparsely coded and are stored in cycles of synchronous neural activities. The cycles of activity for some ranges of parameter increase the capacity of the model. We discuss basic properties of the model and some of the implementation issues, namely optimizing of the algorithms. We describe the modification of the Hebb learning rule, the learning algorithm, the generation of patterns, decomposition of patterns into cycles and pattern recall.     

Detection of protein coding sequences using a mixture model for local protein amino acid sequence.

Locating protein coding regions in genomic DNA is a critical step in accessing the information generated by large scale sequencing projects. Current methods for gene detection depend on statistical measures of content differences between coding and noncoding DNA in addition to the recognition of promoters, splice sites, and other regulatory sites. Here we explore the potential value of recurrent amino acid sequence patterns 3-19 amino acids in length as a content statistic for use in gene finding approaches. A finite mixture model incorporating these patterns can partially discriminate protein sequences which have no (detectable) known homologs from randomized versions of these sequences, and from short (< or = 50 amino acids) non-coding segments extracted from the S. cerevisiea genome. The mixture model derived scores for a collection of human exons were not correlated with the GENSCAN scores, suggesting that the addition of our protein pattern recognition module to current gene recognition programs may improve their performance.     

A model of neural network for spatiotemporal pattern recognition

A model of neural network to recognize spatiotemporal patterns is presented. The network consists of two kinds of neural cells: P-cells and B-cells. A P-cell generates an impulse responding to more than one impulse and embodies two special functions: short term storage (STS) and heterosynaptic facilitation (HSF). A B-cell generates several impulses with high frequency as soon as it receives an impulse. In recognizing process, an impulse generated by a P-cell represents a recognition of stimulus pattern, and triggers the generation of impulses of a B-cell. Inhibitory impulses with high frequency generated by a B-cell reset the activities of all P-cells in the network.Two examples of spatiotemporal pattern recognition are presented. They are achieved by giving different values to the parameters of the network. In one example, the network recognizes both directional and non-directional patterns. The selectivities to directional and non-directional patterns are realized by only adjusting excitatory synaptic weights of P-cells. In the other example, the network recognizes time series of spatial patterns, where the lengths of the series are not necessarily the same and the transitional speeds of spatial patterns are not always the same. In both examples, the HSF signal controls the total activity of the network, which contributes to exact recognition and error recovery. In the latter example, it plays a role to trigger and execute the recognizing process. Finally, we discuss the correspondence between the model and physiological findings.