On problem solving with Hopfield neural networks

Hopfield and Tank have shown that neural networks can be used to solve certain computationally hard problems, in particular they studied the Traveling Salesman Problem (TSP). Based on network simulation results they conclude that analog VLSI neural nets can be promising in solving these problems. Recently, Wilson and Pawley presented the results of their simulations which contradict the original results and cast doubts on the usefulness of neural nets. In this paper we give the results of our simulations that clarify some of the discrepancies. We also investigate the scaling of TSP solutions found by neural nets as the size of the problem increases. Further, we consider the neural net solution of the Clustering Problem, also a computationally hard problem, and discuss the types of problems that appear to be well suited for a neural net approach.     

Sensory segmentation with coupled neural oscillators

We present a model of sensory segmentation that is based on the generation and processing of temporal tags in the form of oscillations, as suggested by the Dynamic Link Architecture. The model forms the basis for a natural solution to the sensory segmentation problem. It can deal with multiple segments, can integrate different cues and has the potential for processing hierarchical structures. Temporally tagged segments can easily be utilized in neural systems and form a natural basis for object recognition and learning. The model consists of a cortical circuit, an array of units that act as local feature detectors. Units are formulated as neural oscillators. Knowledge relevant to segmentation is encoded by connections. In accord with simple Gestalt laws, our concrete model has intracolumnar connections, between all units with overlapping receptive fields, and intercolumnar connections, between units responding to the same quality in different positions. An inhibitory connection system prevents total correlation and controls the grain of the segmentation. In simulations with synthetic input data we show the performance of the circuit, which produces signal correlation within segments and anticorrelation between segments.     

Learning the selectivity of V2 and V4 neurons using non-linear multi-layer wavelet networks

We investigate a non-linear network with two processing stages optimized to reduce the statistical dependencies in natural images. This network serves as a model for the neural information processing in the higher visual areas of primates (visual cortices V2-V4). The resulting population is analyzed with regard to non-linear selectivity and invariance properties. We find units that are very selective with respect to the space spanned by all possible input signals and units that are invariant with respect to certain stimulus classes. In comparison to the measured distribution of selectivity in V2 neurons, the selectivity histogram of the network units shows an even more pronounced tendency towards higher selectivities. A special property of the system is the emergence of non-linear interactions between coefficients from different scales and orientations, which are necessary for the exploitation of higher-order statistical redundancies of natural images. We extend the concept to multi-layer systems and present some simulation results.     

A canonical form for neural nets without circles

By “neural net” will be meant “neural net without circles.” Every neural net effects a transformation from inputs (i.e., firing patterns of the input neurons) to outputs (firing patterns of the output neurons). Two neural nets will be calledequivalent if they effect the same transformation from inputs to outputs. A canonical form is found for neural nets with respect to equivalence; i.e., a class of neural nets is defined, no two of which are equivalent, and which contains a neural net equivalent to any given neural net. This research was supported by the U.S. Air Force under Contract AF 49(638)-414 monitored by the Air Force Office of Scientific Research.     

Neural networks for perceptual processing: from simulation tools to theories

Neural networks are modelling tools that are, in principle, able to capture the input-output behaviour of arbitrary systems that may include the dynamics of animal populations or brain circuits. While a neural network model is useful if it captures phenomenologically the behaviour of the target system in this way, its utility is amplified if key mechanisms of the model can be discovered, and identified with those of the underlying system. In this review, we first describe, at a fairly high level with minimal mathematics, some of the tools used in constructing neural network models. We then go on to discuss the implications of network models for our understanding of the system they are supposed to describe, paying special attention to those models that deal with neural circuits and brain systems. We propose that neural nets are useful for brain modelling if they are viewed in a wider computational framework originally devised by Marr. Here, neural networks are viewed as an intermediate mechanistic abstraction between 'algorithm' and 'implementation', which can provide insights into biological neural representations and their putative supporting architectures.     

Dynamics of excitable neural networks with heterogeneous connectivity

A central issue of neuroscience is to understand how neural units integrates internal and external signals to create coherent states. Recently, it has been shown that the sensitivity and dynamic range of neural assemblies are optimal at a critical coupling among its elements. Complex architectures of connections seem to play a constructive role on the reliable coordination of neural units. Here we show that, the synchronizability and sensitivity of excitable neural networks can be tuned by diversity in the connections strengths. We illustrate our findings for weighted networks with regular, random and complex topologies. Additional comparisons of real brain networks support previous studies suggesting that heterogeneity in the connectivity may play a constructive role on information processing. These findings provide insights into the relationship between structure and function of neural circuits.     

Probabilistic computation by neuromine networks

In this paper, we address the question, can biologically feasible neural nets compute more than can be computed by deterministic polynomial time algorithms? Since we want to maintain a claim of plausibility and reasonableness we restrict ourselves to algorithmically easy to construct nets and we rule out infinite precision in parameters and in any analog parts of the computation. Our approach is to consider the recent advances in randomized algorithms and see if such randomized computations can be described by neural nets. We start with a pair of neurons and show that by connecting them with reciprocal inhibition and some tonic input, then the steady-state will be one neuron ON and one neuron OFF, but which neuron will be ON and which neuron will be OFF will be chosen at random (perhaps, it would be better to say that microscopic noise in the analog computation will be turned into a megascale random bit). We then show that we can build a small network that uses this random bit process to generate repeatedly random bits. This random bit generator can then be connected with a neural net representing the deterministic part of randomized algorithm. We, therefore, demonstrate that these neural nets can carry out probabilistic computation and thus be less limited than classical neural nets.     

Long-range, pattern-dependent contextual effects in early human visual cortex

The standard view of neurons in early visual cortex is that they behave like localized feature detectors. Here we demonstrate that processing in early visual areas goes beyond feature detection by showing that neural responses are greater when a feature deviates from its context compared to when it does not deviate from its context. Using psychophysics, fMRI, and electroencephalography methodologies, we measured neural responses to an oriented Gabor ("target") embedded in various visual patterns as defined by the relative orientation of flanking stimuli. We first show using psychophysical contrast adaptation and fMRI that a target that differs from its context results in more neural activity compared to a target that is contained within an alternating sequence, suggesting that neurons in early visual cortex are sensitive to large-scale orientation patterns. Next, we use event-related potentials to show that orientation deviations affect the earliest sensory components of the target response. Finally, we use forced-choice classification of "noise" stimuli to show that we are more likely to "see" orientations that deviate from the context. Our results suggest that early visual cortex is sensitive to global patterns in images in a way that is markedly different from the predictions of standard models of cortical visual processing.     

A neural net model for multiple memory domains

Previous studies with neural nets constructed of discrete populations of formal neurons have assumed that all neurons have the same probability of connection with any other neuron in the net. However, in this new study we incorporate the behavior of the neural systems in which the neural connections can be set up by means of chemical markers carried by the individual cells. With this new approach we studied the dynamics of isolated neural nets again as well as the dynamics of neural nets with sustained inputs. Results obtained with this approach show simple and multiple hysteresis phenomena. Such hysteresis loops may be considered to represent the basis for short-term memory.     

A Functional Hypothesis for LGN-V1-TRN Connectivities Suggested by Computer Simulation

We employ computer simulation to investigate the function of neural circuitries between thalamic sensory relay nuclei, primary sensory cortices, and the thalamic reticular nucleus (TRN). Computational similarities exist between these circuits and the architecture of a simple artificial neural network. We impose processing parameters on this network architecture in keeping with anatomical and physiological details of the mammalian geniculo-cortical visual pathway, and then run the simulation on a task involving multiple simultaneous inputs from the simulated visual field. After two to three loops through the simulation, activity in cortical and thalamic units whose receptive fields include the stronger stimulus remains constant, while activity in other cortical and thalamic units activated by weaker stimuli declines toward resting values. These results suggest that the modeled neural circuitry functions to prime selective attentional mechanisms further up the visual streams toward specific portions of the total visual stimulus. Besides extending existing models and evidence about the function of these neural circuits, our results also provide physiologists with predicted activity profiles of thalamic and cortical elements of the modeled neural system for a task not yet studied experimentally.     

Implicit attentional selection of bound visual features

Traditionally, research on visual attention has been focused on the processes involved in conscious, explicit selection of task-relevant sensory input. Recently, however, it has been shown that attending to a specific feature of an object automatically increases neural sensitivity to this feature throughout the visual field. Here we show that directing attention to a specific color of an object results in attentional modulation of the processing of task-irrelevant and not consciously perceived motion signals that are spatiotemporally associated with this color throughout the visual field. Such implicit cross-feature spreading of attention takes place according to the veridical physical associations between the color and motion signals, even under special circumstances when they are perceptually misbound. These results imply that the units of implicit attentional selection are spatiotemporally colocalized feature clusters that are automatically bound throughout the visual field.     

Participation of African social scientists in malaria control: identifying enabling and constraining factors

Background

Insecticide-treated nets represent currently a key malaria control strategy, but low insecticide re-treatment rates remain problematic. Olyset? nets are currently one of two long-lasting insecticidal nets recommended by WHO. An assessment was carried out of the effect of Olyset? nets after seven years of use in rural Tanzania.

Methods

A survey of Olyset? nets was conducted in two Tanzanian villages to examine their insecticide dosage, bioassay efficacy and desirability compared with ordinary polyester nets.

Results

Of 103 randomly selected nets distributed in 1994 to 1995, 100 could be traced. Most nets were in a condition likely to offer protection against mosquito biting. Villagers appreciated mainly the durability of Olyset? nets and insecticide persistence. People disliked the small size of these nets and the light blue colour and preferred a smaller mesh size, features that can easily be modified. At equal price, 51% said they would prefer to buy an Olyset? net and 49% opted for an ordinary polyester net. The average permethrin content was 33%-41% of the initial insecticide dose of 20,000 mg/Kg. Bioassay results indicated high knock-down rates at 60 minutes, but the mosquito mortality after 24 hours was rather low (mean: 34%). No significant correlation was found between bioassay results and insecticide concentration in and on the net.

Conclusions

Olyset? nets are popular, durable and with a much longer insecticide persistence than ordinary polyester nets. Hence, Olyset? nets are one of the best choices for ITN programmes in rural malaria-endemic areas.     

Evaluation of a shape-based model of human face discrimination using FMRI and behavioral techniques

Understanding the neural mechanisms underlying object recognition is one of the fundamental challenges of visual neuroscience. While neurophysiology experiments have provided evidence for a "simple-to-complex" processing model based on a hierarchy of increasingly complex image features, behavioral and fMRI studies of face processing have been interpreted as incompatible with this account. We present a neurophysiologically plausible, feature-based model that quantitatively accounts for face discrimination characteristics, including face inversion and "configural" effects. The model predicts that face discrimination is based on a sparse representation of units selective for face shapes, without the need to postulate additional, "face-specific" mechanisms. We derive and test predictions that quantitatively link model FFA face neuron tuning, neural adaptation measured in an fMRI rapid adaptation paradigm, and face discrimination performance. The experimental data are in excellent agreement with the model prediction that discrimination performance should asymptote as faces become dissimilar enough to activate different neuronal populations.     

An olfactory neuronal network for vapor recognition in an artificial nose

Odorant sensitivity and discrimination in the olfactory system appear to involve extensive neural processing of the primary sensory inputs from the olfactory epithelium. To test formally the functional consequences of such processing, we implemented in an artificial chemosensing system a new analytical approach that is based directly on neural circuits of the vertebrate olfactory system. An array of fiber-optic chemosensors, constructed with response properties similar to those of olfactory sensory neurons, provide time-varying inputs to a computer simulation of the olfactory bulb (OB). The OB simulation produces spatiotemporal patterns of neuronal firing that vary with vapor type. These patterns are then recognized by a delay line neural network (DLNN). In the final output of these two processing steps, vapor identity is encoded by the spatial patterning of activity across units in the DLNN, and vapor intensity is encoded by response latency. The OB-DLNN combination thus separates identity and intensity information into two distinct codes carried by the same output units, enabling discrimination among organic vapors over a range of input signal intensities. In addition to providing a well-defined system for investigating olfactory information processing, this biologically based neuronal network performs better than standard feed-forward neural networks in discriminating vapors when small amounts of training data are used. Received: 30 June 1997 / Accepted in revised form: 12 January 1998     

Effects of unexpected chords and of performer's expression on brain responses and electrodermal activity

Background

There is lack of neuroscientific studies investigating music processing with naturalistic stimuli, and brain responses to real music are, thus, largely unknown.

Methodology/Principal Findings

This study investigates event-related brain potentials (ERPs), skin conductance responses (SCRs) and heart rate (HR) elicited by unexpected chords of piano sonatas as they were originally arranged by composers, and as they were played by professional pianists. From the musical excerpts played by the pianists (with emotional expression), we also created versions without variations in tempo and loudness (without musical expression) to investigate effects of musical expression on ERPs and SCRs. Compared to expected chords, unexpected chords elicited an early right anterior negativity (ERAN, reflecting music-syntactic processing) and an N5 (reflecting processing of meaning information) in the ERPs, as well as clear changes in the SCRs (reflecting that unexpected chords also elicited emotional responses). The ERAN was not influenced by emotional expression, whereas N5 potentials elicited by chords in general (regardless of their chord function) differed between the expressive and the non-expressive condition.

Conclusions/Significance

These results show that the neural mechanisms of music-syntactic processing operate independently of the emotional qualities of a stimulus, justifying the use of stimuli without emotional expression to investigate the cognitive processing of musical structure. Moreover, the data indicate that musical expression affects the neural mechanisms underlying the processing of musical meaning. Our data are the first to reveal influences of musical performance on ERPs and SCRs, and to show physiological responses to unexpected chords in naturalistic music.     

An associative memory that can form hypotheses: a phase-coded neural network

 Nonlinear associative memories as realized, e.g., by Hopfield nets are characterized by attractor-type dynamics. When fed with a starting pattern, they converge to exactly one of the stored patterns which is supposed to be most similar. These systems cannot render hypotheses of classification, i.e., render several possible answers to a given classification problem. Inspired by von der Malsburg’s correlation theory of brain function, we extend conventional neural network architectures by introducing additional dynamical variables. Assuming an oscillatory time structure of neural firing, i.e., the existence of neural clocks, we assign a so-called phase to each formal neuron. The phases explicitly describe detailed correlations of neural activities neglected in conventional neural network architectures. Implementing this extension into a simple self-organizing network based on a feature map, we present an associative memory that actually is capable of forming hypotheses of classification. Received: 6 December 1993/Accepted in revised form: 14 July 1994     

Delayed-exponential approximation of a linear homogeneous diffusion model of neuron

The diffusion models of neuronal activity are general yet conceptually simple and flexible enough to be useful in a variety of modeling problems. Unfortunately, even simple diffusion models lead to tedious numerical calculations. Consequently, the existing neural net models use characteristics of a single neuron taken from the pre-diffusion era of neural modeling. Simplistic elements of neural nets forbid to incorporate a single learning neuron structure into the net model. The above drawback cannot be overcome without the use of the adequate structure of the single neuron as an element of a net. A linear (not necessarily homogeneous) diffusion model of a single neuron is a good candidate for such a structure, it must, however, be simplified. In the paper the structure of the diffusion model of neuron is discussed and a linear homogeneous model with reflection is analyzed. For this model an approximation is presented, which is based on the approximation of the first passage time distribution of the Ornstein-Uhlenbeck process by the delayed (shifted) exponential distribution. The resulting model has a simple structure and has a prospective application in neural modeling and in analysis of neural nets.Work supported by Polish Academy of Sciences grant # CPBP 04.01     

Responses of recurrent nets of asymmetric ON and OFF cells

A neural field model of ON and OFF cells with all-to-all inhibitory feedback is investigated. External spatiotemporal stimuli drive the ON and OFF cells with, respectively, direct and inverted polarity. The dynamic differences between networks built of ON and OFF cells (“ON/OFF”) and those having only ON cells (“ON/ON”) are described for the general case where ON and OFF cells can have different spontaneous firing rates; this asymmetric case is generic. Neural responses to nonhomogeneous static and time-periodic inputs are analyzed in regimes close to and away from self-oscillation. Static stimuli can cause oscillatory behavior for certain asymmetry levels. Time-periodic stimuli expose dynamical differences between ON/OFF and ON/ON nets. Outside the stimulated region, we show that ON/OFF nets exhibit frequency doubling, while ON/ON nets cannot. On the other hand, ON/ON networks show antiphase responses between stimulated and unstimulated regions, an effect that does not rely on specific receptive field circuitry. An analysis of the resonance properties of both net types reveals that ON/OFF nets exhibit larger response amplitude. Numerical simulations of the neural field models agree with theoretical predictions for localized static and time-periodic forcing. This is also the case for simulations of a network of noisy integrate-and-fire neurons. We finally discuss the application of the model to the electrosensory system and to frequency-doubling effects in retina.     

Efficient sparse coding in early sensory processing: lessons from signal recovery

Sensory representations are not only sparse, but often overcomplete: coding units significantly outnumber the input units. For models of neural coding this overcompleteness poses a computational challenge for shaping the signal processing channels as well as for using the large and sparse representations in an efficient way. We argue that higher level overcompleteness becomes computationally tractable by imposing sparsity on synaptic activity and we also show that such structural sparsity can be facilitated by statistics based decomposition of the stimuli into typical and atypical parts prior to sparse coding. Typical parts represent large-scale correlations, thus they can be significantly compressed. Atypical parts, on the other hand, represent local features and are the subjects of actual sparse coding. When applied on natural images, our decomposition based sparse coding model can efficiently form overcomplete codes and both center-surround and oriented filters are obtained similar to those observed in the retina and the primary visual cortex, respectively. Therefore we hypothesize that the proposed computational architecture can be seen as a coherent functional model of the first stages of sensory coding in early vision.