Coding of natural scenes in primary visual cortex

Natural scene coding in ferret visual cortex was investigated using a new technique for multi-site recording of neuronal activity from the cortical surface. Surface recordings accurately reflected radially aligned layer 2/3 activity. At individual sites, evoked activity to natural scenes was weakly correlated with the local image contrast structure falling within the cells' classical receptive field. However, a population code, derived from activity integrated across cortical sites having retinotopically overlapping receptive fields, correlated strongly with the local image contrast structure. Cell responses demonstrated high lifetime sparseness, population sparseness, and high dispersal values, implying efficient neural coding in terms of information processing. These results indicate that while cells at an individual cortical site do not provide a reliable estimate of the local contrast structure in natural scenes, cell activity integrated across distributed cortical sites is closely related to this structure in the form of a sparse and dispersed code.     

Distributed neural plasticity for shape learning in the human visual cortex

Expertise in recognizing objects in cluttered scenes is a critical skill for our interactions in complex environments and is thought to develop with learning. However, the neural implementation of object learning across stages of visual analysis in the human brain remains largely unknown. Using combined psychophysics and functional magnetic resonance imaging (fMRI), we show a link between shape-specific learning in cluttered scenes and distributed neuronal plasticity in the human visual cortex. We report stronger fMRI responses for trained than untrained shapes across early and higher visual areas when observers learned to detect low-salience shapes in noisy backgrounds. However, training with high-salience pop-out targets resulted in lower fMRI responses for trained than untrained shapes in higher occipitotemporal areas. These findings suggest that learning of camouflaged shapes is mediated by increasing neural sensitivity across visual areas to bolster target segmentation and feature integration. In contrast, learning of prominent pop-out shapes is mediated by associations at higher occipitotemporal areas that support sparser coding of the critical features for target recognition. We propose that the human brain learns novel objects in complex scenes by reorganizing shape processing across visual areas, while taking advantage of natural image correlations that determine the distinctiveness of target shapes.     

Timing Precision in Population Coding of Natural Scenes in the Early Visual System

The timing of spiking activity across neurons is a fundamental aspect of the neural population code. Individual neurons in the retina, thalamus, and cortex can have very precise and repeatable responses but exhibit degraded temporal precision in response to suboptimal stimuli. To investigate the functional implications for neural populations in natural conditions, we recorded in vivo the simultaneous responses, to movies of natural scenes, of multiple thalamic neurons likely converging to a common neuronal target in primary visual cortex. We show that the response of individual neurons is less precise at lower contrast, but that spike timing precision across neurons is relatively insensitive to global changes in visual contrast. Overall, spike timing precision within and across cells is on the order of 10 ms. Since closely timed spikes are more efficient in inducing a spike in downstream cortical neurons, and since fine temporal precision is necessary to represent the more slowly varying natural environment, we argue that preserving relative spike timing at a ~10-ms resolution is a crucial property of the neural code entering cortex.     

Fractal-like image statistics in visual art: similarity to natural scenes

Both natural scenes and visual art are often perceived as esthetically pleasing. It is therefore conceivable that the two types of visual stimuli share statistical properties. For example, natural scenes display a Fourier power spectrum that tends to fall with spatial frequency according to a power-law. This result indicates that natural scenes have fractal-like, scale-invariant properties. In the present study, we asked whether visual art displays similar statistical properties by measuring their Fourier power spectra. Our analysis was restricted to graphic art from the Western hemisphere. For comparison, we also analyzed images, which generally display relatively low or no esthetic quality (household and laboratory objects, parts of plants, and scientific illustrations). Graphic art, but not the other image categories, resembles natural scenes in showing fractal-like, scale-invariant statistics. This property is universal in our sample of graphic art; it is independent of cultural variables, such as century and country of origin, techniques used or subject matter. We speculate that both graphic art and natural scenes share statistical properties because visual art is adapted to the structure of the visual system which, in turn, is adapted to process optimally the image statistics of natural scenes.     

Responses of neurons in primary and inferior temporal visual cortices to natural scenes.

The primary visual cortex (V1) is the first cortical area to receive visual input, and inferior temporal (IT) areas are among the last along the ventral visual pathway. We recorded, in area V1 of anaesthetized cats and area IT of awake macaque monkeys, responses of neurons to videos of natural scenes. Responses were analysed to test various hypotheses concerning the nature of neural coding in these two regions. A variety of spike-train statistics were measured including spike-count distributions, interspike interval distributions, coefficients of variation, power spectra, Fano factors and different sparseness measures. All statistics showed non-Poisson characteristics and several revealed self-similarity of the spike trains. Spike-count distributions were approximately exponential in both visual areas for eight different videos and for counting windows ranging from 50 ms to 5 seconds. The results suggest that the neurons maximize their information carrying capacity while maintaining a fixed long-term-average firing rate, or equivalently, minimize their average firing rate for a fixed information carrying capacity.     

A theory for the use of visual orientation information which exploits the columnar structure of striate cortex

A neural model is constructed based on the structure of a visual orientation hypercolumn in mammalian striate cortex. It is then assumed that the perceived orientation of visual contours is determined by the pattern of neuronal activity across orientation columns. Using statistical estimation theory, limits on the precision of orientation estimation and discrimination are calculated. These limits are functions of single unit response properties such as orientation tuning width, response amplitude and response variability, as well as the degree of organization in the neural network. It is shown that a network of modest size, consisting of broadly orientation selective units, can reliably discriminate orientation with a precision equivalent to human performance. Of the various network parameters, the discrimination threshold depends most critically on the number of cells in the hypercolumn. The form of the dependence on cell number correctly predicts the results of psychophysical studies of orientation discrimination. The model system's performance is also consistent with psychophysical data in two situations in which human performance is not optimal. First, interference with orientation discrimination occurs when multiple stimuli activate cells in the same hypercolumn. Second, systematic errors in the estimation of orientation can occur when a stimulus is composed of intersecting lines. The results demonstrate that it is possible to relate neural activity to visual performance by an examination of the pattern of activity across orientation columns. This provides support for the hypothesis that perceived orientation is determined by the distributed pattern of neural activity. The results also encourage the view of neural activity. The results also are determined by the responses of many neurons rather than the sensitivity of individual cells.     

A model for feature linking via collective oscillations in the primary visual cortex

A neural network model for explaining experimentally observed neuronal responses in cat primary visual cortex is proposed. In our model, the basic functional unit is an orientation column which is represented by a large homogeneous population of neurons modeled as integrate-and-fire type excitable elements. The orientation column exhibits spontaneous collective oscillations in activity in response to suitable visual stimuli. Such oscillations are caused by mutual synchronization among the neurons within the column. Numerical simulation for various stimulus patterns shows that as a result of activity correlations between different columns, the amplitude and the phase of the oscillation in each column depend strongly on the global feature of the stimulus pattern. These results satisfactorily account for experimental observations.     

Emergence of visual saliency from natural scenes via context-mediated probability distributions coding

Visual saliency is the perceptual quality that makes some items in visual scenes stand out from their immediate contexts. Visual saliency plays important roles in natural vision in that saliency can direct eye movements, deploy attention, and facilitate tasks like object detection and scene understanding. A central unsolved issue is: What features should be encoded in the early visual cortex for detecting salient features in natural scenes? To explore this important issue, we propose a hypothesis that visual saliency is based on efficient encoding of the probability distributions (PDs) of visual variables in specific contexts in natural scenes, referred to as context-mediated PDs in natural scenes. In this concept, computational units in the model of the early visual system do not act as feature detectors but rather as estimators of the context-mediated PDs of a full range of visual variables in natural scenes, which directly give rise to a measure of visual saliency of any input stimulus. To test this hypothesis, we developed a model of the context-mediated PDs in natural scenes using a modified algorithm for independent component analysis (ICA) and derived a measure of visual saliency based on these PDs estimated from a set of natural scenes. We demonstrated that visual saliency based on the context-mediated PDs in natural scenes effectively predicts human gaze in free-viewing of both static and dynamic natural scenes. This study suggests that the computation based on the context-mediated PDs of visual variables in natural scenes may underlie the neural mechanism in the early visual cortex for detecting salient features in natural scenes.     

The human visual system is optimised for processing the spatial information in natural visual images

A fundamental tenet of visual science is that the detailed properties of visual systems are not capricious accidents, but are closely matched by evolution and neonatal experience to the environments and lifestyles in which those visual systems must work. This has been shown most convincingly for fish and insects. For mammalian vision, however, this tenet is based more upon theoretical arguments than upon direct observations. Here, we describe experiments that require human observers to discriminate between pictures of slightly different faces or objects. These are produced by a morphing technique that allows small, quantifiable changes to be made in the stimulus images. The independent variable is designed to give increasing deviation from natural visual scenes, and is a measure of the Fourier composition of the image (its second-order statistics). Performance in these tests was best when the pictures had natural second-order spatial statistics, and degraded when the images were made less natural. Furthermore, performance can be explained with a simple model of contrast coding, based upon the properties of simple cells in the mammalian visual cortex. The findings thus provide direct empirical support for the notion that human spatial vision is optimised to the second-order statistics of the optical environment.     

Temporal encoding of spatial information during active visual fixation

Humans and other species continually perform microscopic eye movements, even when attending to a single point. These movements, which include drifts and microsaccades, are under oculomotor control, elicit strong neural responses, and have been thought to serve important functions. The influence of these fixational eye movements on the acquisition and neural processing of visual information remains unclear. Here, we show that during viewing of natural scenes, microscopic eye movements carry out a crucial information-processing step: they remove predictable correlations in natural scenes by equalizing the spatial power of the retinal image within the frequency range of ganglion cells' peak sensitivity. This transformation, which had been attributed to center-surround receptive field organization, occurs prior to any neural processing and reveals a form of matching between the statistics of natural images and those of normal eye movements. We further show that the combined effect of microscopic eye movements and retinal receptive field organization is to convert spatial luminance discontinuities into synchronous firing events, beginning the process of edge detection. Thus, microscopic eye movements are fundamental to two goals of early visual processing: redundancy reduction and feature extraction.     

A neural network model on self-organizing emergence of simple-cell receptive field with orientation selectivity in visual cortex

In order to probe into the self-organizing emergence of simple cell orientation selectivity, we tried to construct a neural network model that consists of LGN neurons and simple cells in visual cortex and obeys the Hebbian learning rule. We investigated the neural coding and representation of simple cells to a natural image by means of this model. The results show that the structures of their receptive fields are determined by the preferred orientation selectivity of simple cells. However, they are also decided by the emergence of self-organization in the unsupervision learning process. This kind of orientation selectivity results from dynamic self-organization based on the interactions between LGN and cortex.     

A neural network model on self-organizing emergence of simple-cell receptive field with orientation selectivity in visual cortex

In order to probe into the self-organizing emergence of simple cell orientation selectivity, we tried to construct a neural network model that consists of LGN neurons and simple cells in visual cortex and obeys the Hebbian learning rule. We investigated the neural coding and representation of simple cells to a natural image by means of this model. The results show that the structures of their receptive fields are determined by the preferred orientation selectivity of simple cells. However, they are also decided by the emergence of self-organization in the unsupervision learning process. This kind of orientation selectivity results from dynamic self-organization based on the interactions between LGN and cortex.     

Neural correlates of contrast detection at threshold

Human psychophysical studies have demonstrated that, for stimuli near the threshold of visibility, detection of motion in one direction is unaffected by the superimposition of motion in the opposite direction. To investigate the neural basis for this perceptual phenomenon, we recorded from directionally selective neurons in macaque visual area MT (middle temporal visual area). Contrast thresholds obtained for single gratings moving in a neuron's preferred direction were compared with those obtained for motion presented simultaneously in the neuron's preferred and antipreferred directions. A simple model based on probability summation between neurons tuned to opposite directions could sufficiently account for contrast thresholds revealed psychophysically, suggesting that area MT is likely to provide the neural basis for contrast detection of stimuli modulated in time.     

One picture is worth at least a million neurons

How many neurons participate in the representation of a single visual image? Answering this question is critical for constraining biologically inspired models of object recognition, which vary greatly in their assumptions from few "grandmother cells" to numerous neurons in widely distributed networks. Functional imaging techniques, such as fMRI, provide an opportunity to explore this issue, since they allow the simultaneous detection of the entire neuronal population responding to each stimulus. Several studies have shown that fMRI BOLD signal is approximately proportional to neuronal activity. However, since it provides an indirect measure of this activity, obtaining a realistic estimate of the number of activated neurons requires several intervening steps. Here, we used the extensive knowledge of primate V1 to yield a conservative estimate of the ratio between hemodynamic response and neuronal firing. This ratio was then used, in addition to several cautious assumptions, to assess the number of neurons responding to a single-object image in the entire visual cortex and particularly in object-related areas. Our results show that at least a million neurons in object-related cortex and about two hundred million neurons in the entire visual cortex are involved in the representation of a single-object image.     

Functional implications of temporal structure in primate cortical information processing

Processing of information in the cerebral cortex of primates is characterized by distributed representations and processing in neuronal assemblies rather than by detector neurons, cardinal cells or command neurons. Responses of individual neurons in sensory cortical areas contain limited and ambiguous information on common features of the natural environment which is disambiguated by comparison with the responses of other, related neurons. Distributed representations are also capable to represent the enormous complexity and variability of the natural environment by the large number of possible combinations of neurons that can engage in the representation of a stimulus or other content. A critical problem of distributed representation and processing is the superposition of several assemblies activated at the same time since interpretation and processing of a population code requires that the responses related to a single representation can be identified and distinguished from other, related activity. A possible mechanism which tags related responses is the synchronization of neuronal responses of the same assembly with a precision in the millisecond range. This mechanism also supports the separate processing of distributed activity and dynamic assembly formation. Experimental evidence from electrophysiological investigations of non-human primates and human subjects shows that synchronous activity can be found in visual, auditory and motor areas of the cortex. Simultaneous recordings of neurons in the visual cortex indicate that individual neurons synchronize their activity with each other, if they respond to the same stimulus but not if they are part of different assemblies representing different contents. Furthermore, evidence for synchronous activity related to perception, expectation, memory, and attention has been observed.     

A universal model of esthetic perception based on the sensory coding of natural stimuli

Philosophers have pointed out that there is a close relation between the esthetics of art and the beauty of natural scenes. Supporting this similarity at the experimental level, we have recently shown that visual art and natural scenes share fractal-like, scale-invariant statistical properties. Moreover, evidence from neurophysiological experiments shows that the visual system uses an efficient (sparse) code to process optimally the statistical properties of natural stimuli. In the present work, a hypothetical model of esthetic perception is described that combines both lines of evidence. Specifically, it is proposed that an artist creates a work of art so that it induces a specific resonant state in the visual system. This resonant state is thought to be based on the adaptation of the visual system to natural scenes. The proposed model is universal and predicts that all human beings share the same general concept of esthetic judgment. The model implies that esthetic perception, like the coding of natural stimuli, depends on stimulus form rather than content, depends on higher-order statistics of the stimuli, and is non-intuitive to cognitive introspection. The model accommodates the central tenet of neuroesthetic theory that esthetic perception reflects fundamental functional properties of the nervous system.     

What we see is how we are: new paradigms in visual research

As most sensory modalities, the visual system needs to deal with very fast changes in the environment. Instead of processing all sensory stimuli, the brain is able to construct a perceptual experience by combining selected sensory input with an ongoing internal activity. Thus, the study of visual perception needs to be approached by examining not only the physical properties of stimuli, but also the brain's ongoing dynamical states onto which these perturbations are imposed. At least three different models account for this internal dynamics. One model is based on cardinal cells where the activity of few cells by itself constitutes the neuronal correlate of perception, while a second model is based on a population coding that states that the neuronal correlate of perception requires distributed activity throughout many areas of the brain. A third proposition, known as the temporal correlation hypothesis states that the distributed neuronal populations that correlate with perception, are also defined by synchronization of the activity on a millisecond time scale. This would serve to encode contextual information by defining relations between the features of visual objects. If temporal properties of neural activity are important to establish the neural mechanisms of perception, then the study of appropriate dynamical stimuli should be instrumental to determine how these systems operate. The use of natural stimuli and natural behaviors such as free viewing, which features fast changes of internal brain states as seen by motor markers, is proposed as a new experimental paradigm to study visual perception.     

Cortical sensitivity to visual features in natural scenes

A central hypothesis concerning sensory processing is that the neuronal circuits are specifically adapted to represent natural stimuli efficiently. Here we show a novel effect in cortical coding of natural images. Using spike-triggered average or spike-triggered covariance analyses, we first identified the visual features selectively represented by each cortical neuron from its responses to natural images. We then measured the neuronal sensitivity to these features when they were present in either natural images or random stimuli. We found that in the responses of complex cells, but not of simple cells, the sensitivity was markedly higher for natural images than for random stimuli. Such elevated sensitivity leads to increased detectability of the visual features and thus an improved cortical representation of natural scenes. Interestingly, this effect is due not to the spatial power spectra of natural images, but to their phase regularities. These results point to a distinct visual-coding strategy that is mediated by contextual modulation of cortical responses tuned to the spatial-phase structure of natural scenes.     

Orientation hypercolumns of the visual cortex: Ring model

A hypercolumn of the visual cortex is a functional unit formed of neighboring columns whose neurons respond to a stimulus of particular orientation. The function of the hypercolumn is to amplify the orientation tuning of visually evoked responses. According to the conventional simple model of a hypercolumn, neuronal populations with different orientation preferences are distributed on a ring. Every population is described by a firing rate (FR) model. To determine the limitations of the FR-ring model, it was compared with a more detailed ring model, which takes into account the distribution of neurons of each population according to their voltage values. In the case of leaky integrate-and-fire neurons, every neuronal population is described by the Fokker-Planck equation (FPE). The mapping of parameters was obtained. The simulations revealed differences in the behavior of the two models. The FPE-based model reacts faster to a change in stimulus orientation. The FPE ring model gives a steady-state solution in the form of waves of activity traveling on the ring, whereas the FR ring model presents amplitude instability for the same parameter set. The FPE ring model reproduces the characteristic effects of the FR ring model: virtual rotation and symmetry breaking.     

Distributed dendritic processing facilitates object detection: a computational analysis on the visual system of the fly

Background

Detecting objects is an important task when moving through a natural environment. Flies, for example, may land on salient objects or may avoid collisions with them. The neuronal ensemble of Figure Detection cells (FD-cells) in the visual system of the fly is likely to be involved in controlling these behaviours, as these cells are more sensitive to objects than to extended background structures. Until now the computations in the presynaptic neuronal network of FD-cells and, in particular, the functional significance of the experimentally established distributed dendritic processing of excitatory and inhibitory inputs is not understood.

Methodology/Principal Findings

We use model simulations to analyse the neuronal computations responsible for the preference of FD-cells for small objects. We employed a new modelling approach which allowed us to account for the spatial spread of electrical signals in the dendrites while avoiding detailed compartmental modelling. The models are based on available physiological and anatomical data. Three models were tested each implementing an inhibitory neural circuit, but differing by the spatial arrangement of the inhibitory interaction. Parameter optimisation with an evolutionary algorithm revealed that only distributed dendritic processing satisfies the constraints arising from electrophysiological experiments. In contrast to a direct dendro-dendritic inhibition of the FD-cell (Direct Distributed Inhibition model), an inhibition of its presynaptic retinotopic elements (Indirect Distributed Inhibition model) requires smaller changes in input resistance in the inhibited neurons during visual stimulation.

Conclusions/Significance

Distributed dendritic inhibition of retinotopic elements as implemented in our Indirect Distributed Inhibition model is the most plausible wiring scheme for the neuronal circuit of FD-cells. This microcircuit is computationally similar to lateral inhibition between the retinotopic elements. Hence, distributed inhibition might be an alternative explanation of perceptual phenomena currently explained by lateral inhibition networks.