Luminance,Colour, Viewpoint and Border Enhanced Disparity Energy Model

The visual cortex is able to extract disparity information through the use of binocular cells. This process is reflected by the Disparity Energy Model, which describes the role and functioning of simple and complex binocular neuron populations, and how they are able to extract disparity. This model uses explicit cell parameters to mathematically determine preferred cell disparities, like spatial frequencies, orientations, binocular phases and receptive field positions. However, the brain cannot access such explicit cell parameters; it must rely on cell responses. In this article, we implemented a trained binocular neuronal population, which encodes disparity information implicitly. This allows the population to learn how to decode disparities, in a similar way to how our visual system could have developed this ability during evolution. At the same time, responses of monocular simple and complex cells can also encode line and edge information, which is useful for refining disparities at object borders. The brain should then be able, starting from a low-level disparity draft, to integrate all information, including colour and viewpoint perspective, in order to propagate better estimates to higher cortical areas.     

Perceptual “Read-Out” of Conjoined Direction and Disparity Maps in Extrastriate Area MT

Cortical neurons are frequently tuned to several stimulus dimensions, and many cortical areas contain intercalated maps of multiple variables. Relatively little is known about how information is “read out” of these multidimensional maps. For example, how does an organism extract information relevant to the task at hand from neurons that are also tuned to other, irrelevant stimulus dimensions? We addressed this question by employing microstimulation techniques to examine the contribution of disparity-tuned neurons in the middle temporal (MT) visual area to performance on a direction discrimination task. Most MT neurons are tuned to both binocular disparity and the direction of stimulus motion, and MT contains topographic maps of both parameters. We assessed the effect of microstimulation on direction judgments after first characterizing the disparity tuning of each stimulation site. Although the disparity of the stimulus was irrelevant to the required task, we found that microstimulation effects were strongly modulated by the disparity tuning of the stimulated neurons. For two of three monkeys, microstimulation of nondisparity-selective sites produced large biases in direction judgments, whereas stimulation of disparity-selective sites had little or no effect. The binocular disparity was optimized for each stimulation site, and our result could not be explained by variations in direction tuning, response strength, or any other tuning property that we examined. When microstimulation of a disparity-tuned site did affect direction judgments, the effects tended to be stronger at the preferred disparity of a stimulation site than at the nonpreferred disparity, indicating that monkeys can selectively monitor direction columns that are best tuned to an appropriate conjunction of parameters. We conclude that the contribution of neurons to behavior can depend strongly upon tuning to stimulus dimensions that appear to be irrelevant to the current task, and we suggest that these findings are best explained in terms of the strategy used by animals to perform the task.     

Perceptual “Read-Out” of Conjoined Direction and Disparity Maps in Extrastriate Area MT

Cortical neurons are frequently tuned to several stimulus dimensions, and many cortical areas contain intercalated maps of multiple variables. Relatively little is known about how information is “read out” of these multidimensional maps. For example, how does an organism extract information relevant to the task at hand from neurons that are also tuned to other, irrelevant stimulus dimensions? We addressed this question by employing microstimulation techniques to examine the contribution of disparity-tuned neurons in the middle temporal (MT) visual area to performance on a direction discrimination task. Most MT neurons are tuned to both binocular disparity and the direction of stimulus motion, and MT contains topographic maps of both parameters. We assessed the effect of microstimulation on direction judgments after first characterizing the disparity tuning of each stimulation site. Although the disparity of the stimulus was irrelevant to the required task, we found that microstimulation effects were strongly modulated by the disparity tuning of the stimulated neurons. For two of three monkeys, microstimulation of nondisparity-selective sites produced large biases in direction judgments, whereas stimulation of disparity-selective sites had little or no effect. The binocular disparity was optimized for each stimulation site, and our result could not be explained by variations in direction tuning, response strength, or any other tuning property that we examined. When microstimulation of a disparity-tuned site did affect direction judgments, the effects tended to be stronger at the preferred disparity of a stimulation site than at the nonpreferred disparity, indicating that monkeys can selectively monitor direction columns that are best tuned to an appropriate conjunction of parameters. We conclude that the contribution of neurons to behavior can depend strongly upon tuning to stimulus dimensions that appear to be irrelevant to the current task, and we suggest that these findings are best explained in terms of the strategy used by animals to perform the task.     

Disparity tuning as simulated by a neural net

Previous research has suggested that the processing of binocular disparity in complex cells may be described with an energy formalism. The energy formalism allows for a representation of disparity by differences in the position or in the phase of monocular receptive subfields of binocular cells, or by combination of these two types. We studied the coding of disparities with an approach complementary to previous algorithmic investigations. Since realization of these representations is probably not genetically determined but learned during ontogeny, we used backpropagation networks to study which of these three possibilities were realized within neural nets. Three types of networks were trained with noise patterns in analogy to the three types of energy models. The networks learned the task and generalized to untrained correlated noise pattern input. Outputs were broadly tuned to spatial frequency and did not respond to anti-correlated noise patterns. Although the energy model was not explicitly implemented, we could analyze the outputs of the networks using predictions of the energy formalism. After learning was completed, the model neurons preferred position shifts over phase shifts in representing disparity. We discuss the general meaning of these findings and the correspondences and deviations between the energy model, V1 neurons, and our networks. Received: 6 August 1999 / Accepted in revised form: 26 January 2000     

Binocular vision: an orientation to disparity coding

A new study has shown that neurons in the visual cortex are specialized to encode the larger range of horizontal - relative to vertical - disparities that occurs in central vision. These results challenge the established 'energy' model of disparity processing.     

Stereoscopic Vision in the Absence of the Lateral Occipital Cortex

Both dorsal and ventral cortical visual streams contain neurons sensitive to binocular disparities, but the two streams may underlie different aspects of stereoscopic vision. Here we investigate stereopsis in the neurological patient D.F., whose ventral stream, specifically lateral occipital cortex, has been damaged bilaterally, causing profound visual form agnosia. Despite her severe damage to cortical visual areas, we report that DF''s stereo vision is strikingly unimpaired. She is better than many control observers at using binocular disparity to judge whether an isolated object appears near or far, and to resolve ambiguous structure-from-motion. DF is, however, poor at using relative disparity between features at different locations across the visual field. This may stem from a difficulty in identifying the surface boundaries where relative disparity is available. We suggest that the ventral processing stream may play a critical role in enabling healthy observers to extract fine depth information from relative disparities within one surface or between surfaces located in different parts of the visual field.     

Spatial stereoresolution for depth corrugations may be set in primary visual cortex

Stereo "3D" depth perception requires the visual system to extract binocular disparities between the two eyes' images. Several current models of this process, based on the known physiology of primary visual cortex (V1), do this by computing a piecewise-frontoparallel local cross-correlation between the left and right eye's images. The size of the "window" within which detectors examine the local cross-correlation corresponds to the receptive field size of V1 neurons. This basic model has successfully captured many aspects of human depth perception. In particular, it accounts for the low human stereoresolution for sinusoidal depth corrugations, suggesting that the limit on stereoresolution may be set in primary visual cortex. An important feature of the model, reflecting a key property of V1 neurons, is that the initial disparity encoding is performed by detectors tuned to locally uniform patches of disparity. Such detectors respond better to square-wave depth corrugations, since these are locally flat, than to sinusoidal corrugations which are slanted almost everywhere. Consequently, for any given window size, current models predict better performance for square-wave disparity corrugations than for sine-wave corrugations at high amplitudes. We have recently shown that this prediction is not borne out: humans perform no better with square-wave than with sine-wave corrugations, even at high amplitudes. The failure of this prediction raised the question of whether stereoresolution may actually be set at later stages of cortical processing, perhaps involving neurons tuned to disparity slant or curvature. Here we extend the local cross-correlation model to include existing physiological and psychophysical evidence indicating that larger disparities are detected by neurons with larger receptive fields (a size/disparity correlation). We show that this simple modification succeeds in reconciling the model with human results, confirming that stereoresolution for disparity gratings may indeed be limited by the size of receptive fields in primary visual cortex.     

The Role of Binocular Disparity in Stereoscopic Images of Objects in the Macaque Anterior Intraparietal Area

Neurons in the macaque Anterior Intraparietal area (AIP) encode depth structure in random-dot stimuli defined by gradients of binocular disparity, but the importance of binocular disparity in real-world objects for AIP neurons is unknown. We investigated the effect of binocular disparity on the responses of AIP neurons to images of real-world objects during passive fixation. We presented stereoscopic images of natural and man-made objects in which the disparity information was congruent or incongruent with disparity gradients present in the real-world objects, and images of the same objects where such gradients were absent. Although more than half of the AIP neurons were significantly affected by binocular disparity, the great majority of AIP neurons remained image selective even in the absence of binocular disparity. AIP neurons tended to prefer stimuli in which the depth information derived from binocular disparity was congruent with the depth information signaled by monocular depth cues, indicating that these monocular depth cues have an influence upon AIP neurons. Finally, in contrast to neurons in the inferior temporal cortex, AIP neurons do not represent images of objects in terms of categories such as animate-inanimate, but utilize representations based upon simple shape features including aspect ratio.     

Neural coding of 3D features of objects for hand action in the parietal cortex of the monkey.

In our previous studies of hand manipulation task-related neurons, we found many neurons of the parietal association cortex which responded to the sight of three-dimensional (3D) objects. Most of the task-related neurons in the AIP area (the lateral bank of the anterior intraparietal sulcus) were visually responsive and half of them responded to objects for manipulation. Most of these neurons were selective for the 3D features of the objects. More recently, we have found binocular visual neurons in the lateral bank of the caudal intraparietal sulcus (c-IPS area) that preferentially respond to a luminous bar or place at a particular orientation in space. We studied the responses of axis-orientation selective (AOS) neurons and surface-orientation selective (SOS) neurons in this area with stimuli presented on a 3D computer graphics display. The AOS neurons showed a stronger response to elongated stimuli and showed tuning to the orientation of the longitudinal axis. Many of them preferred a tilted stimulus in depth and appeared to be sensitive to orientation disparity and/or width disparity. The SOS neurons showed a stronger response to a flat than to an elongated stimulus and showed tuning to the 3D orientation of the surface. Their responses increased with the width or length of the stimulus. A considerable number of SOS neurons responded to a square in a random dot stereogram and were tuned to orientation in depth, suggesting their sensitivity to the gradient of disparity. We also found several SOS neurons that responded to a square with tilted or slanted contours, suggesting their sensitivity to orientation disparity and/or width disparity. Area c-IPS is likely to send visual signals of the 3D features of an object to area AIP for the visual guidance of hand actions.     

Stereopsis and the random element in the organization of the striate cortex.

Receptive field position and orientation disparities are both properties of binocularly discharged striate neurons. Receptive field position desparities have been used as a key element in the neural theory for binocular depth discrimination. Since most striate cells in the cat are binocular, these position disparities require that cells immediately adjacent to one another in the cortex should show a random scatter in their monocular receptive field positions. Superimposed on the progressive topographical representation of the visual field on the striate cortex there is experimental evidence for a localized monocular receptive field position scatter. The suggestion is examined that the binocular position disparities are built up out of the two monocular position scatters. An examination of receptive field orientation disparities and their relation to the random variation in the monocular preferred orientations of immediately adjacent striate neurons also leads to the conclusion that binocular orientation disparities are a consequence of the two monocular scatters. As for receptive field position, the local scatter in preferred orientation is superimposed on a progressive representation of orientation over larger areas of the cortex. The representation in the striate cortex of visual field position and of stimulus orientation is examined in relation to the correlation between the disparities in receptive field position and preferred orientation. The role of orientation disparities in binocular vision is reviewed.     

A binocular model for the simple cell

We authors propose a mathematical model for simple cell binocular response. It comprises two Gabor-type receptive fields (RF) having the same RF center, preferred spatial frequency, and preferred orientation. The model integrates the equally weighted signals from both eyes and performs a threshold operation. Poggio and Fischer (1977) classified binocular disparity cells in the striate cortex into four groups: tuned excitatory (TE), tuned inhibitory (TI), near, and far cells. They also found that most of the TE cells are ocularly balanced and that the other three types are usually unbalanced. This model can imitate these four types of disparity sensitivities and their ocular dominance tendency. We perform model fittings to Poggio's data using the “simulated annealing” method and discuss parameter dependence of the model's response. The model can also respond with exceptional disparity sensitivity: i.e., flat type, alternating type, and intermediate type.     

视差检测：简单细胞、复杂细胞及能量模型

立体视觉信息的处理在于皮层双眼性细胞的活动.皮层中简单细胞对视差的编码方式被认为有两种：位置差（position shift）和相位差（phase shift）,但简单细胞并不适合作为视差检测器.对一些复杂细胞的视差响应特性的生理研究,发现复杂细胞是一种比较适合的视差检测器.模型的研究提出基于这类简单细胞的复杂细胞能量模型,可以很好的检测视差,并可以较好的解释一些生理现象.     

Two-dimensional substructure of stereo and motion interactions in macaque visual cortex

The analysis of object motion and stereoscopic depth are important tasks that are begun at early stages of the primate visual system. Using sparse white noise, we mapped the receptive field substructure of motion and disparity interactions in neurons in V1 and MT of alert monkeys. Interactions in both regions revealed subunits similar in structure to V1 simple cells. For both motion and stereo, the scale and shape of the receptive field substructure could be predicted from conventional tuning for bars or dot-field stimuli, indicating that the small-scale interactions were repeated across the receptive fields. We also found neurons in V1 and in MT that were tuned to combinations of spatial and temporal binocular disparities, suggesting a possible neural substrate for the perceptual Pulfrich phenomenon. Our observations constrain computational and developmental models of motion-stereo integration.     

Size-disparity correlation in human binocular depth perception.

To use the small horizontal disparities between images projected to the eyes for the recovery of three-dimensional information, our visual system must first identify which feature in one eye's image corresponds with which in the other. The earliest level of disparity processing in primates (V1) contains cells that are spatial-frequency tuned. If such cells have a disparity range that covers only a single period of their mean tuning frequency, there will always be exactly one potential match within this range. Here, this 'size-disparity' hypothesis was tested by measuring the contrast sensitivity of stereopsis as a function of disparity for single bandpass-filtered items. It was found that thresholds were low and relatively constant up to disparities an order of magnitude larger than is predicted by this constraint. Furthermore, peak sensitivity was relatively independent of spatial frequency. A control experiment showed that binocular correlation of the carrier is necessary for this task. In a third experiment, the maximum disparity that supports threshold performance was compared for an isolated bandpass item and bandpass-filtered noise. This limit was found to be five times larger for the isolated stimuli. In summary, these findings show that the initial stage of disparity detection is not limited by the size-disparity constraint. For stimuli with multiple false targets, however, processes subsequent to this stage reduce the disparity range over which the correspondence problem can be solved.     

On the Contribution of Binocular Disparity to the Long-Term Memory for Natural Scenes

Binocular disparity is a fundamental dimension defining the input we receive from the visual world, along with luminance and chromaticity. In a memory task involving images of natural scenes we investigate whether binocular disparity enhances long-term visual memory. We found that forest images studied in the presence of disparity for relatively long times (7s) were remembered better as compared to 2D presentation. This enhancement was not evident for other categories of pictures, such as images containing cars and houses, which are mostly identified by the presence of distinctive artifacts rather than by their spatial layout. Evidence from a further experiment indicates that observers do not retain a trace of stereo presentation in long-term memory.     

fMRI Activity in Posterior Parietal Cortex Relates to the Perceptual Use of Binocular Disparity for Both Signal-In-Noise and Feature Difference Tasks

Visually guided action and interaction depends on the brain’s ability to (a) extract and (b) discriminate meaningful targets from complex retinal inputs. Binocular disparity is known to facilitate this process, and it is an open question how activity in different parts of the visual cortex relates to these fundamental visual abilities. Here we examined fMRI responses related to performance on two different tasks (signal-in-noise “coarse” and feature difference “fine” tasks) that have been widely used in previous work, and are believed to differentially target the visual processes of signal extraction and feature discrimination. We used multi-voxel pattern analysis to decode depth positions (near vs. far) from the fMRI activity evoked while participants were engaged in these tasks. To look for similarities between perceptual judgments and brain activity, we constructed ‘fMR-metric’ functions that described decoding performance as a function of signal magnitude. Thereafter we compared fMR-metric and psychometric functions, and report an association between judged depth and fMRI responses in the posterior parietal cortex during performance on both tasks. This highlights common stages of processing during perceptual performance on these tasks.     

Evidence for implication of primate area V1 in neural 3-D spatial localization processing.

We investigated the neural mechanisms underlying visual localization in 3-D space in area V1 of behaving monkeys. Three different sources of information, retinal disparity, viewing distance and gaze direction, that participate in these neural mechanisms are being reviewed. The way they interact with each other is studied by combining retinal and extraretinal signals. Interactions between retinal disparity and viewing distance have been shown in foveal V1; we have observed a strong modulation of the spontaneous activity and of the visual response of most V1 cells that was highly correlated with the vergence angle. As a consequence of these gain effects, neural horizontal disparity coding is favoured or refined for particular distances of fixation. Changing the gaze direction in the fronto-parallel plane also produces strong gains in the visual response of half of the cells in foveal V1. Cells tested for horizontal disparity and orientation selectivities show gain effects that occur coherently for the same spatial coordinates of the eyes. Shifts in preferred disparity also occurred in several neurons. Cells tested in calcarine V1 at retinal eccentricities larger than 10 degrees , show that horizontal disparity is encoded at least up to 20 degrees around both the horizontal and vertical meridians. At these large retinal eccentricities we found that vertical disparity is also encoded with tuning profiles similar to those of horizontal disparity coding. Combinations of horizontal and vertical disparity signals show that most cells encode both properties. In fact the expression of horizontal disparity coding depends on the vertical disparity signals that produce strong gain effects and frequent changes in peak selectivities. We conclude that the vertical disparity signal and the eye position signal serve to disambiguate the horizontal disparity signal to provide information on 3-D spatial coordinates in terms of distance, gaze direction and retinal eccentricity. We suggest that the relative weight among these different signals is the determining factor involved in the neural processing that gives information on 3-D spatial localization.     

The direction of retinal motion facilitates binocular stereopsis.

Visual information from binocular disparity and from relative motion provide information about three-dimensional structure and layout of the world. Although the mechanisms that process these cues have typically been studied independently, there is now a substantial body of evidence that suggests that they interact in the visual pathway. This paper investigates one advantage of such an interaction: whether retinal motion can be used as a matching constraint in the binocular correspondence process. Stimuli that contained identical disparity and motion signals but which differed in their fine-scale correlation were created to establish whether the direction, or the speed, of motion could enhance performance in a psychophysical task in which binocular matching is a limiting factor. The results of these experiments provide clear evidence that different directions of motion, but not different speeds, are processed separately in stereopsis. The results fit well with properties of neurons early in the cortical visual pathway which are thought to be involved in determining local matches between features in the two eyes'' images.