Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

Vertical Binocular Disparity is Encoded Implicitly within a Model Neuronal Population Tuned to Horizontal Disparity and Orientation

Primary visual cortex is often viewed as a “cyclopean retina”, performing the initial encoding of binocular disparities between left and right images. Because the eyes are set apart horizontally in the head, binocular disparities are predominantly horizontal. Yet, especially in the visual periphery, a range of non-zero vertical disparities do occur and can influence perception. It has therefore been assumed that primary visual cortex must contain neurons tuned to a range of vertical disparities. Here, I show that this is not necessarily the case. Many disparity-selective neurons are most sensitive to changes in disparity orthogonal to their preferred orientation. That is, the disparity tuning surfaces, mapping their response to different two-dimensional (2D) disparities, are elongated along the cell''s preferred orientation. Because of this, even if a neuron''s optimal 2D disparity has zero vertical component, the neuron will still respond best to a non-zero vertical disparity when probed with a sub-optimal horizontal disparity. This property can be used to decode 2D disparity, even allowing for realistic levels of neuronal noise. Even if all V1 neurons at a particular retinotopic location are tuned to the expected vertical disparity there (for example, zero at the fovea), the brain could still decode the magnitude and sign of departures from that expected value. This provides an intriguing counter-example to the common wisdom that, in order for a neuronal population to encode a quantity, its members must be tuned to a range of values of that quantity. It demonstrates that populations of disparity-selective neurons encode much richer information than previously appreciated. It suggests a possible strategy for the brain to extract rarely-occurring stimulus values, while concentrating neuronal resources on the most commonly-occurring situations.     

The direction of retinal motion facilitates binocular stereopsis.

Visual information from binocular disparity and from relative motion provide information about three-dimensional structure and layout of the world. Although the mechanisms that process these cues have typically been studied independently, there is now a substantial body of evidence that suggests that they interact in the visual pathway. This paper investigates one advantage of such an interaction: whether retinal motion can be used as a matching constraint in the binocular correspondence process. Stimuli that contained identical disparity and motion signals but which differed in their fine-scale correlation were created to establish whether the direction, or the speed, of motion could enhance performance in a psychophysical task in which binocular matching is a limiting factor. The results of these experiments provide clear evidence that different directions of motion, but not different speeds, are processed separately in stereopsis. The results fit well with properties of neurons early in the cortical visual pathway which are thought to be involved in determining local matches between features in the two eyes'' images.     

Perceptual “Read-Out” of Conjoined Direction and Disparity Maps in Extrastriate Area MT

Cortical neurons are frequently tuned to several stimulus dimensions, and many cortical areas contain intercalated maps of multiple variables. Relatively little is known about how information is “read out” of these multidimensional maps. For example, how does an organism extract information relevant to the task at hand from neurons that are also tuned to other, irrelevant stimulus dimensions? We addressed this question by employing microstimulation techniques to examine the contribution of disparity-tuned neurons in the middle temporal (MT) visual area to performance on a direction discrimination task. Most MT neurons are tuned to both binocular disparity and the direction of stimulus motion, and MT contains topographic maps of both parameters. We assessed the effect of microstimulation on direction judgments after first characterizing the disparity tuning of each stimulation site. Although the disparity of the stimulus was irrelevant to the required task, we found that microstimulation effects were strongly modulated by the disparity tuning of the stimulated neurons. For two of three monkeys, microstimulation of nondisparity-selective sites produced large biases in direction judgments, whereas stimulation of disparity-selective sites had little or no effect. The binocular disparity was optimized for each stimulation site, and our result could not be explained by variations in direction tuning, response strength, or any other tuning property that we examined. When microstimulation of a disparity-tuned site did affect direction judgments, the effects tended to be stronger at the preferred disparity of a stimulation site than at the nonpreferred disparity, indicating that monkeys can selectively monitor direction columns that are best tuned to an appropriate conjunction of parameters. We conclude that the contribution of neurons to behavior can depend strongly upon tuning to stimulus dimensions that appear to be irrelevant to the current task, and we suggest that these findings are best explained in terms of the strategy used by animals to perform the task.     

Perceptual “Read-Out” of Conjoined Direction and Disparity Maps in Extrastriate Area MT

Cortical neurons are frequently tuned to several stimulus dimensions, and many cortical areas contain intercalated maps of multiple variables. Relatively little is known about how information is “read out” of these multidimensional maps. For example, how does an organism extract information relevant to the task at hand from neurons that are also tuned to other, irrelevant stimulus dimensions? We addressed this question by employing microstimulation techniques to examine the contribution of disparity-tuned neurons in the middle temporal (MT) visual area to performance on a direction discrimination task. Most MT neurons are tuned to both binocular disparity and the direction of stimulus motion, and MT contains topographic maps of both parameters. We assessed the effect of microstimulation on direction judgments after first characterizing the disparity tuning of each stimulation site. Although the disparity of the stimulus was irrelevant to the required task, we found that microstimulation effects were strongly modulated by the disparity tuning of the stimulated neurons. For two of three monkeys, microstimulation of nondisparity-selective sites produced large biases in direction judgments, whereas stimulation of disparity-selective sites had little or no effect. The binocular disparity was optimized for each stimulation site, and our result could not be explained by variations in direction tuning, response strength, or any other tuning property that we examined. When microstimulation of a disparity-tuned site did affect direction judgments, the effects tended to be stronger at the preferred disparity of a stimulation site than at the nonpreferred disparity, indicating that monkeys can selectively monitor direction columns that are best tuned to an appropriate conjunction of parameters. We conclude that the contribution of neurons to behavior can depend strongly upon tuning to stimulus dimensions that appear to be irrelevant to the current task, and we suggest that these findings are best explained in terms of the strategy used by animals to perform the task.     

Binocular visual processing in the owl's telencephalon.

Single neurons recorded from the owl's visual Wulst are surprisingly similar to those found in mammalian striate cortex. The receptive fields of Wulst neurons are elaborated, in an apparently hierarchical fashion, from those of their monocular, concentrically organized inputs to produce binocular interneurons with increasingly sophisticated requirements for stimulus orientation, movement and binocular disparity. Output neurons located in the superficial laminae of the Wulst are the most sophisticated of all, with absolute requirements for a combination of stimuli, which include binocular presentation at a particular horizontal binocular disparity, and with no response unless all of the stimulus conditions are satisfied simultaneously. Such neurons have the properties required for 'global stereopsis', including a receptive field size many times larger than their optimal stimulus, which is more closely matched to the receptive fields of the simpler, disparity-selective interneurons. These marked similarities in functional organization between the avian and mammalian systems exist in spite of a number of structural differences which reflect their separate evoluntionary origins. Discussion therefore includes the possibility that there may exist for nervous systems only a very small number of possible solutions, perhaps a unique one, to the problem of stereopsis.     

A laminar cortical model of stereopsis and 3D surface perception: closure and da Vinci stereopsis

A laminar cortical model of stereopsis and 3D surface perception is developed and simulated. The model describes how monocular and binocular oriented filtering interact with later stages of 3D boundary formation and surface filling-in in the LGN and cortical areas V1, V2, and V4. It proposes how interactions between layers 4, 3B, and 2/3 in V1 and V2 contribute to stereopsis, and how binocular and monocular information combine to form 3D boundary and surface representations. The model includes two main new developments: (1) It clarifies how surface-to-boundary feedback from V2 thin stripes to pale stripes helps to explain data about stereopsis. This feedback has previously been used to explain data about 3D figure-ground perception. (2) It proposes that the binocular false match problem is subsumed under the Gestalt grouping problem. In particular, the disparity filter, which helps to solve the correspondence problem by eliminating false matches, is realized using inhibitory interneurons as part of the perceptual grouping process by horizontal connections in layer 2/3 of cortical area V2. The enhanced model explains all the psychophysical data previously simulated by Grossberg and Howe (2003), such as contrast variations of dichoptic masking and the correspondence problem, the effect of interocular contrast differences on stereoacuity, Panum's limiting case, the Venetian blind illusion, stereopsis with polarity-reversed stereograms, and da Vinci stereopsis. It also explains psychophysical data about perceptual closure and variations of da Vinci stereopsis that previous models cannot yet explain.     

The development of the binocular depth cells in the secondary visual cortex of the lamb.

In most respects, the response properties of cells in the secondary visual cortex of the newborn lamb were indistinguishable from those in the adult. The cells were sharply selective to orientation; the orientation preferences were the same in each eye, and they varied systematically as the electrode penetrated the cortex. The receptive-field organization did not differ noticeably from that in adults, and complex, hypercomplex, and a few simple cells were all observed. The ocular dominance distribution was similar to that in the adult. Most importantly, binocular cells were found with disparate receptive fields even in newborn, visually inexperienced animals. As in the adult, the disparities were largely horizontal, and they appeared to be arranged in columns. Many of the cells responded preferentially to a binocular stimulus at a particular disparity setting (often approximately zero), but unlike those in the adult almost all the binocular cells in the newborn lamb would also respond monocularly, and the enhancement at the optimal disparity was less than in the adult. The full development of binocular selectivity took several weeks, and was blocked by binocular deprivation. We conclude that the basic wiring of stereoscopic mechanisms is innate, but the development of mature binocular interaction may depend on an adaptive process which makes use of the visual information received during binocular stimulation.     

A model of binocular stereopsis including a global consistency constraint

 The binocular correspondence problem was solved by implementing the uniqueness constraint and the continuity constraint, as proposed by Marr and Poggio [Marr D, PoggioT (1976) Science 194: 283–287]. However, these constraints are not sufficient to define the proper correspondence uniquely. With these constraints, random-dot stereograms (RDSs), consisting of the periodic textures in each image, are treated as a correspondence of surfaces composed of patches of alternating values of disparity. This is quite different from the surface we perceive through the RDSs, that is a surface characterized by a single depth. Because these constraints are local, they cannot produce the global optimum of correspondence. To obtain the global optimum of correspondence, we propose a model of binocular stereopsis in which a global measure of correspondence is explicitly employed. The model consists of two hierarchical systems. First, the lower system processes various correspondences based on the uniqueness constraint. Second, the higher system provides a global measure of correspondence for the disparity in question. The higher system uniquely determines the global optimum of correspondence in the lower system through the recurrent loop between hierarchical systems. The convergence of the recurrent loop is determined by the consistency between the hierarchical systems. The condition is termed the `global consistency constraint. Received: 27 August 1998 / Accepted in revised form: 8 November 1999     

Touch Interacts with Vision during Binocular Rivalry with a Tight Orientation Tuning

Multisensory integration is a common feature of the mammalian brain that allows it to deal more efficiently with the ambiguity of sensory input by combining complementary signals from several sensory sources. Growing evidence suggests that multisensory interactions can occur as early as primary sensory cortices. Here we present incompatible visual signals (orthogonal gratings) to each eye to create visual competition between monocular inputs in primary visual cortex where binocular combination would normally take place. The incompatibility prevents binocular fusion and triggers an ambiguous perceptual response in which the two images are perceived one at a time in an irregular alternation. One key function of multisensory integration is to minimize perceptual ambiguity by exploiting cross-sensory congruence. We show that a haptic signal matching one of the visual alternatives helps disambiguate visual perception during binocular rivalry by both prolonging the dominance period of the congruent visual stimulus and by shortening its suppression period. Importantly, this interaction is strictly tuned for orientation, with a mismatch as small as 7.5° between visual and haptic orientations sufficient to annul the interaction. These results indicate important conclusions: first, that vision and touch interact at early levels of visual processing where interocular conflicts are first detected and orientation tunings are narrow, and second, that haptic input can influence visual signals outside of visual awareness, bringing a stimulus made invisible by binocular rivalry suppression back to awareness sooner than would occur without congruent haptic input.     

Neural coding of 3D features of objects for hand action in the parietal cortex of the monkey.

In our previous studies of hand manipulation task-related neurons, we found many neurons of the parietal association cortex which responded to the sight of three-dimensional (3D) objects. Most of the task-related neurons in the AIP area (the lateral bank of the anterior intraparietal sulcus) were visually responsive and half of them responded to objects for manipulation. Most of these neurons were selective for the 3D features of the objects. More recently, we have found binocular visual neurons in the lateral bank of the caudal intraparietal sulcus (c-IPS area) that preferentially respond to a luminous bar or place at a particular orientation in space. We studied the responses of axis-orientation selective (AOS) neurons and surface-orientation selective (SOS) neurons in this area with stimuli presented on a 3D computer graphics display. The AOS neurons showed a stronger response to elongated stimuli and showed tuning to the orientation of the longitudinal axis. Many of them preferred a tilted stimulus in depth and appeared to be sensitive to orientation disparity and/or width disparity. The SOS neurons showed a stronger response to a flat than to an elongated stimulus and showed tuning to the 3D orientation of the surface. Their responses increased with the width or length of the stimulus. A considerable number of SOS neurons responded to a square in a random dot stereogram and were tuned to orientation in depth, suggesting their sensitivity to the gradient of disparity. We also found several SOS neurons that responded to a square with tilted or slanted contours, suggesting their sensitivity to orientation disparity and/or width disparity. Area c-IPS is likely to send visual signals of the 3D features of an object to area AIP for the visual guidance of hand actions.     

Effects of Prism Eyeglasses on Objective and Subjective Fixation Disparity

In optometry of binocular vision, the question may arise whether prisms should be included in eyeglasses to compensate an oculomotor and/or sensory imbalance between the two eyes. The corresponding measures of objective and subjective fixation disparity may be reduced by the prisms, or the adaptability of the binocular vergence system may diminish effects of the prisms over time. This study investigates effects of wearing prisms constantly for about 5 weeks in daily life. Two groups of 12 participants received eyeglasses with prisms having either a base-in direction or a base-out direction with an amount up to 8 prism diopters. Prisms were prescribed based on clinical fixation disparity test plates at 6 m. Two dependent variables were used: (1) subjective fixation disparity was indicated by a perceived offset of dichoptic nonius lines that were superimposed on the fusion stimuli and (2) objective fixation disparity was measured with a video based eye tracker relative to monocular calibration. Stimuli were presented at 6 m and included either central or more peripheral fusion stimuli. Repeated measurements were made without the prisms and with the prisms after about 5 weeks of wearing these prisms. Objective and subjective fixation disparity were correlated, but the type of fusion stimulus and the direction of the required prism may play a role. The prisms did not reduce the fixation disparity to zero, but induced significant changes in fixation disparity with large effect sizes. Participants receiving base-out prisms showed hypothesized effects, which were concurrent in both types of fixation disparity. In participants receiving base-in prisms, the individual effects of subjective and objective effects were negatively correlated: the larger the subjective (sensory) effect, the smaller the objective (motor) effect. This response pattern was related to the vergence adaptability, i.e. the individual fusional vergence reserves.     

Geometry of binocular vision and a model for stereopsis

If a binocular observer looks at surfaces, the disparity is a continuous vector field defined on the manifold of cyclopean visual directions. We derive this field for the general case that the observer is presented with a curved surface and fixates an arbitrary point. We expand the disparity field in the neighbourhood of a visual direction. The first order approximation can be decomposed into congruences, similarities and deformations. The deformation component is described by the traceless part of the symmetric part of the gradient of the disparity. The deformation component carries all information concerning the slant of a surface element that is contained in the disparity field itself; it is invariant for changes of fixation, differential cyclotorsion and uniform aniseikonia. The deformation component can be found from a comparison of the orientation of surface details in the left and right retinal images. The theory provides a geometric explanation of the percepts obtained with uniform and oblique meridional aniseikonia. We utilize the geometric theory to construct a mechanistic model of stereopsis that obviates the need for internal zooming mechanisms, but nevertheless is insensitive to differential cyclotorsion or uniform aniseikonia.     

Detecting binocular 3D motion in static 3D noise: no effect of viewing distance

Relative binocular disparity cannot tell us the absolute 3D shape of an object, nor the 3D trajectory of its motion, unless the visual system has independent access to how far away the object is at any moment. Indeed, as the viewing distance is changed, the same disparate retinal motions will correspond to very different real 3D trajectories. In this paper we were interested in whether binocular 3D motion detection is affected by viewing distance. A visual search task was used, in which the observer is asked to detect a target dot, moving in 3D, amidst 3D stationary distractor dots. We found that distance does not affect detection performance. Motion-in-depth is consistently harder to detect than the equivalent lateral motion, for all viewing distances. For a constant retinal motion with both lateral and motion-in-depth components, detection performance is constant despite variations in viewing distance that produce large changes in the direction of the 3D trajectory. We conclude that binocular 3D motion detection relies on retinal, not absolute, visual signals.     

Columnar architecture and computational anatomy in primate visual cortex: segmentation and feature extraction via spatial frequency coded difference mapping

Columnar architecture is a well established organizational principle for a variety of cortical systems. If two topographically mapped receptor systems, which receive slightly different views of the same physical stimulus, are interlaced as columns, then the difference map of the afferent inputs is coded within a spatial frequency channel of the resultant map. The difference map of the left and right retinal views of a three dimensensional scene contains cues for the binocular disparity of the objects in the scene. Physical objects which are located at a common distance from the observer will be represented by area's of difference mapping which possesss common cortical textural values. Thus, segmentation of the cortical representation of the visual scene by values of positional disparity may be accomplished by conventional monocular segmentation techniques, applied to the cortical representation.The difference map is carried by a spatial frequency modulation determined by the period of the columnar interlacing. Ocular dominance columns in human striate cortex suggest a spatial frequency carrier which is roughly equal to the inverse of Panum's area. Since the difference mapping is a global attribute of the cortical representation, and is not contingent on the existence of labeled single cell feature extractors, the difference mapping algorithm represents a distinct alternative to conventional single cell approaches to feature extraction.The difference mapping algorithm is briefly discussed in relation to other difference channels, such as color opponent segmentation and binocular orientation disparity. It is suggested that difference mapping may reflect a general synergistic mechanism relating topographic mapping and columnar architecture, which reduces the problem of feature extraction and segmentation for depth and color opponent channels to a single textural mechanism.     

Panum's phenomenon and the confluence of signals from the two eyes in stereoscopy

The signals from the two eyes must be routed to allow either eye to have access to the processing mechanisms for position, shape, colour, etc.; at the same time, information as to the eye of origin must be retained for the purposes of stereoscopy. The study of this confluence of signals from the two eyes was approached psychophysically by studying induced position and depth changes of adjacent binocular and monocular stimuli in the human fovea. It was demonstrated that a monocular visual stimulus located near a binocular one acquires a depth signal, according to a scheme originally proposed by Panum. The effect is unspecific as regards feature shape and brightness, and falls off with a length constant of about 15 minutes of arc in the fovea. A monocular stimulus also affects the apparent depth of its binocular neighbour in a centre-surround manner; disparity pooling changes to disparity repulsion when features are separated by distances of about 3 minutes of arc in the fovea. The findings led to the development of a scheme of uniocular connectivity to a matrix of depth units. Excitation patterns here would depend on the state of the input lines, the intrinsic neuronal interaction properties, and contextural configuring influences from other parts of the nervous system. Experiments showing the spatial extent of pooling and repulsive interaction within the disparity domain help to characterize the stimulus processing in this neural ensemble.     

On the inverse problem of binocular 3D motion perception

It is shown that existing processing schemes of 3D motion perception such as interocular velocity difference, changing disparity over time, as well as joint encoding of motion and disparity, do not offer a general solution to the inverse optics problem of local binocular 3D motion. Instead we suggest that local velocity constraints in combination with binocular disparity and other depth cues provide a more flexible framework for the solution of the inverse problem. In the context of the aperture problem we derive predictions from two plausible default strategies: (1) the vector normal prefers slow motion in 3D whereas (2) the cyclopean average is based on slow motion in 2D. Predicting perceived motion directions for ambiguous line motion provides an opportunity to distinguish between these strategies of 3D motion processing. Our theoretical results suggest that velocity constraints and disparity from feature tracking are needed to solve the inverse problem of 3D motion perception. It seems plausible that motion and disparity input is processed in parallel and integrated late in the visual processing hierarchy.     

On the role of attention in binocular rivalry: electrophysiological evidence

During binocular rivalry visual consciousness fluctuates between two dissimilar monocular images. We investigated the role of attention in this phenomenon by comparing event-related potentials (ERPs) when binocular-rivalry stimuli were attended with when they were unattended. Stimuli were dichoptic, orthogonal gratings that yielded binocular rivalry and dioptic, identically oriented gratings that yielded binocular fusion. Events were all possible orthogonal changes in orientation of one or both gratings. We had two attention conditions: In the attend-to-grating condition, participants had to report changes in perceived orientation, focussing their attention on the gratings. In the attend-to-fixation condition participants had to report changes in a central fixation target, taking attention away from the gratings. We found, surprisingly, that attending to rival gratings yielded a smaller ERP component (the N1, from 160-210 ms) than attending to the fixation target. To explain this paradoxical effect of attention, we propose that rivalry occurs in the attend-to-fixation condition (we found an ERP signature of rivalry in the form of a sustained negativity from 210-300 ms) but that the mechanism processing the stimulus changes is more adapted in the attend-to-grating condition than in the attend-to-fixation condition. This is consistent with the theory that adaptation gives rise to changes of visual consciousness during binocular rivalry.     

Size-disparity correlation in human binocular depth perception.

To use the small horizontal disparities between images projected to the eyes for the recovery of three-dimensional information, our visual system must first identify which feature in one eye's image corresponds with which in the other. The earliest level of disparity processing in primates (V1) contains cells that are spatial-frequency tuned. If such cells have a disparity range that covers only a single period of their mean tuning frequency, there will always be exactly one potential match within this range. Here, this 'size-disparity' hypothesis was tested by measuring the contrast sensitivity of stereopsis as a function of disparity for single bandpass-filtered items. It was found that thresholds were low and relatively constant up to disparities an order of magnitude larger than is predicted by this constraint. Furthermore, peak sensitivity was relatively independent of spatial frequency. A control experiment showed that binocular correlation of the carrier is necessary for this task. In a third experiment, the maximum disparity that supports threshold performance was compared for an isolated bandpass item and bandpass-filtered noise. This limit was found to be five times larger for the isolated stimuli. In summary, these findings show that the initial stage of disparity detection is not limited by the size-disparity constraint. For stimuli with multiple false targets, however, processes subsequent to this stage reduce the disparity range over which the correspondence problem can be solved.