Cue combination in the motion correspondence problem

Image motion is a primary source of visual information about the world. However, before this information can be used the visual system must determine the spatio-temporal displacements of the features in the dynamic retinal image, which originate from objects moving in space. This is known as the motion correspondence problem. We investigated whether cross-cue matching constraints contribute to the solution of this problem, which would be consistent with physiological reports that many directionally selective cells in the visual cortex also respond to additional visual cues. We measured the maximum displacement limit (Dmax) for two-frame apparent motion sequences. Dmax increases as the number of elements in such sequences decreases. However, in our displays the total number of elements was kept constant while the number of a subset of elements, defined by a difference in contrast polarity, binocular disparity or colour, was varied. Dmax increased as the number of elements distinguished by a particular cue was decreased. Dmax was affected by contrast polarity for all observers, but only some observers were influenced by binocular disparity and others by colour information. These results demonstrate that the human visual system exploits local, cross-cue matching constraints in the solution of the motion correspondence problem.     

A laminar cortical model of stereopsis and 3D surface perception: closure and da Vinci stereopsis

A laminar cortical model of stereopsis and 3D surface perception is developed and simulated. The model describes how monocular and binocular oriented filtering interact with later stages of 3D boundary formation and surface filling-in in the LGN and cortical areas V1, V2, and V4. It proposes how interactions between layers 4, 3B, and 2/3 in V1 and V2 contribute to stereopsis, and how binocular and monocular information combine to form 3D boundary and surface representations. The model includes two main new developments: (1) It clarifies how surface-to-boundary feedback from V2 thin stripes to pale stripes helps to explain data about stereopsis. This feedback has previously been used to explain data about 3D figure-ground perception. (2) It proposes that the binocular false match problem is subsumed under the Gestalt grouping problem. In particular, the disparity filter, which helps to solve the correspondence problem by eliminating false matches, is realized using inhibitory interneurons as part of the perceptual grouping process by horizontal connections in layer 2/3 of cortical area V2. The enhanced model explains all the psychophysical data previously simulated by Grossberg and Howe (2003), such as contrast variations of dichoptic masking and the correspondence problem, the effect of interocular contrast differences on stereoacuity, Panum's limiting case, the Venetian blind illusion, stereopsis with polarity-reversed stereograms, and da Vinci stereopsis. It also explains psychophysical data about perceptual closure and variations of da Vinci stereopsis that previous models cannot yet explain.     

Binocular fusion in Panum''''s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the "correspondence problem". While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum's limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum's limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum's limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum's limiting case and that the uniqueness constraint is still valid.     

Blur and disparity are complementary cues to depth

Estimating depth from binocular disparity is extremely precise, and the cue does not depend on statistical regularities in the environment. Thus, disparity is commonly regarded as the best visual cue for determining 3D layout. But depth from disparity is only precise near where one is looking; it is quite imprecise elsewhere. Away from fixation, vision resorts to using other depth cues-e.g., linear perspective, familiar size, aerial perspective. But those cues depend on statistical regularities in the environment and are therefore not always reliable. Depth from defocus blur relies on fewer assumptions and has the same geometric constraints as disparity but different physiological constraints. Blur could in principle fill in the parts of visual space where disparity is imprecise. We tested this possibility with a depth-discrimination experiment. Disparity was more precise near fixation and blur was indeed more precise away from fixation. When both cues were available, observers relied on the more informative one. Blur appears to play an important, previously unrecognized role in depth perception. Our findings lead to a new hypothesis about the evolution of slit-shaped pupils and have implications for the design and implementation of stereo 3D displays.     

Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

Detecting binocular 3D motion in static 3D noise: no effect of viewing distance

Relative binocular disparity cannot tell us the absolute 3D shape of an object, nor the 3D trajectory of its motion, unless the visual system has independent access to how far away the object is at any moment. Indeed, as the viewing distance is changed, the same disparate retinal motions will correspond to very different real 3D trajectories. In this paper we were interested in whether binocular 3D motion detection is affected by viewing distance. A visual search task was used, in which the observer is asked to detect a target dot, moving in 3D, amidst 3D stationary distractor dots. We found that distance does not affect detection performance. Motion-in-depth is consistently harder to detect than the equivalent lateral motion, for all viewing distances. For a constant retinal motion with both lateral and motion-in-depth components, detection performance is constant despite variations in viewing distance that produce large changes in the direction of the 3D trajectory. We conclude that binocular 3D motion detection relies on retinal, not absolute, visual signals.     

Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

Spatial stereoresolution for depth corrugations may be set in primary visual cortex

Stereo "3D" depth perception requires the visual system to extract binocular disparities between the two eyes' images. Several current models of this process, based on the known physiology of primary visual cortex (V1), do this by computing a piecewise-frontoparallel local cross-correlation between the left and right eye's images. The size of the "window" within which detectors examine the local cross-correlation corresponds to the receptive field size of V1 neurons. This basic model has successfully captured many aspects of human depth perception. In particular, it accounts for the low human stereoresolution for sinusoidal depth corrugations, suggesting that the limit on stereoresolution may be set in primary visual cortex. An important feature of the model, reflecting a key property of V1 neurons, is that the initial disparity encoding is performed by detectors tuned to locally uniform patches of disparity. Such detectors respond better to square-wave depth corrugations, since these are locally flat, than to sinusoidal corrugations which are slanted almost everywhere. Consequently, for any given window size, current models predict better performance for square-wave disparity corrugations than for sine-wave corrugations at high amplitudes. We have recently shown that this prediction is not borne out: humans perform no better with square-wave than with sine-wave corrugations, even at high amplitudes. The failure of this prediction raised the question of whether stereoresolution may actually be set at later stages of cortical processing, perhaps involving neurons tuned to disparity slant or curvature. Here we extend the local cross-correlation model to include existing physiological and psychophysical evidence indicating that larger disparities are detected by neurons with larger receptive fields (a size/disparity correlation). We show that this simple modification succeeds in reconciling the model with human results, confirming that stereoresolution for disparity gratings may indeed be limited by the size of receptive fields in primary visual cortex.     

Isotropic integration of binocular disparity and relative motion in the perception of three-dimensional shape

Richards (1985) showed that veridical three-dimensional shape may be recovered from the integration of binocular disparity and retinal motion information, but proposed that this integration may only occur for horizontal retinal motion. Psychophysical evidence supporting the combination of stereo and motion information is limited to the case of horizontal motion (Johnston et al., 1994), and has been criticised on the grounds of potential object boundary cues to shape present in the stimuli. We investigated whether veridical shape can be recovered under more general conditions. Observers viewed cylinders that were defined by binocular disparity, two-frame motion or a combination of disparity and motion, presented at simulated distances of 30 cm, 90 cm or 150 cm. Horizontally and vertically oriented cylinders were rotated about vertical and horizontal axes. When rotation was about the cylinder's own axis, no boundary cues to shape were introduced. Settings were biased for the disparity and two-frame motion stimuli, while more veridical shape judgements were made under all conditions for combined cue stimuli. These results demonstrate that the improved perception of three-dimensional shape in these stimuli is not a consequence of the presence of object boundary cues, and that the combination of disparity and motion is not restricted to horizontal image motion.     

From 1D to 2D via 3D: dynamics of surface motion segmentation for ocular tracking in primates.

In primates, tracking eye movements help vision by stabilising onto the retinas the images of a moving object of interest. This sensorimotor transformation involves several stages of motion processing, from the local measurement of one-dimensional luminance changes up to the integration of first and higher-order local motion cues into a global two-dimensional motion immune to antagonistic motions arising from the surrounding. The dynamics of this surface motion segmentation is reflected into the various components of the tracking responses and its underlying neural mechanisms can be correlated with behaviour at both single-cell and population levels. I review a series of behavioural studies which demonstrate that the neural representation driving eye movements evolves over time from a fast vector average of the outputs of linear and non-linear spatio-temporal filtering to a progressive and slower accurate solution for global motion. Because of the sensitivity of earliest ocular following to binocular disparity, antagonistic visual motion from surfaces located at different depths are filtered out. Thus, global motion integration is restricted within the depth plane of the object to be tracked. Similar dynamics were found at the level of monkey extra-striate areas MT and MST and I suggest that several parallel pathways along the motion stream are involved albeit with different latencies to build-up this accurate surface motion representation. After 200-300 ms, most of the computational problems of early motion processing (aperture problem, motion integration, motion segmentation) are solved and the eye velocity matches the global object velocity to maintain a clear and steady retinal image.     

Cortical processing and perceived timing

As of yet, it is unclear how we determine relative perceived timing. One controversial suggestion is that timing perception might be related to when analyses are completed in the cortex of the brain. An alternate proposal suggests that perceived timing is instead related to the point in time at which cortical analyses commence. Accordingly, timing illusions should not occur owing to cortical analyses, but they could occur if there were differential delays between signals reaching cortex. Resolution of this controversy therefore requires that the contributions of cortical processing be isolated from the influence of subcortical activity. Here, we have done this by using binocular disparity changes, which are known to be detected via analyses that originate in cortex. We find that observers require longer stimulus exposures to detect small, relative to larger, disparity changes; observers are slower to react to smaller disparity changes and observers misperceive smaller disparity changes as being perceptually delayed. Interestingly, disparity magnitude influenced perceived timing more dramatically than it did stimulus change detection. Our data therefore suggest that perceived timing is both influenced by cortical processing and is shaped by sensory analyses subsequent to those that are minimally necessary for stimulus change perception.     

A model of binocular stereopsis including a global consistency constraint

 The binocular correspondence problem was solved by implementing the uniqueness constraint and the continuity constraint, as proposed by Marr and Poggio [Marr D, PoggioT (1976) Science 194: 283–287]. However, these constraints are not sufficient to define the proper correspondence uniquely. With these constraints, random-dot stereograms (RDSs), consisting of the periodic textures in each image, are treated as a correspondence of surfaces composed of patches of alternating values of disparity. This is quite different from the surface we perceive through the RDSs, that is a surface characterized by a single depth. Because these constraints are local, they cannot produce the global optimum of correspondence. To obtain the global optimum of correspondence, we propose a model of binocular stereopsis in which a global measure of correspondence is explicitly employed. The model consists of two hierarchical systems. First, the lower system processes various correspondences based on the uniqueness constraint. Second, the higher system provides a global measure of correspondence for the disparity in question. The higher system uniquely determines the global optimum of correspondence in the lower system through the recurrent loop between hierarchical systems. The convergence of the recurrent loop is determined by the consistency between the hierarchical systems. The condition is termed the `global consistency constraint. Received: 27 August 1998 / Accepted in revised form: 8 November 1999     

Stochastic correlative firing for figure-ground segregation

Segregation of sensory inputs into separate objects is a central aspect of perception and arises in all sensory modalities. The figure-ground segregation problem requires identifying an object of interest in a complex scene, in many cases given binaural auditory or binocular visual observations. The computations required for visual and auditory figure-ground segregation share many common features and can be cast within a unified framework. Sensory perception can be viewed as a problem of optimizing information transmission. Here we suggest a stochastic correlative firing mechanism and an associative learning rule for figure-ground segregation in several classic sensory perception tasks, including the cocktail party problem in binaural hearing, binocular fusion of stereo images, and Gestalt grouping in motion perception.     

Experiment Research on Binocular Perceptual Characteristics around Peripheral Vision for Development of Head Mounted Display with Wide View

The head mounted display （HMD） is widely used in virtual reality technology. In common HMD, however, the binocular disparity is set to an equal fixed value in the entire range of view. Such HMD systems have several shortcomings when used for wide views. In this study, in order to realize a natural stereo sensation of HMD with wide view, we measure the characteristics of binocular stereo perception and binocular light perception. Results show that both the stereoacuity and light sensitivity decrease as the retina＇s eccentricity increases from fovea to periphery. However, the decrease of the stereoacuity is more rapid than that of the light sensitivity. These results suggest that the binocular disparity at the peripheral field should be small, otherwise double images would be observed instead of a stereo view. Based on the results we develop a relative binocular stereoacuity model which can be applied for the design of HMD systems with wide view.     

Lesions to right posterior parietal cortex impair visual depth perception from disparity but not motion cues

The posterior parietal cortex (PPC) is understood to be active when observers perceive three-dimensional (3D) structure. However, it is not clear how central this activity is in the construction of 3D spatial representations. Here, we examine whether PPC is essential for two aspects of visual depth perception by testing patients with lesions affecting this region. First, we measured subjects'' ability to discriminate depth structure in various 3D surfaces and objects using binocular disparity. Patients with lesions to right PPC (N = 3) exhibited marked perceptual deficits on these tasks, whereas those with left hemisphere lesions (N = 2) were able to reliably discriminate depth as accurately as control subjects. Second, we presented an ambiguous 3D stimulus defined by structure from motion to determine whether PPC lesions influence the rate of bistable perceptual alternations. Patients'' percept durations for the 3D stimulus were generally within a normal range, although the two patients with bilateral PPC lesions showed the fastest perceptual alternation rates in our sample. Intermittent stimulus presentation reduced the reversal rate similarly across subjects. Together, the results suggest that PPC plays a causal role in both inferring and maintaining the perception of 3D structure with stereopsis supported primarily by the right hemisphere, but do not lend support to the view that PPC is a critical contributor to bistable perceptual alternations.This article is part of the themed issue ‘Vision in our three-dimensional world’.     

Effect of pictorial depth cues, binocular disparity cues and motion parallax depth cues on lightness perception in three-dimensional virtual scenes

Background

Surface lightness perception is affected by scene interpretation. There is some experimental evidence that perceived lightness under bi-ocular viewing conditions is different from perceived lightness in actual scenes but there are also reports that viewing conditions have little or no effect on perceived color. We investigated how mixes of depth cues affect perception of lightness in three-dimensional rendered scenes containing strong gradients of illumination in depth.

Methodology/Principal Findings

Observers viewed a virtual room (4 m width×5 m height×17.5 m depth) with checkerboard walls and floor. In four conditions, the room was presented with or without binocular disparity (BD) depth cues and with or without motion parallax (MP) depth cues. In all conditions, observers were asked to adjust the luminance of a comparison surface to match the lightness of test surfaces placed at seven different depths (8.5–17.5 m) in the scene. We estimated lightness versus depth profiles in all four depth cue conditions. Even when observers had only pictorial depth cues (no MP, no BD), they partially but significantly discounted the illumination gradient in judging lightness. Adding either MP or BD led to significantly greater discounting and both cues together produced the greatest discounting. The effects of MP and BD were approximately additive. BD had greater influence at near distances than far.

Conclusions/Significance

These results suggest the surface lightness perception is modulated by three-dimensional perception/interpretation using pictorial, binocular-disparity, and motion-parallax cues additively. We propose a two-stage (2D and 3D) processing model for lightness perception.     

Stereoscopic Offset Makes Objects Easier to Recognize

Binocular vision is obviously useful for depth perception, but it might also enhance other components of visual processing, such as image segmentation. We used naturalistic images to determine whether giving an object a stereoscopic offset of 15-120 arcmin of crossed disparity relative to its background would make the object easier to recognize in briefly presented (33-133 ms), temporally masked displays. Disparity had a beneficial effect across a wide range of disparities and display durations. Most of this benefit occurred whether or not the stereoscopic contour agreed with the object’s luminance contour. We attribute this benefit to an orienting of spatial attention that selected the object and its local background for enhanced 2D pattern processing. At longer display durations, contour agreement provided an additional benefit, and a separate experiment using random-dot stimuli confirmed that stereoscopic contours plausibly contributed to recognition at the longer display durations in our experiment. We conclude that in real-world situations binocular vision confers an advantage not only for depth perception, but also for recognizing objects from their luminance patterns and bounding contours.