Cue combination in the motion correspondence problem

Image motion is a primary source of visual information about the world. However, before this information can be used the visual system must determine the spatio-temporal displacements of the features in the dynamic retinal image, which originate from objects moving in space. This is known as the motion correspondence problem. We investigated whether cross-cue matching constraints contribute to the solution of this problem, which would be consistent with physiological reports that many directionally selective cells in the visual cortex also respond to additional visual cues. We measured the maximum displacement limit (Dmax) for two-frame apparent motion sequences. Dmax increases as the number of elements in such sequences decreases. However, in our displays the total number of elements was kept constant while the number of a subset of elements, defined by a difference in contrast polarity, binocular disparity or colour, was varied. Dmax increased as the number of elements distinguished by a particular cue was decreased. Dmax was affected by contrast polarity for all observers, but only some observers were influenced by binocular disparity and others by colour information. These results demonstrate that the human visual system exploits local, cross-cue matching constraints in the solution of the motion correspondence problem.     

Detecting binocular 3D motion in static 3D noise: no effect of viewing distance

Relative binocular disparity cannot tell us the absolute 3D shape of an object, nor the 3D trajectory of its motion, unless the visual system has independent access to how far away the object is at any moment. Indeed, as the viewing distance is changed, the same disparate retinal motions will correspond to very different real 3D trajectories. In this paper we were interested in whether binocular 3D motion detection is affected by viewing distance. A visual search task was used, in which the observer is asked to detect a target dot, moving in 3D, amidst 3D stationary distractor dots. We found that distance does not affect detection performance. Motion-in-depth is consistently harder to detect than the equivalent lateral motion, for all viewing distances. For a constant retinal motion with both lateral and motion-in-depth components, detection performance is constant despite variations in viewing distance that produce large changes in the direction of the 3D trajectory. We conclude that binocular 3D motion detection relies on retinal, not absolute, visual signals.     

The direction of retinal motion facilitates binocular stereopsis.

Visual information from binocular disparity and from relative motion provide information about three-dimensional structure and layout of the world. Although the mechanisms that process these cues have typically been studied independently, there is now a substantial body of evidence that suggests that they interact in the visual pathway. This paper investigates one advantage of such an interaction: whether retinal motion can be used as a matching constraint in the binocular correspondence process. Stimuli that contained identical disparity and motion signals but which differed in their fine-scale correlation were created to establish whether the direction, or the speed, of motion could enhance performance in a psychophysical task in which binocular matching is a limiting factor. The results of these experiments provide clear evidence that different directions of motion, but not different speeds, are processed separately in stereopsis. The results fit well with properties of neurons early in the cortical visual pathway which are thought to be involved in determining local matches between features in the two eyes'' images.     

The Role of Binocular Disparity in Stereoscopic Images of Objects in the Macaque Anterior Intraparietal Area

Neurons in the macaque Anterior Intraparietal area (AIP) encode depth structure in random-dot stimuli defined by gradients of binocular disparity, but the importance of binocular disparity in real-world objects for AIP neurons is unknown. We investigated the effect of binocular disparity on the responses of AIP neurons to images of real-world objects during passive fixation. We presented stereoscopic images of natural and man-made objects in which the disparity information was congruent or incongruent with disparity gradients present in the real-world objects, and images of the same objects where such gradients were absent. Although more than half of the AIP neurons were significantly affected by binocular disparity, the great majority of AIP neurons remained image selective even in the absence of binocular disparity. AIP neurons tended to prefer stimuli in which the depth information derived from binocular disparity was congruent with the depth information signaled by monocular depth cues, indicating that these monocular depth cues have an influence upon AIP neurons. Finally, in contrast to neurons in the inferior temporal cortex, AIP neurons do not represent images of objects in terms of categories such as animate-inanimate, but utilize representations based upon simple shape features including aspect ratio.     

Two-dimensional substructure of stereo and motion interactions in macaque visual cortex

The analysis of object motion and stereoscopic depth are important tasks that are begun at early stages of the primate visual system. Using sparse white noise, we mapped the receptive field substructure of motion and disparity interactions in neurons in V1 and MT of alert monkeys. Interactions in both regions revealed subunits similar in structure to V1 simple cells. For both motion and stereo, the scale and shape of the receptive field substructure could be predicted from conventional tuning for bars or dot-field stimuli, indicating that the small-scale interactions were repeated across the receptive fields. We also found neurons in V1 and in MT that were tuned to combinations of spatial and temporal binocular disparities, suggesting a possible neural substrate for the perceptual Pulfrich phenomenon. Our observations constrain computational and developmental models of motion-stereo integration.     

Binocular coordination: reading stereoscopic sentences in depth

The present study employs a stereoscopic manipulation to present sentences in three dimensions to subjects as they read for comprehension. Subjects read sentences with (a) no depth cues, (b) a monocular depth cue that implied the sentence loomed out of the screen (i.e., increasing retinal size), (c) congruent monocular and binocular (retinal disparity) depth cues (i.e., both implied the sentence loomed out of the screen) and (d) incongruent monocular and binocular depth cues (i.e., the monocular cue implied the sentence loomed out of the screen and the binocular cue implied it receded behind the screen). Reading efficiency was mostly unaffected, suggesting that reading in three dimensions is similar to reading in two dimensions. Importantly, fixation disparity was driven by retinal disparity; fixations were significantly more crossed as readers progressed through the sentence in the congruent condition and significantly more uncrossed in the incongruent condition. We conclude that disparity depth cues are used on-line to drive binocular coordination during reading.     

Vestibular facilitation of optic flow parsing

Simultaneous object motion and self-motion give rise to complex patterns of retinal image motion. In order to estimate object motion accurately, the brain must parse this complex retinal motion into self-motion and object motion components. Although this computational problem can be solved, in principle, through purely visual mechanisms, extra-retinal information that arises from the vestibular system during self-motion may also play an important role. Here we investigate whether combining vestibular and visual self-motion information improves the precision of object motion estimates. Subjects were asked to discriminate the direction of object motion in the presence of simultaneous self-motion, depicted either by visual cues alone (i.e. optic flow) or by combined visual/vestibular stimuli. We report a small but significant improvement in object motion discrimination thresholds with the addition of vestibular cues. This improvement was greatest for eccentric heading directions and negligible for forward movement, a finding that could reflect increased relative reliability of vestibular versus visual cues for eccentric heading directions. Overall, these results are consistent with the hypothesis that vestibular inputs can help parse retinal image motion into self-motion and object motion components.     

On the inverse problem of binocular 3D motion perception

It is shown that existing processing schemes of 3D motion perception such as interocular velocity difference, changing disparity over time, as well as joint encoding of motion and disparity, do not offer a general solution to the inverse optics problem of local binocular 3D motion. Instead we suggest that local velocity constraints in combination with binocular disparity and other depth cues provide a more flexible framework for the solution of the inverse problem. In the context of the aperture problem we derive predictions from two plausible default strategies: (1) the vector normal prefers slow motion in 3D whereas (2) the cyclopean average is based on slow motion in 2D. Predicting perceived motion directions for ambiguous line motion provides an opportunity to distinguish between these strategies of 3D motion processing. Our theoretical results suggest that velocity constraints and disparity from feature tracking are needed to solve the inverse problem of 3D motion perception. It seems plausible that motion and disparity input is processed in parallel and integrated late in the visual processing hierarchy.     

Object Recognition in Flight: How Do Bees Distinguish between 3D Shapes?

Honeybees (Apis mellifera) discriminate multiple object features such as colour, pattern and 2D shape, but it remains unknown whether and how bees recover three-dimensional shape. Here we show that bees can recognize objects by their three-dimensional form, whereby they employ an active strategy to uncover the depth profiles. We trained individual, free flying honeybees to collect sugar water from small three-dimensional objects made of styrofoam (sphere, cylinder, cuboids) or folded paper (convex, concave, planar) and found that bees can easily discriminate between these stimuli. We also tested possible strategies employed by the bees to uncover the depth profiles. For the card stimuli, we excluded overall shape and pictorial features (shading, texture gradients) as cues for discrimination. Lacking sufficient stereo vision, bees are known to use speed gradients in optic flow to detect edges; could the bees apply this strategy also to recover the fine details of a surface depth profile? Analysing the bees’ flight tracks in front of the stimuli revealed specific combinations of flight maneuvers (lateral translations in combination with yaw rotations), which are particularly suitable to extract depth cues from motion parallax. We modelled the generated optic flow and found characteristic patterns of angular displacement corresponding to the depth profiles of our stimuli: optic flow patterns from pure translations successfully recovered depth relations from the magnitude of angular displacements, additional rotation provided robust depth information based on the direction of the displacements; thus, the bees flight maneuvers may reflect an optimized visuo-motor strategy to extract depth structure from motion signals. The robustness and simplicity of this strategy offers an efficient solution for 3D-object-recognition without stereo vision, and could be employed by other flying insects, or mobile robots.     

Motion, not Shape, Facilitates Association of Predation Risk with Novel Objects by Fathead Minnows (Pimephales promelas)

Injury‐released chemical cues are reliable indicators of predation risk among many aquatic taxa. When a novel, neutral stimulus is presented in tandem with chemical cues from an injured conspecific, an association is formed between the novel stimulus and apparent risk. Learned recognition of predation risk is well documented for fathead minnows, Pimephales promelas. When minnows detect alarm cues in nature they are also potentially exposed to multiple environmental stimuli, few of which are likely to be relevant indicators of risk. How do minnows discern among candidate stimuli potentially associated with predation risk? Two possibilities are shape and motion. In this study, individual piscivore‐naïve minnows were presented simultaneously with conspecific chemical alarm cues and two stimulus objects. One object was a darkened tube with its long axis in the horizontal plane (fish‐like). The second object was a black disk. Following introduction of chemical alarm cues, one of the objects was raised and lowered repeatedly. After a single conditioning trial, minnows associated risk significantly more with the previously moving object than the previously stationary object, as indicated by reduced activity. Object shape had no significant effect on response intensity in test trials. Our data suggest that minnows have been selected to form aversive responses to moving objects at a site of recent predation because movement is a more predictable indicator of predator identity than shape.     

From 1D to 2D via 3D: dynamics of surface motion segmentation for ocular tracking in primates.

In primates, tracking eye movements help vision by stabilising onto the retinas the images of a moving object of interest. This sensorimotor transformation involves several stages of motion processing, from the local measurement of one-dimensional luminance changes up to the integration of first and higher-order local motion cues into a global two-dimensional motion immune to antagonistic motions arising from the surrounding. The dynamics of this surface motion segmentation is reflected into the various components of the tracking responses and its underlying neural mechanisms can be correlated with behaviour at both single-cell and population levels. I review a series of behavioural studies which demonstrate that the neural representation driving eye movements evolves over time from a fast vector average of the outputs of linear and non-linear spatio-temporal filtering to a progressive and slower accurate solution for global motion. Because of the sensitivity of earliest ocular following to binocular disparity, antagonistic visual motion from surfaces located at different depths are filtered out. Thus, global motion integration is restricted within the depth plane of the object to be tracked. Similar dynamics were found at the level of monkey extra-striate areas MT and MST and I suggest that several parallel pathways along the motion stream are involved albeit with different latencies to build-up this accurate surface motion representation. After 200-300 ms, most of the computational problems of early motion processing (aperture problem, motion integration, motion segmentation) are solved and the eye velocity matches the global object velocity to maintain a clear and steady retinal image.     

Depth generalization from stereo to motion parallax in the owl

Although many sources of three-dimensional information have been isolated and demonstrated to contribute independently, to depth vision in animal studies, it is not clear whether these distinct cues are perceived to be perceptually equivalent. Such ability is observed in humans and would seem to be advantageous for animals as well in coping with the often co-varying (or ambiguous) information about the layout of physical space. We introduce the expression primary-depth-cue equivalence to refer to the ability to perceive mutually consistent information about differences in depth from either stereopsis or motion-parallax. We found that owls trained to detect relative depth as a perceptual category (objects versus holes) when specified by binocular disparity alone (stereopsis), immediately transferred this discrimination to novel stimuli where the equivalent depth categories were available only through differences in motion information produced by head movements (observer-produced motion-parallax). Motion-parallax discrimination did occur under monocular viewing conditions and reliable performance depended heavily on the amplitude of side-to-side head movements. The presence of primary-depth-cue equivalence in the visual system of the owl provides further conformation of the hypothesis that neural systems evolved to detect differences in either disparity or motion information are likely to share similar processing mechanisms.     

Extrapolating and interpolating surfaces in depth

Random-dot stereograms were generated with a blank area placed in part of the right-hand image so making a patchwork of monocular and binocular areas. The perceived depth and shape of the monocular region, where depth was not explicitly marked, depended in part on the depth and surface orientation of adjacent binocular areas. Thus a monocular rectangle flanked by two binocular rectangles which were placed in different fronto-parallel planes was seen as a sloping surface spanning the depth between the binocular regions, and, under some conditions, the gradient of a sloping binocular plane extended into a neighbouring monocular area. Division of the monocular region into two by textural discontinuities or discontinuities of motion sometimes altered the shape of the extrapolated surface. Often, though, the shape was unchanged by such discontinuities implying that both two- and three-dimensional features are used to segment a scene into separate surfaces. Pictorial cues also contribute to the shape and apparent depth of the monocular surface. For instance, when subjects viewed a display consisting of portions of a cube of which two ends were shown stereoscopically and one side monocularly, the monocular side was seen in three dimensions filling the gap between the ends. When stereo cues were pitted against pictorial cues, sometimes pictorial cues and sometimes stereo cues dominated, and sometimes the surface contained sharp discontinuities enabling both to be accommodated.     

A Bayesian model of stereopsis depth and motion direction discrimination

 The extraction of stereoscopic depth from retinal disparity, and motion direction from two-frame kinematograms, requires the solution of a correspondence problem. In previous psychophysical work [Read and Eagle (2000) Vision Res 40: 3345–3358], we compared the performance of the human stereopsis and motion systems with correlated and anti-correlated stimuli. We found that, although the two systems performed similarly for narrow-band stimuli, broad-band anti-correlated kinematograms produced a strong perception of reversed motion, whereas the stereograms appeared merely rivalrous. I now model these psychophysical data with a computational model of the correspondence problem based on the known properties of visual cortical cells. Noisy retinal images are filtered through a set of Fourier channels tuned to different spatial frequencies and orientations. Within each channel, a Bayesian analysis incorporating a prior preference for small disparities is used to assess the probability of each possible match. Finally, information from the different channels is combined to arrive at a judgement of stimulus disparity. Each model system – stereopsis and motion – has two free parameters: the amount of noise they are subject to, and the strength of their preference for small disparities. By adjusting these parameters independently for each system, qualitative matches are produced to psychophysical data, for both correlated and anti-correlated stimuli, across a range of spatial frequency and orientation bandwidths. The motion model is found to require much higher noise levels and a weaker preference for small disparities. This makes the motion model more tolerant of poor-quality reverse-direction false matches encountered with anti-correlated stimuli, matching the strong perception of reversed motion that humans experience with these stimuli. In contrast, the lower noise level and tighter prior preference used with the stereopsis model means that it performs close to chance with anti-correlated stimuli, in accordance with human psychophysics. Thus, the key features of the experimental data can be reproduced assuming that the motion system experiences more effective noise than the stereoscopy system and imposes a less stringent preference for small disparities. Received: 2 March 2001 / Accepted in revised form: 5 July 2001     

Linear and nonlinear transparencies in binocular vision.

When the product of a vertical square-wave grating (contrast envelope) and a horizontal sinusoidal grating (carrier) are viewed binocularly with different disparity cues they can be perceived transparently at different depths. We found, however, that the transparency was asymmetric; it only occurred when the envelope was perceived to be the overlaying surface. When the same two signals were added, the percept of transparency was symmetrical; either signal could be seen in front of or behind the other at different depths. Differences between these multiplicative and additive signal combinations were examined in two experiments. In one, we measured disparity thresholds for transparency as a function of the spatial frequency of the envelope. In the other, we measured disparity discrimination thresholds. In both experiments the thresholds for the multiplicative condition, unlike the additive condition, showed distinct minima at low envelope frequencies. The different sensitivity curves found for multiplicative and additive signal combinations suggest that different processes mediated the disparity signal. The data are consistent with a two-channel model of binocular matching, with multiple depth cues represented at single retinal locations.     

Humans ignore motion and stereo cues in favor of a fictional stable world

As we move through the world, our eyes acquire a sequence of images. The information from this sequence is sufficient to determine the structure of a three-dimensional scene, up to a scale factor determined by the distance that the eyes have moved. Previous evidence shows that the human visual system accounts for the distance the observer has walked and the separation of the eyes when judging the scale, shape, and distance of objects. However, in an immersive virtual-reality environment, observers failed to notice when a scene expanded or contracted, despite having consistent information about scale from both distance walked and binocular vision. This failure led to large errors in judging the size of objects. The pattern of errors cannot be explained by assuming a visual reconstruction of the scene with an incorrect estimate of interocular separation or distance walked. Instead, it is consistent with a Bayesian model of cue integration in which the efficacy of motion and disparity cues is greater at near viewing distances. Our results imply that observers are more willing to adjust their estimate of interocular separation or distance walked than to accept that the scene has changed in size.     

Depth resolution in the pigeon

Summary Pigeons possess a binocular visual field and a retinal region of higher cellular density pointing to the center of this overlap. These features and the precision of pecking behavior suggest that in this lateral-eyed bird cues other than monocular ones might participate in depth judgements.Pigeons were trained with an operant procedure to discriminate between luminous points differing in depth which appeared to the observer as floating in the dark. The accuracy of depth judgements was found to be a function of the ratio between the interstimulus distance and the mean eyes-to-stimulus distance. In a first test (experiment I) no external binocular disparity cues were available, the animal only seeing one luminous point at a time (near or far). In a second test (experiment II) where binocular disparity cues were available, the animal having this time to discriminate a pair of points placed at equal depth from a pair placed at unequal depths, only one pair being visible at a time, depth resolution did not improve. This suggests that, at least within the range of distances explored, the pigeon has no stereoscopic vision. Notwithstanding this, binocular cues do play a role, since when tests were done comparing binocular with monocular viewing (experiment III), monocular depth resolution was significantly worse.     

Discrimination Contours for Moving Sounds Reveal Duration and Distance Cues Dominate Auditory Speed Perception

Evidence that the auditory system contains specialised motion detectors is mixed. Many psychophysical studies confound speed cues with distance and duration cues and present sound sources that do not appear to move in external space. Here we use the ‘discrimination contours’ technique to probe the probabilistic combination of speed, distance and duration for stimuli moving in a horizontal arc around the listener in virtual auditory space. The technique produces a set of motion discrimination thresholds that define a contour in the distance-duration plane for different combination of the three cues, based on a 3-interval oddity task. The orientation of the contour (typically elliptical in shape) reveals which cue or combination of cues dominates. If the auditory system contains specialised motion detectors, stimuli moving over different distances and durations but defining the same speed should be more difficult to discriminate. The resulting discrimination contours should therefore be oriented obliquely along iso-speed lines within the distance-duration plane. However, we found that over a wide range of speeds, distances and durations, the ellipses aligned with distance-duration axes and were stretched vertically, suggesting that listeners were most sensitive to duration. A second experiment showed that listeners were able to make speed judgements when distance and duration cues were degraded by noise, but that performance was worse. Our results therefore suggest that speed is not a primary cue to motion in the auditory system, but that listeners are able to use speed to make discrimination judgements when distance and duration cues are unreliable.     

Acoustic facilitation of object movement detection during self-motion

In humans, as well as most animal species, perception of object motion is critical to successful interaction with the surrounding environment. Yet, as the observer also moves, the retinal projections of the various motion components add to each other and extracting accurate object motion becomes computationally challenging. Recent psychophysical studies have demonstrated that observers use a flow-parsing mechanism to estimate and subtract self-motion from the optic flow field. We investigated whether concurrent acoustic cues for motion can facilitate visual flow parsing, thereby enhancing the detection of moving objects during simulated self-motion. Participants identified an object (the target) that moved either forward or backward within a visual scene containing nine identical textured objects simulating forward observer translation. We found that spatially co-localized, directionally congruent, moving auditory stimuli enhanced object motion detection. Interestingly, subjects who performed poorly on the visual-only task benefited more from the addition of moving auditory stimuli. When auditory stimuli were not co-localized to the visual target, improvements in detection rates were weak. Taken together, these results suggest that parsing object motion from self-motion-induced optic flow can operate on multisensory object representations.     

Motion-boundary illusions and their regularization

Humans use various cues to understand the structure of the world from images. One such cue is the contours of an object formed by occlusion or from surface discontinuities. It is known that contours in the image of an object provide various amounts of information about the shape of the object in view, depending on assumptions that the observer makes. Another powerful cue is motion. The ability of the human visual system to discern structure from a motion stimulus is well known and has a solid theoretical and experimental foundation. However, when humans interpret a visual scene they use various cues to understand what they observe, and the interpretation comes from combining the information acquired from the various modules devoted to specific cues. In such an integration of modules it seems that each cue carries a different weight and importance. We performed several experiments where we made sure that the only cues available to the observer were contour and motion. It turns out that when humans combine information from contour and motion to reconstruct the shape of an object in view, if the results of the two modules--shape from contour and structure from motion--are inconsistent, they experience a perceptual result which is due to the combination of the two modules, with the influence of the contour dominating, thus giving rise to the illusion. We describe here examples of such illusions and identify the conditions under which they happen. Finally, we introduce a computational theory for combining contour and motion using the theory of regularization. The theory explains such illusions and predicts many more.(ABSTRACT TRUNCATED AT 250 WORDS)