Object Segmentation from Motion Discontinuities and Temporal Occlusions–A Biologically Inspired Model

Background

Optic flow is an important cue for object detection. Humans are able to perceive objects in a scene using only kinetic boundaries, and can perform the task even when other shape cues are not provided. These kinetic boundaries are characterized by the presence of motion discontinuities in a local neighbourhood. In addition, temporal occlusions appear along the boundaries as the object in front covers the background and the objects that are spatially behind it.

Methodology/Principal Findings

From a technical point of view, the detection of motion boundaries for segmentation based on optic flow is a difficult task. This is due to the problem that flow detected along such boundaries is generally not reliable. We propose a model derived from mechanisms found in visual areas V1, MT, and MSTl of human and primate cortex that achieves robust detection along motion boundaries. It includes two separate mechanisms for both the detection of motion discontinuities and of occlusion regions based on how neurons respond to spatial and temporal contrast, respectively. The mechanisms are embedded in a biologically inspired architecture that integrates information of different model components of the visual processing due to feedback connections. In particular, mutual interactions between the detection of motion discontinuities and temporal occlusions allow a considerable improvement of the kinetic boundary detection.

Conclusions/Significance

A new model is proposed that uses optic flow cues to detect motion discontinuities and object occlusion. We suggest that by combining these results for motion discontinuities and object occlusion, object segmentation within the model can be improved. This idea could also be applied in other models for object segmentation. In addition, we discuss how this model is related to neurophysiological findings. The model was successfully tested both with artificial and real sequences including self and object motion.     

A neurally plausible model of the dynamics of motion integration in smooth eye pursuit based on recursive Bayesian estimation

In this study, we describe a model of motion integration in smooth eye pursuit based on a recursive Bayesian estimation process, which displays a dynamic behaviour qualitatively similar to the dynamics of the motion integration process observed experimentally, both psychophysically in humans and monkeys, and physiologically in monkeys. By formulating the model as an approximate version of a Kalman filter algorithm, we have been able to show that it can be put into a neurally plausible, distributed recurrent form which coarsely corresponds to the recurrent circuitry of visual cortical areas V1 and MT. The model thus provides further support for the notion that the motion integration process is based on a form of Bayesian estimation, as has been suggested by many psychophysical studies, and moreover suggests that the observed dynamic properties of this process are the result of the recursive nature of the motion estimation.     

Predicting Cortical Dark/Bright Asymmetries from Natural Image Statistics and Early Visual Transforms

The nervous system has evolved in an environment with structure and predictability. One of the ubiquitous principles of sensory systems is the creation of circuits that capitalize on this predictability. Previous work has identified predictable non-uniformities in the distributions of basic visual features in natural images that are relevant to the encoding tasks of the visual system. Here, we report that the well-established statistical distributions of visual features -- such as visual contrast, spatial scale, and depth -- differ between bright and dark image components. Following this analysis, we go on to trace how these differences in natural images translate into different patterns of cortical input that arise from the separate bright (ON) and dark (OFF) pathways originating in the retina. We use models of these early visual pathways to transform natural images into statistical patterns of cortical input. The models include the receptive fields and non-linear response properties of the magnocellular (M) and parvocellular (P) pathways, with their ON and OFF pathway divisions. The results indicate that there are regularities in visual cortical input beyond those that have previously been appreciated from the direct analysis of natural images. In particular, several dark/bright asymmetries provide a potential account for recently discovered asymmetries in how the brain processes visual features, such as violations of classic energy-type models. On the basis of our analysis, we expect that the dark/bright dichotomy in natural images plays a key role in the generation of both cortical and perceptual asymmetries.     

Polarization contrast and motion detection

Form and motion perception rely upon the visual system’s capacity to segment the visual scene based upon local differences in luminance or wavelength. It is not clear if polarization contrast is a sufficient basis for motion detection. Here we show that crayfish optomotor responses elicited by the motion of images derived from spatiotemporal variations in e-vector angles are comparable to contrast-elicited responses. Response magnitude increases with the difference in e-vector angles in adjacent segments of the scene and with the degree of polarization but the response is relatively insensitive to the absolute values of e-vector angles that compose the stimulus. The results indicate that polarization contrast can support visual motion detection.     

Mathematical analysis and modeling of motion direction selectivity in the retina

Motion detection is one of the most important and primitive computations performed by our visual system. Specifically in the retina, ganglion cells producing motion direction-selective responses have been addressed by different disciplines, such as mathematics, neurophysiology and computational modeling, since the beginnings of vision science. Although a number of studies have analyzed theoretical and mathematical considerations for such responses, a clear picture of the underlying cellular mechanisms is only recently emerging. In general, motion direction selectivity is based on a non-linear asymmetric computation inside a receptive field differentiating cell responses between preferred and null direction stimuli. To what extent can biological findings match these considerations? In this review, we outline theoretical and mathematical studies of motion direction selectivity, aiming to map the properties of the models onto the neural circuitry and synaptic connectivity found in the retina. Additionally, we review several compartmental models that have tried to fill this gap. Finally, we discuss the remaining challenges that computational models will have to tackle in order to fully understand the retinal motion direction-selective circuitry.     

Speed tuning in elementary motion detectors of the correlation type

A prominent model of visual motion detection is the so-called correlation or Reichardt detector. Whereas this model can account for many properties of motion vision, from humans to insects (review, Borst and Egelhaaf 1989), it has been commonly assumed that this scheme of motion detection is not well suited to the measurement of image velocity. This is because the commonly used version of the model, which incorporates two unidirectional motion detectors with opposite preferred directions, produces a response which varies not only with the velocity of the image, but also with its spatial structure and contrast. On the other hand, information on image velocity can be crucial in various contexts, and a number of recent behavioural experiments suggest that insects do extract velocity for navigational purposes (review, Srinivasan et al. 1996). Here we show that other versions of the correlation model, which consists of a single unidirectional motion detector or incorporates two oppositely directed detectors with unequal sensitivities, produce responses which vary with image speed and display tuning curves that are substantially independent of the spatial structure of the image. This surprising feature suggests simple strategies of reducing ambiguities in the estimation of speed by using components of neural hardware that are already known to exist in the visual system. Received: 30 April 1998 / Accepted in revised form: 18 September 1998     

Non-directional motion detectors can be used to mimic optic flow dependent behaviors

Insect navigational behaviors including obstacle avoidance, grazing landings, and visual odometry are dependent on the ability to estimate flight speed based only on visual cues. In honeybees, this visual estimate of speed is largely independent of both the direction of motion and the spatial frequency content of the image. Electrophysiological recordings from the motion-sensitive cells believed to underlie these behaviors have long supported spatio-temporally tuned correlation-type models of visual motion detection whose speed tuning changes as the spatial frequency of a stimulus is varied. The result is an apparent conflict between behavioral experiments and the electrophysiological and modeling data. In this article, we demonstrate that conventional correlation-type models are sufficient to reproduce some of the speed-dependent behaviors observed in honeybees when square wave gratings are used, contrary to the theoretical predictions. However, these models fail to match the behavioral observations for sinusoidal stimuli. Instead, we show that non-directional motion detectors, which underlie the correlation-based computation of directional motion, can be used to mimic these same behaviors even when narrowband gratings are used. The existence of such non-directional motion detectors is supported both anatomically and electrophysiologically, and they have been hypothesized to be critical in the Dipteran elementary motion detector (EMD) circuit.     

Learning receptive fields using predictive feedback.

Previously, it was suggested that feedback connections from higher- to lower-level areas carry predictions of lower-level neural activities, whereas feedforward connections carry the residual error between the predictions and the actual lower-level activities [Rao, R.P.N., Ballard, D.H., 1999. Nature Neuroscience 2, 79-87.]. A computational model implementing the hypothesis learned simple cell receptive fields when exposed to natural images. Here, we use predictive feedback to explain tuning properties in medial superior temporal area (MST). We implement the hypothesis using a new, biologically plausible, algorithm based on matching pursuit, which retains all the features of the previous implementation, including its ability to efficiently encode input. When presented with natural images, the model developed receptive field properties as found in primary visual cortex. In addition, when exposed to visual motion input resulting from movements through space, the model learned receptive field properties resembling those in MST. These results corroborate the idea that predictive feedback is a general principle used by the visual system to efficiently encode natural input.     

A Dynamic Neural Field Model of Mesoscopic Cortical Activity Captured with Voltage-Sensitive Dye Imaging

A neural field model is presented that captures the essential non-linear characteristics of activity dynamics across several millimeters of visual cortex in response to local flashed and moving stimuli. We account for physiological data obtained by voltage-sensitive dye (VSD) imaging which reports mesoscopic population activity at high spatio-temporal resolution. Stimulation included a single flashed square, a single flashed bar, the line-motion paradigm – for which psychophysical studies showed that flashing a square briefly before a bar produces sensation of illusory motion within the bar – and moving squares controls. We consider a two-layer neural field (NF) model describing an excitatory and an inhibitory layer of neurons as a coupled system of non-linear integro-differential equations. Under the assumption that the aggregated activity of both layers is reflected by VSD imaging, our phenomenological model quantitatively accounts for the observed spatio-temporal activity patterns. Moreover, the model generalizes to novel similar stimuli as it matches activity evoked by moving squares of different speeds. Our results indicate that feedback from higher brain areas is not required to produce motion patterns in the case of the illusory line-motion paradigm. Physiological interpretation of the model suggests that a considerable fraction of the VSD signal may be due to inhibitory activity, supporting the notion that balanced intra-layer cortical interactions between inhibitory and excitatory populations play a major role in shaping dynamic stimulus representations in the early visual cortex.     

Detecting binocular 3D motion in static 3D noise: no effect of viewing distance

Relative binocular disparity cannot tell us the absolute 3D shape of an object, nor the 3D trajectory of its motion, unless the visual system has independent access to how far away the object is at any moment. Indeed, as the viewing distance is changed, the same disparate retinal motions will correspond to very different real 3D trajectories. In this paper we were interested in whether binocular 3D motion detection is affected by viewing distance. A visual search task was used, in which the observer is asked to detect a target dot, moving in 3D, amidst 3D stationary distractor dots. We found that distance does not affect detection performance. Motion-in-depth is consistently harder to detect than the equivalent lateral motion, for all viewing distances. For a constant retinal motion with both lateral and motion-in-depth components, detection performance is constant despite variations in viewing distance that produce large changes in the direction of the 3D trajectory. We conclude that binocular 3D motion detection relies on retinal, not absolute, visual signals.     

Neural correlates of contrast detection at threshold

Human psychophysical studies have demonstrated that, for stimuli near the threshold of visibility, detection of motion in one direction is unaffected by the superimposition of motion in the opposite direction. To investigate the neural basis for this perceptual phenomenon, we recorded from directionally selective neurons in macaque visual area MT (middle temporal visual area). Contrast thresholds obtained for single gratings moving in a neuron's preferred direction were compared with those obtained for motion presented simultaneously in the neuron's preferred and antipreferred directions. A simple model based on probability summation between neurons tuned to opposite directions could sufficiently account for contrast thresholds revealed psychophysically, suggesting that area MT is likely to provide the neural basis for contrast detection of stimuli modulated in time.     

Response properties of motion-sensitive visual interneurons in the lobula plate of Drosophila melanogaster

The crystalline-like structure of the optic lobes of the fruit fly Drosophila melanogaster has made them a model system for the study of neuronal cell-fate determination, axonal path finding, and target selection. For functional studies, however, the small size of the constituting visual interneurons has so far presented a formidable barrier. We have overcome this problem by establishing in vivo whole-cell recordings from genetically targeted visual interneurons of Drosophila. Here, we describe the response properties of six motion-sensitive large-field neurons in the lobula plate that form a network consisting of individually identifiable, directionally selective cells most sensitive to vertical image motion (VS cells). Individual VS cell responses to visual motion stimuli exhibit all the characteristics that are indicative of presynaptic input from elementary motion detectors of the correlation type. Different VS cells possess distinct receptive fields that are arranged sequentially along the eye's azimuth, corresponding to their characteristic cellular morphology and position within the retinotopically organized lobula plate. In addition, lateral connections between individual VS cells cause strongly overlapping receptive fields that are wider than expected from their dendritic input. Our results suggest that motion vision in different dipteran fly species is accomplished in similar circuitries and according to common algorithmic rules. The underlying neural mechanisms of population coding within the VS cell network and of elementary motion detection, respectively, can now be analyzed by the combination of electrophysiology and genetic intervention in Drosophila.     

Cooperative Integration and Representation Underlying Bilateral Network of Fly Motion-Sensitive Neurons

How is binocular motion information integrated in the bilateral network of wide-field motion-sensitive neurons, called lobula plate tangential cells (LPTCs), in the visual system of flies? It is possible to construct an accurate model of this network because a complete picture of synaptic interactions has been experimentally identified. We investigated the cooperative behavior of the network of horizontal LPTCs underlying the integration of binocular motion information and the information representation in the bilateral LPTC network through numerical simulations on the network model. First, we qualitatively reproduced rotational motion-sensitive response of the H2 cell previously reported in vivo experiments and ascertained that it could be accounted for by the cooperative behavior of the bilateral network mainly via interhemispheric electrical coupling. We demonstrated that the response properties of single H1 and Hu cells, unlike H2 cells, are not influenced by motion stimuli in the contralateral visual hemi-field, but that the correlations between these cell activities are enhanced by the rotational motion stimulus. We next examined the whole population activity by performing principal component analysis (PCA) on the population activities of simulated LPTCs. We showed that the two orthogonal patterns of correlated population activities given by the first two principal components represent the rotational and translational motions, respectively, and similar to the H2 cell, rotational motion produces a stronger response in the network than does translational motion. Furthermore, we found that these population-coding properties are strongly influenced by the interhemispheric electrical coupling. Finally, to test the generality of our conclusions, we used a more simplified model and verified that the numerical results are not specific to the network model we constructed.     

Internal structure of the fly elementary motion detector

Recent experiments have shown that motion detection in Drosophila starts with splitting the visual input into two parallel channels encoding brightness increments (ON) or decrements (OFF). This suggests the existence of either two (ON-ON, OFF-OFF) or four (for all pairwise interactions) separate motion detectors. To decide between these possibilities, we stimulated flies using sequences of ON and OFF brightness pulses while recording from motion-sensitive tangential cells. We found direction-selective responses to sequences of same sign (ON-ON, OFF-OFF), but not of opposite sign (ON-OFF, OFF-ON), refuting the existence of four separate detectors. Based on further measurements, we propose a model that reproduces a variety of additional experimental data sets, including ones that were previously interpreted as support for four separate detectors. Our experiments and the derived model mark an important step in guiding further dissection of the fly motion detection circuit.     

Modeling and measuring the visual detection of ecologically relevant motion by an Anolis lizard

Motion in the visual periphery of lizards, and other animals, often causes a shift of visual attention toward the moving object. This behavioral response must be more responsive to relevant motion (predators, prey, conspecifics) than to irrelevant motion (windblown vegetation). Early stages of visual motion detection rely on simple local circuits known as elementary motion detectors (EMDs). We presented a computer model consisting of a grid of correlation-type EMDs, with videos of natural motion patterns, including prey, predators and windblown vegetation. We systematically varied the model parameters and quantified the relative response to the different classes of motion. We carried out behavioral experiments with the lizard Anolis sagrei and determined that their visual response could be modeled with a grid of correlation-type EMDs with a spacing parameter of 0.3° visual angle, and a time constant of 0.1 s. The model with these parameters gave substantially stronger responses to relevant motion patterns than to windblown vegetation under equivalent conditions. However, the model is sensitive to local contrast and viewer-object distance. Therefore, additional neural processing is probably required for the visual system to reliably distinguish relevant from irrelevant motion under a full range of natural conditions.     

Manuo-ocular coordination in target tracking. II. Comparing the model with human behavior

Several studies have shown that humans track a moving visual target with their eyes better if the movement of this target is directly controlled by the observer's hand. The improvement in performance has been attributed to coordination control between the arm motor system and the smooth pursuit (SP) system. In such a task, the SP system shows characteristics that differ from those observed during eye-alone tracking: latency (between the target-arm and the eye motion onsets) is shorter, maximum SP velocity is higher and the maximum target motion frequency at which the SP can function effectively is also higher. The aim of this article is to qualitatively evaluate the behavior of a dynamical model simulating the oculomotor system and the arm motor system when both are involved in tracking visual targets. The evaluation is essentially based on a comparison of the behavior of the model with the behavior of human subjects tracking visual targets under different conditions. The model has been introduced and quantitatively evaluated in a companion paper. The model is based on an exchange of internal information between the two sensorimotor systems, mediated by sensory signals (vision, arm muscle proprioception) and motor signals (arm motor command copy). The exchange is achieved by a specialized structure of the central nervous system, previously identified as a part of the cerebellum. Computer simulation of the model yielded results that fit the behavior of human subjects observed during previously reported experiments, both qualitatively and quantitatively. The parallelism between physiology and human behavior on the one hand, and structure and simulation of the model on the other hand, is discussed. Received: 6 March 1997 / Accepted in revised form: 15 July 1997     

An information integration model of the primary visual cortex under grating stimulations

During the course of information processing, a visual system extracts characteristic information of the visual image and integrates the spatial and temporal visual information simultaneously. In this study, we investigate the integration effect of neurons in the primary visual cortex (V1 area) under the grating stimulation. First, an information integration model was established based on the receptive field properties of the extracted features of the visual images features, the interaction between neurons and the nonlinear integration of those neurons. Then the neuropsychological experiments were designed both to provide parameters for the model and to verify its effect. The experimental results with factual visual image were largely consistent with the model’s forecast output. This demonstrates that our model can truly reflect the integration effect of the primary visual system when being subjected to grating stimulations with different orientations. Our results indicate the primary visual system integrates the visual information in the following manner: it first extracts visual information through different types of receptive field, and then its neurons interact with each other in a non-linear manner, finally the neurons fire spikes recorded as responses to the visual stimulus.     

Quantification of phase synchronization phenomena and their importance for verbal memory processes

The tangential neurons in the lobula plate region of the flies are known to respond to visual motion across broad receptive fields in visual space.When intracellular recordings are made from tangential neurons while the intact animal is stimulated visually with moving natural imagery,we find that neural response depends upon speed of motion but is nearly invariant with respect to variations in natural scenery. We refer to this invariance as velocity constancy. It is remarkable because natural scenes, in spite of similarities in spatial structure, vary considerably in contrast, and contrast dependence is a feature of neurons in the early visual pathway as well as of most models for the elementary operations of visual motion detection. Thus, we expect that operations must be present in the processing pathway that reduce contrast dependence in order to approximate velocity constancy.We consider models for such operations, including spatial filtering, motion adaptation, saturating nonlinearities, and nonlinear spatial integration by the tangential neurons themselves, and evaluate their effects in simulations of a tangential neuron and precursor processing in response to animated natural imagery. We conclude that all such features reduce interscene variance in response, but that the model system does not approach velocity constancy as closely as the biological tangential cell.     

On the contours of apparent motion: A new perspective on visual space-time

Previous results on the perception of motion indicate that perceived motion paths cannot be explained solely in terms of simple feature-specific analyzers. This is particularly true of apparent (phi) motion. In this paper we develop a dynamic network, with simple filtering and summation properties, which can predict the geometric paths of apparent motion in various spatio-temporal configurations. The network assumptions predict a non-Euclidean metric for the visual space-time of motion perception and we consider the implications of such distortions for various visual displays, including illusions.     

Learning the selectivity of V2 and V4 neurons using non-linear multi-layer wavelet networks

We investigate a non-linear network with two processing stages optimized to reduce the statistical dependencies in natural images. This network serves as a model for the neural information processing in the higher visual areas of primates (visual cortices V2-V4). The resulting population is analyzed with regard to non-linear selectivity and invariance properties. We find units that are very selective with respect to the space spanned by all possible input signals and units that are invariant with respect to certain stimulus classes. In comparison to the measured distribution of selectivity in V2 neurons, the selectivity histogram of the network units shows an even more pronounced tendency towards higher selectivities. A special property of the system is the emergence of non-linear interactions between coefficients from different scales and orientations, which are necessary for the exploitation of higher-order statistical redundancies of natural images. We extend the concept to multi-layer systems and present some simulation results.