A computational approach to motion perception

In this paper it is shown that the computation of the optical flow from a sequence of timevarying images is not, in general, an underconstrained problem. A local algorithm for the computation of the optical flow which uses second order derivatives of the image brightness pattern, and that avoids the aperture problem, is presented. The obtained optical flow is very similar to the true motion field — which is the vector field associated with moving features on the image plane — and can be used to recover 3D motion information. Experimental results on sequences of real images, together with estimates of relevant motion parameters, like time-to-crash for translation and angular velocity for rotation, are presented and discussed. Due to the remarkable accuracy which can be achieved in estimating motion parameters, the proposed method is likely to be very useful in a number of computer vision applications.     

Computation of optical motion by movement detectors

The first part of this paper deals with a system-theoretical approach for the decomposition of multi-input systems into the sum of simpler systems. This approach is applied here to analyse the algorithm which represents the computations underlying the extraction of motion information from the optical environment by biological movement detectors. The second part concentrates on a specific model for motion computation known to be realized by the visual system of insects and of man. These detectors provide the visual system with information on both the velocity and structural properties of a moving pattern. In the third part of this article the properties of two-dimensional arrays of movement detectors are analyzed and their relations to meaningful physiological responses are discussed.     

Tracking facilitates 3-D motion estimation

The recently emerging paradigm of Active Vision advocates studying visual problems in form of modules that are directly related to a visual task for observers that are active. Along these lines, we are arguing that in many cases when an object is moving in an unrestricted manner (translation and rotation) in the 3D world, we are just interested in the motion's translational components. For a monocular observer, using only the normal flow — the spatio-temporal derivatives of the image intensity function — we solve the problem of computing the direction of translation and the time to collision. We do not use optical flow since its computation is an ill-posed problem, and in the general case it is not the same as the motion field — the projection of 3D motion on the image plane. The basic idea of our motion parameter estimation strategy lies in the employment of fixation and tracking. Fixation simplifies much of the computation by placing the object at the center of the visual field, and the main advantage of tracking is the accumulation of information over time. We show how tracking is accomplished using normal flow measurements and use it for two different tasks in the solution process. First it serves as a tool to compensate for the lack of existence of an optical flow field and thus to estimate the translation parallel to the image plane; and second it gathers information about the motion component perpendicular to the image plane.     

Evaluation of optical motion information by movement detectors

Summary The paper is dealing in its first part with a system-theoretical approach for the decomposition of multi-input systems into the sum of simpler systems. By this approach the algorithm for the computations underlying the extraction of motion information from the optical environment by biological movement detectors is analysed. In the second part it concentrates on a specific model for motion computation known to be realized by the visual system of insects and of man. These motion detectors provide the visual system with information on both, velocity and structural properties of a moving pattern. The last part of the paper deals with the functional properties of two-dimensional arrays of movement detectors. They are analyzed and their relations to meaningful physiological responses are discussed.     

A two dimensional field theory for motion computation

The local extraction of motion information from brightness patterns by individual movement detectors of the correlation-type is considered in the first part of the paper. A two-dimensional field theory of movement detection is developed by treating the distance between two adjacent photoreceptors as a differential. In the first approximation of the theory we only consider linear terms of the time interval between the reception of a contrast element and its delayed representation by the detector and linear terms of the spatial distances between adjacent photoreceptors. As a result we may neglect terms of higher order than quadratic in a Taylor series development of the brightness pattern. The responses of pairs of individual movement detectors are combined to a local response vector. In the first approximation of the detector field theory the response vector is proportional to the instantaneous pattern velocity vector and linearly dependent on local properties of the moving pattern. The linear dependence on pattern properties is represented by a two by two tensor consisting of elements which are nonlinear, local functional of the moving pattern. Some of the properties of the tensor elements are treated in detail. So, for instance, it is shown that the off-diagonal elements of the tensor disappear when the moving pattern consists of x- and y-dependent separable components. In the second part of the paper the tensor relation leading to the output of a movement detector pair is spatially integrated. The result of the integration is an approximation to a summation of the outputs of an array of detector pairs. The spatially integrated detector tensor relates the translatory motion vector to the resultant output vector. It is shown that the angle between the motion vector and the resultant output vector is always smaller than ±90° whereas the angle between the motion vector and local response vectors, elicited by detector pairs, may cover the entire angular range. In the discussion of the paper the limits of the field theory for motion computation as well as its higher approximations are pointed out in some detail. In a special chapter the dependence of the detector response on the pattern properties is treated and in another chapter questions connected with the so called aperture problem are discussed. Furthermore, properties for compensation of the pattern dependent deviation angle by spatial physiological integration are mentioned in the discussion.     

Computational aspects of motion perception in natural and artificial vision systems.

In this paper a computational scheme for motion perception in artificial and natural vision systems is described. The scheme is motivated by a mathematical analysis in which first-order spatial properties of optical flow, such as singular points and elementary components of optical flow, are shown to be salient features for the computation and analysis of visual motion. The fact that different methods for the computation of optical flow produce similar results is explained in terms of the simple spatial structure of the image motion of rigid bodies. Singular points and elementary flow components are used to compute motion parameters, such as time-to-collision and angular velocity, and also to segment the visual field into areas which correspond to different motions. Then a number of biological implications are discussed. Electrophysiological findings suggest that the brain perceives visual motion by detecting and analysing optical flow components. However, the cortical neurons, which seem to detect elementary flow components, are not able to extract these components from more complex flows. A simple model for the organization of the receptive field of these cells, which is consistent with anatomical and electrophysiological data, is described at the end of the paper.     

End-stopping and the aperture problem: two-dimensional motion signals in macaque V1

Our perception of fine visual detail relies on small receptive fields at early stages of visual processing. However, small receptive fields tend to confound the orientation and velocity of moving edges, leading to ambiguous or inaccurate motion measurements (the aperture problem). Thus, it is often assumed that neurons in primary visual cortex (V1) carry only ambiguous motion information. Here we show that a subpopulation of V1 neurons is capable of signaling motion direction in a manner that is independent of contour orientation. Specifically, end-stopped V1 neurons obtain accurate motion measurements by responding only to the endpoints of long contours, a strategy which renders them largely immune to the aperture problem. Furthermore, the time course of end-stopping is similar to the time course of motion integration by MT neurons. These results suggest that cortical neurons might represent object motion by responding selectively to two-dimensional discontinuities in the visual scene.     

The visual ambiguity of a moving plane

It is shown that the optic flow field arising from motion relative to a visually textured plane may be characterized by eight parameters that depend on the observer's linear and angular velocity and the coordinate vector of the plane. These three vectors are not, however, uniquely determined by the values of the eight parameters. First, the optic flow field does not supply independent values for the observer's speed and distance from the plane; it only gives the ratio of these two quantities. But more unexpectedly, the equations relating the observer's linear velocity and the plane's coordinate vector to the eight parameters are still satisfied if the two vectors are interchanged or reversed in direction, or both. So in addition to the veridical interpretation of the optic flow field there exist three spurious interpretations to be considered and if possible excluded. This purpose is served by the condition that an interpretation can be seriously entertained only if it attributes every image element to a light source in the observer's field of view. This condition immediately eliminates one of the spurious interpretations, and exhibits the other two as mutually inconsistent: one of them is tenable only if all the visible sources lie on the forward half of the plane (relative to the observer's linear velocity); the other only if they all lie on the backward half-plane. If the sources are distributed over both halves of the plane, only the veridical interpretation survives. Its computation involves solving a 3 X 3 eigenvalue problem derived from the flow field. If the upper two eigenvalues coincide, the observer must be moving directly towards the plane; if the lower two eigenvalues coincide, his motion must be directly away from it; in both cases the spurious interpretation merges with the veridical one. If all three eigenvalues are equal, it may be inferred that either the observer's linear velocity vanishes or the plane is infinitely distant.     

An investigation of the mechanisms of eye movement control

Summary Feedback mechanisms exist in all the periferal sense organs including the eye, which acts as a highly efficient position control servo system. Histological studies so far have not revealed the precise circuitry of the eye movement control system but some information about it can be obtained by a study of the sources of feedback. Existing theories have considered three types of feedback originating in the oculomotor tract, in the proprioceptive fibres of the extrinsic eye muscles and from retinal image displacement. In the present experiments an optical arrangement has been used to vary or eliminate the amount of information available from retinal image motion, and the response of the eye to simple harmonic displacement of a target has been recorded. The response curves of gain (eyeball movement divided by target motion) against frequency indicate that the system is lion linear when the image falls in the retinal region which is insensitive to position. Outside this area, retinal image position is used as negative feedback but the information from the oculomotor tract must be regenerative. There is also evidence for feedback proportional to the first derivative of eyeball position and this function is ascribed to the proprioceptive signals; this form of feedback appears to saturate for large amplitude movements, thus avoiding heavy damping of the flick movements.A schematic eye movement control system having the same characteristics as the eye is proposed. The transfer function of this system indicates that it should be unstable if the sign of the retinal image feedback loop is reversed. Experiments with this form of feedback show that steady fixation is impossible and the eye performs a pendular nystagmus.     

Dynamics parametrically controlled by image correlations organize robot navigation

Behavior-based robot designs confront the problem of how different elementary behaviors can be integrated. We address two aspects of this problem: the stabilization of behavioral decisions that are induced by changing sensory information and the fusion of multiple sources of sensory information. The concrete context is homing and obstacle avoidance in a vision-guided mobile robot. Obstacle avoidance is based on extracting time-to-contact information from optic flow. A dynamical system controls heading direction and velocity. Time-to-contact estimates parametrically control this dynamical system, the attractors of which generate robot movement. Decisions come about through bifurcations of the dynamics and are stabilized through hysteresis. Homing is based on image correlations between memorized and current views. These control parametrically a dynamics of ego-position estimation, which converges in closed loop so as to position the robot at the home position. Unreliable visual information and more continous open-loop dead-reckoning information are integrated within this dynamics. This permits vision-based homing, but also stabilizes the behavior during periods of absent or erroneous visual information through the internal state of the dynamical system. The navigation scheme is demonstrated on a robot platform in real time. Received: 2 May 1995 / Accepted in revised form: 10 June 1996     

Correlation versus gradient type motion detectors: the pros and cons

Visual motion contains a wealth of information about self-motion as well as the three-dimensional structure of the environment. Therefore, it is of utmost importance for any organism with eyes. However, visual motion information is not explicitly represented at the photoreceptor level, but rather has to be computed by the nervous system from the changing retinal images as one of the first processing steps. Two prominent models have been proposed to account for this neural computation: the Reichardt detector and the gradient detector. While the Reichardt detector correlates the luminance levels derived from two adjacent image points, the gradient detector provides an estimate of the local retinal image velocity by dividing the spatial and the temporal luminance gradient. As a consequence of their different internal processing structure, both the models differ in a number of functional aspects such as their dependence on the spatial-pattern structure as well as their sensitivity to photon noise. These different properties lead to the proposal that an ideal motion detector should be of Reichardt type at low luminance levels, but of gradient type at high luminance levels. However, experiments on the fly visual systems provided unambiguous evidence in favour of the Reichardt detector under all luminance conditions. Does this mean that the fly nervous system uses suboptimal computations, or is there a functional aspect missing in the optimality criterion? In the following, I will argue in favour of the latter, showing that Reichardt detectors have an automatic gain control allowing them to dynamically adjust their input–output relationships to the statistical range of velocities presented, while gradient detectors do not have this property. As a consequence, Reichardt detectors, but not gradient detectors, always provide a maximum amount of information about stimulus velocity over a large range of velocities. This important property might explain why Reichardt type of computations have been demonstrated to underlie the extraction of motion information in the fly visual system under all luminance levels.     

Pattern-dependent response modulations in motion-sensitive visual interneurons--a model study

Even if a stimulus pattern moves at a constant velocity across the receptive field of motion-sensitive neurons, such as lobula plate tangential cells (LPTCs) of flies, the response amplitude modulates over time. The amplitude of these response modulations is related to local pattern properties of the moving retinal image. On the one hand, pattern-dependent response modulations have previously been interpreted as 'pattern-noise', because they deteriorate the neuron's ability to provide unambiguous velocity information. On the other hand, these modulations might also provide the system with valuable information about the textural properties of the environment. We analyzed the influence of the size and shape of receptive fields by simulations of four versions of LPTC models consisting of arrays of elementary motion detectors of the correlation type (EMDs). These models have previously been suggested to account for many aspects of LPTC response properties. Pattern-dependent response modulations decrease with an increasing number of EMDs included in the receptive field of the LPTC models, since spatial changes within the visual field are smoothed out by the summation of spatially displaced EMD responses. This effect depends on the shape of the receptive field, being the more pronounced--for a given total size--the more elongated the receptive field is along the direction of motion. Large elongated receptive fields improve the quality of velocity signals. However, if motion signals need to be localized the velocity coding is only poor but the signal provides--potentially useful--local pattern information. These modelling results suggest that motion vision by correlation type movement detectors is subject to uncertainty: you cannot obtain both an unambiguous and a localized velocity signal from the output of a single cell. Hence, the size and shape of receptive fields of motion sensitive neurons should be matched to their potential computational task.     

Dynamic response properties of movement detectors: Theoretical analysis and electrophysiological investigation in the visual system of the fly

Dynamic aspects of the computation of visual motion information are analysed both theoretically and experimentally. The theoretical analysis is based on the type of movement detector which has been proposed to be realized in the visual system of insects (e.g. Hassenstein and Reichardt 1956; Reichardt 1957, 1961; Buchner 1984), but also of man (e.g. van Doorn and Koenderink 1982a, b; van Santen and Sperling 1984; Wilson 1985). The output of both a single movement detector and a one-dimensional array of detectors is formulated mathematically as a function of time. The resulting movement detector theory can be applied to a much wider range of moving stimuli than has been possible on the basis of previous formulations of the detector output. These stimuli comprise one-dimensional smooth detector input functions, i.e. functions which can be expanded into a time-dependent convergent Taylor series for any value of the spatial coordinate.The movement detector response can be represented by a power series. Each term of this series consists of one exclusively time-dependent component and of another component that depends, in addition, on the properties of the pattern. Even the exclusively time-dependent components of the movement detector output are not solely determined by the stimulus velocity. They rather depend in a non-linear way on the weighted sum of the instantaneous velocity and all its higher order time derivatives. The latter point represents another reason — not discussed so far in the literature — that movement detectors of the type analysed here do not represent pure velocity sensors.The significance of this movement detector theory is established for the visual system of the fly. This is done by comparing the spatially integrated movement detector response with the functional properties of the directionally-selective motion-sensitive. Horizontal Cells of the third visual ganglion of the fly's brain.These integrate local motion information over large parts of the visual field. The time course of the spatially integrated movement detector response is about proportional to the velocity of the stimulus pattern only as long as the pattern velocity and its time derivatives are sufficiently small. For large velocities and velocity changes of the stimulus pattern characteristic deviations of the response profiles from being proportional to pattern velocity are predicted on the basis of the detector theory developed here. These deviations are clearly reflected in the response of the wide-field Horizontal Cells, thus, providing very specific evidence that the movement detector theory developed here can be applied to motion detection in the fly. The characteristic dynamic features of the theoretically predicted and the experimentally determined cellular responses are exploited to estimate the time constant of the movement detector filter.     

Movement correction of digital sequence angiographies of the retina]

Retinal hemodynamics can be quantified from videoangiographic image sequences by digital image processing. Intensity changes of dye dilution curves provide dynamics parameters of the local retinal blood flow. The measuring points of dye dilution curves have to be fixed on identical image contents in each image of a complete image sequence. To obtain measurements for every pixel on the retinal surface a motion-compensated image sequence is required. A new method adapted to the compensation of eye motion and movement artifacts in Scanning Laser Ophthalmoscopy in long image sequences (300-500 images) is presented in this paper. To inhibit error propagation of time sequential motion estimation, the eye movement is divided into two dynamic movements components. The method presented permits compensation for eye motion in retinal fluorescein angiographic sequences. Owing to the short calculation times, this algorithm can be used in clinical routine.     

A computational model of the analysis of some first-order and second-order motion patterns by simple and complex cells.

Although spatio-temporal gradient schemes are widely used in the computation of image motion, algorithms are ill conditioned for particular classes of input. This paper addresses this problem. Motion is computed as the space-time direction in which the difference in image illuminance from the local mean is conserved. This method can reliably detect motion in first-order and some second-order motion stimuli. Components of the model can be identified with directionally asymmetric and directionally selective simple cells. A stage in which we compute spatial and temporal derivatives of the difference between image illuminance and the local mean illuminance using a truncated Taylor series gives rise to a phase-invariant output reminiscent of the response of complex cells.     

Dynamic Lung Modeling and Tumor Tracking Using Deformable Image Registration and Geometric Smoothing

A greyscale-based fully automatic deformable image registration algorithm, based on an optical flow method together with geometric smoothing, is developed for dynamic lung modeling and tumor tracking. In our computational processing pipeline, the input data is a set of 4D CT images with 10 phases. The triangle mesh of the lung model is directly extracted from the more stable exhale phase (Phase 5). In addition, we represent the lung surface model in 3D volumetric format by applying a signed distance function and then generate tetrahedral meshes. Our registration algorithm works for both triangle and tetrahedral meshes. In CT images, the intensity value reflects the local tissue density. For each grid point, we calculate the displacement from the static image (Phase 5) to match with the moving image (other phases) by using merely intensity values of the CT images. The optical flow computation is followed by a regularization of the deformation field using geometric smoothing. Lung volume change and the maximum lung tissue movement are used to evaluate the accuracy of the application. Our testing results suggest that the application of deformable registration algorithm is an effective way for delineating and tracking tumor motion in image-guided radiotherapy.     

Speed tuning in elementary motion detectors of the correlation type

A prominent model of visual motion detection is the so-called correlation or Reichardt detector. Whereas this model can account for many properties of motion vision, from humans to insects (review, Borst and Egelhaaf 1989), it has been commonly assumed that this scheme of motion detection is not well suited to the measurement of image velocity. This is because the commonly used version of the model, which incorporates two unidirectional motion detectors with opposite preferred directions, produces a response which varies not only with the velocity of the image, but also with its spatial structure and contrast. On the other hand, information on image velocity can be crucial in various contexts, and a number of recent behavioural experiments suggest that insects do extract velocity for navigational purposes (review, Srinivasan et al. 1996). Here we show that other versions of the correlation model, which consists of a single unidirectional motion detector or incorporates two oppositely directed detectors with unequal sensitivities, produce responses which vary with image speed and display tuning curves that are substantially independent of the spatial structure of the image. This surprising feature suggests simple strategies of reducing ambiguities in the estimation of speed by using components of neural hardware that are already known to exist in the visual system. Received: 30 April 1998 / Accepted in revised form: 18 September 1998