Tracking facilitates 3-D motion estimation

The recently emerging paradigm of Active Vision advocates studying visual problems in form of modules that are directly related to a visual task for observers that are active. Along these lines, we are arguing that in many cases when an object is moving in an unrestricted manner (translation and rotation) in the 3D world, we are just interested in the motion's translational components. For a monocular observer, using only the normal flow — the spatio-temporal derivatives of the image intensity function — we solve the problem of computing the direction of translation and the time to collision. We do not use optical flow since its computation is an ill-posed problem, and in the general case it is not the same as the motion field — the projection of 3D motion on the image plane. The basic idea of our motion parameter estimation strategy lies in the employment of fixation and tracking. Fixation simplifies much of the computation by placing the object at the center of the visual field, and the main advantage of tracking is the accumulation of information over time. We show how tracking is accomplished using normal flow measurements and use it for two different tasks in the solution process. First it serves as a tool to compensate for the lack of existence of an optical flow field and thus to estimate the translation parallel to the image plane; and second it gathers information about the motion component perpendicular to the image plane.     

Non-directional motion detectors can be used to mimic optic flow dependent behaviors

Insect navigational behaviors including obstacle avoidance, grazing landings, and visual odometry are dependent on the ability to estimate flight speed based only on visual cues. In honeybees, this visual estimate of speed is largely independent of both the direction of motion and the spatial frequency content of the image. Electrophysiological recordings from the motion-sensitive cells believed to underlie these behaviors have long supported spatio-temporally tuned correlation-type models of visual motion detection whose speed tuning changes as the spatial frequency of a stimulus is varied. The result is an apparent conflict between behavioral experiments and the electrophysiological and modeling data. In this article, we demonstrate that conventional correlation-type models are sufficient to reproduce some of the speed-dependent behaviors observed in honeybees when square wave gratings are used, contrary to the theoretical predictions. However, these models fail to match the behavioral observations for sinusoidal stimuli. Instead, we show that non-directional motion detectors, which underlie the correlation-based computation of directional motion, can be used to mimic these same behaviors even when narrowband gratings are used. The existence of such non-directional motion detectors is supported both anatomically and electrophysiologically, and they have been hypothesized to be critical in the Dipteran elementary motion detector (EMD) circuit.     

Figure tracking by flies is supported by parallel visual streams

Visual figures may be distinguished based on elementary motion or higher-order non-Fourier features, and flies track both. The canonical elementary motion detector, a compact computation for Fourier motion direction and amplitude, can also encode higher-order signals provided elaborate preprocessing. However, the way in which a fly tracks a moving figure containing both elementary and higher-order signals has not been investigated. Using a novel white noise approach, we demonstrate that (1) the composite response to an object containing both elementary motion (EM) and uncorrelated higher-order figure motion (FM) reflects the linear superposition of each component; (2) the EM-driven component is velocity-dependent, whereas the FM component is driven by retinal position; (3) retinotopic variation in EM and FM responses are different from one another; (4) the FM subsystem superimposes saccadic turns upon smooth pursuit; and (5) the two systems in combination are necessary and sufficient to predict the full range of figure tracking behaviors, including those that generate no EM cues at all. This analysis requires an extension of the model that fly motion vision is based on simple elementary motion detectors and provides a novel method to characterize the subsystems responsible for the pursuit of visual figures.     

A computational approach to motion perception

In this paper it is shown that the computation of the optical flow from a sequence of timevarying images is not, in general, an underconstrained problem. A local algorithm for the computation of the optical flow which uses second order derivatives of the image brightness pattern, and that avoids the aperture problem, is presented. The obtained optical flow is very similar to the true motion field — which is the vector field associated with moving features on the image plane — and can be used to recover 3D motion information. Experimental results on sequences of real images, together with estimates of relevant motion parameters, like time-to-crash for translation and angular velocity for rotation, are presented and discussed. Due to the remarkable accuracy which can be achieved in estimating motion parameters, the proposed method is likely to be very useful in a number of computer vision applications.     

Fly-like visuomotor responses of a robot using aVLSI motion-sensitive chips

 We explore the use of continuous-time analog very-large-scale-integrated (aVLSI) neuromorphic visual preprocessors together with a robotic platform in generating bio-inspired behaviors. Both the aVLSI motion sensors and the robot behaviors described in this work are inspired by the motion computation in the fly visual system and two different fly behaviors. In most robotic systems, the visual information comes from serially scanned imagers. This restricts the form of computation of the visual image and slows down the input rate to the controller system of the robot, hence increasing the reaction time of the robot. These aVLSI neuromorphic sensors reduce the computational load and power consumption of the robot, thus making it possible to explore continuous-time visuomotor control systems that react in real-time to the environment. The motion sensor provides two outputs: one for the preferred direction and the other for the null direction. These motion outputs are created from the aggregation of six elementary motion detectors that implement a variant of Reichardt's correlation algorithm. The four analog continuous-time outputs from the motion chips go to the control system on the robot which generates a mixture of two behaviors – course stabilization and fixation – from the outputs of these sensors. Since there are only four outputs, the amount of information transmitted to the controller is reduced (as compared to using a CCD sensor), and the reaction time of the robot is greatly decreased. In this work, the robot samples the motion sensors every 3.3 ms during the behavioral experiments. Received: 4 October 1999 / Accepted in revised form: 26 April 2001     

一种感知视觉运动信息的简化脑模型

视觉运动信息的感知过程,包括从局域运动检测到对模式整体运动的感知过程.我们以蝇视觉系统的图形-背景相对运动分辨的神经回路网络为基本框架,采用初级运动检测器的六角形阵列作为输入层,构造了一种感知视觉运动信息的简化脑模型,模拟了运动信息应该神经计算模型各个层次上的处理.该模型对差分行为实验结果作出了正确预测.本文并对空间生理整合的神经机制作了讨论.     

A system of insect neurons sensitive to horizontal and vertical image motion connects the medulla and midbrain

Summary The response properties and gross morphologies of neurons that connect the medulla and midbrain in the butterfly Papilio aegeus are described. The neurons presented give direction-selective responses, i.e. they are excited by motion in the preferred direction and the background activity of the cells is inhibited by motion in the opposite, null, direction. The neurons are either maximally sensitive to horizontal motion or to slightly off-axis vertical upward or vertical downward motion, when tested in the frontal visual field. The responses of the cells are dependent on the contrast frequency of the stimulus with peak values at 5–10 Hz. The receptive fields of the medulla neurons are large and are most sensitive in the frontal visual field. Examination of the local and global properties of the receptive fields of the medulla neurons indicates that (1) they are fed by local elementary motion-detectors consistent with the correlation model and (2) there is a non-linear spatial integration mechanism in operation.     

Speed tuning in elementary motion detectors of the correlation type

A prominent model of visual motion detection is the so-called correlation or Reichardt detector. Whereas this model can account for many properties of motion vision, from humans to insects (review, Borst and Egelhaaf 1989), it has been commonly assumed that this scheme of motion detection is not well suited to the measurement of image velocity. This is because the commonly used version of the model, which incorporates two unidirectional motion detectors with opposite preferred directions, produces a response which varies not only with the velocity of the image, but also with its spatial structure and contrast. On the other hand, information on image velocity can be crucial in various contexts, and a number of recent behavioural experiments suggest that insects do extract velocity for navigational purposes (review, Srinivasan et al. 1996). Here we show that other versions of the correlation model, which consists of a single unidirectional motion detector or incorporates two oppositely directed detectors with unequal sensitivities, produce responses which vary with image speed and display tuning curves that are substantially independent of the spatial structure of the image. This surprising feature suggests simple strategies of reducing ambiguities in the estimation of speed by using components of neural hardware that are already known to exist in the visual system. Received: 30 April 1998 / Accepted in revised form: 18 September 1998     

How Lovebirds Maneuver Rapidly Using Super-Fast Head Saccades and Image Feature Stabilization

Diurnal flying animals such as birds depend primarily on vision to coordinate their flight path during goal-directed flight tasks. To extract the spatial structure of the surrounding environment, birds are thought to use retinal image motion (optical flow) that is primarily induced by motion of their head. It is unclear what gaze behaviors birds perform to support visuomotor control during rapid maneuvering flight in which they continuously switch between flight modes. To analyze this, we measured the gaze behavior of rapidly turning lovebirds in a goal-directed task: take-off and fly away from a perch, turn on a dime, and fly back and land on the same perch. High-speed flight recordings revealed that rapidly turning lovebirds perform a remarkable stereotypical gaze behavior with peak saccadic head turns up to 2700 degrees per second, as fast as insects, enabled by fast neck muscles. In between saccades, gaze orientation is held constant. By comparing saccade and wingbeat phase, we find that these super-fast saccades are coordinated with the downstroke when the lateral visual field is occluded by the wings. Lovebirds thus maximize visual perception by overlying behaviors that impair vision, which helps coordinate maneuvers. Before the turn, lovebirds keep a high contrast edge in their visual midline. Similarly, before landing, the lovebirds stabilize the center of the perch in their visual midline. The perch on which the birds land swings, like a branch in the wind, and we find that retinal size of the perch is the most parsimonious visual cue to initiate landing. Our observations show that rapidly maneuvering birds use precisely timed stereotypic gaze behaviors consisting of rapid head turns and frontal feature stabilization, which facilitates optical flow based flight control. Similar gaze behaviors have been reported for visually navigating humans. This finding can inspire more effective vision-based autopilots for drones.     

‘Vector white noise’: a technique for mapping the motion receptive fields of direction-selective visual neurons

A technique is described and tested for mapping the sensitivities and preferred directions of motion at different locations within the receptive fields of direction-selective motion-detecting visual neurons. The procedure is to record the responses to a number of visual stimuli, each stimulus presentation consisting of a set of short, randomly-oriented, moving bars arranged in a square grid. Each bar moves perpendicularly to its long axis. The vector describing the sensitivity and preferred direction of motion at each grid location is obtained as a sum of the unit vectors defining the directions of motion of the bars in each of the stimuli at that location, weighted by the strengths of the corresponding responses. The resulting vector field specifies the optimum flow field for the neuron. The advantage of this technique over the conventional approach of probing the receptive field sequentially at each grid location is that the parallel nature of the stimulus is sensitive to nonlinear interactions (such as shunting inhibition for mutual facilitation) between different regions of the visual field. The technique is used to determine accurately the motion receptive fields of direction-selective motion detecting neurons in the optic lobes of insects. It is potentially applicable to motion-sensitive neurons with highly structured receptive fields, such as those in the optic tectum of the pigeon or in area MST of the monkey.     

Visual perception of surface curvature. The spin variation and its physiological implications

In the context of the models of structure from motion visual processing, we propose that the optic-flow field is a source of information for the perception of the curvature of a smooth surface in motion. In particular, it is shown how the spin variation (SV), a second spatial derivative of the retinal velocity field, is mathematically related to the curvature of the surface. Under the hypothesis that the visual system relies on SV to analyse the structure of a moving surface, a neural scheme for SV detection is proposed and psychophysical predictions are developed. Results obtained on artificial images show that the SV scheme presents a rather weak sensitivity to noise in conditions of low image velocity.     

Inverse perspective mapping simplifies optical flow computation and obstacle detection

We present a scheme for obstacle detection from optical flow which is based on strategies of biological information processing. Optical flow is established by a local voting (non-maximum suppression) over the outputs of correlation-type motion detectors similar to those found in the fly visual system. The computational theory of obstacle detection is discussed in terms of space-variances of the motion field. An efficient mechanism for the detection of disturbances in the expected motion field is based on inverse perspective mapping, i.e., a coordinate transform or retinotopic mapping applied to the image. It turns out that besides obstacle detection, inverse perspective mapping has additional advantages for regularizing optical flow algorithms. Psychophysical evidence for body-scaled obstacle detection and related neurophysiological results are discussed.     

Quantification of phase synchronization phenomena and their importance for verbal memory processes

The tangential neurons in the lobula plate region of the flies are known to respond to visual motion across broad receptive fields in visual space.When intracellular recordings are made from tangential neurons while the intact animal is stimulated visually with moving natural imagery,we find that neural response depends upon speed of motion but is nearly invariant with respect to variations in natural scenery. We refer to this invariance as velocity constancy. It is remarkable because natural scenes, in spite of similarities in spatial structure, vary considerably in contrast, and contrast dependence is a feature of neurons in the early visual pathway as well as of most models for the elementary operations of visual motion detection. Thus, we expect that operations must be present in the processing pathway that reduce contrast dependence in order to approximate velocity constancy.We consider models for such operations, including spatial filtering, motion adaptation, saturating nonlinearities, and nonlinear spatial integration by the tangential neurons themselves, and evaluate their effects in simulations of a tangential neuron and precursor processing in response to animated natural imagery. We conclude that all such features reduce interscene variance in response, but that the model system does not approach velocity constancy as closely as the biological tangential cell.     

Visual detection of paradoxical motion in flies

Summary From psychophysics it is known that humans easily perceive motion in Fourier-stimuli in which dots are displaced coherently into one direction. Furthermore, motion can be extracted from Drift-balanced stimuli in which the dots on average have no distinct direction of motion, or even in paradox -motion stimuli where the dots are displaced opposite to the perceived direction of motion. Whereas Fourier-motion can be explained by very basic motion detectors and nonlinear preprocessing of the input can account for the detection of Drift-balanced motion, a hierarchical model with two layers of motion detectors was proposed to explain the perception of -motion. The well described visual system of the fly allows to investigate whether these complex motion stimuli can be detected in a comparatively simple brain.The detection of such motion stimuli was analyzed for various random-dot cinematograms with extracellular recordings from the motion-sensitive Hl-neuron in the third visual ganglion of the blowfly Calliphora erythrocephala. The results were compared to computer-simulations of a hierarchical model of motion detector networks.For Fourier- and Drift-balanced motion stimuli, the Hl-neuron responds directionally selective to the moving object, whereas for -motion stimuli, the preferred direction is given by the dot displacement. Assuming nonlinear preprocessing of the detector input, such as a half-wave rectification, elementary motion detectors of the correlation type can account for these results.Abbreviations EMD elementary motion detector     

An image-interpolation technique for the computation of optic flow and egomotion

 A technique for measuring the motion of a rigid, textured plane in the frontoparallel plane is developed and tested on synthetic and real image sequences. The parameters of motion – translation in two dimensions, and rotation about a previously unspecified axis perpendicular to the plane – are computed by a single-stage, non-iterative process which interpolates the position of the moving image with respect to a set of reference images. The method can be extended to measure additional parameters of motion, such as expansion or shear. Advantages of the technique are that it does not require tracking of features, measurement of local image velocities or computation of high-order spatial or temporal derivatives of the image. The technique is robust to noise, and it offers a simple, novel way of tackling the ‘aperture’ problem. An application to the computation of robot egomotion is also described. Received: 3 September 1993/Accepted in revised form: 16 April 1994     

Diversity and survival of artificial lifeforms under sedimentation and random motion

Cellular automata are often used to explore the numerous possible scenarios of what could have occurred at the origins of life and before, during the prebiotic ages, when very simple molecules started to assemble and organise into larger catalytic or informative structures, or to simulate ecosystems. Artificial self-maintained spatial structures emerge in cellular automata and are often used to represent molecules or living organisms. They converge generally towards homogeneous stationary soups of still-life creatures. It is hard for an observer to believe they are similar to living systems, in particular because nothing is moving anymore within such simulated environments after few computation steps, because they present isotropic spatial organisation, because the diversity of self-maintained morphologies is poor, and because when stationary states are reached the creatures are immortal. Natural living systems, on the contrary, are composed of a high diversity of creatures in interaction having limited lifetimes and generally present a certain anisotropy of their spatial organisation, in particular frontiers and interfaces. In the present work, we propose that the presence of directional weak fields such as gravity may counter-balance the excess of mixing and disorder caused by Brownian motion and favour the appearance of specific regions, i.e. different strata or environmental layers, in which physical–chemical conditions favour the emergence and the survival of self-maintained spatial structures including living systems. We test this hypothesis by way of numerical simulations of a very simplified ecosystem model. We use the well-known Game of Life to which we add rules simulating both sedimentation forces and thermal agitation. We show that this leads to more active (vitality and biodiversity) and robust (survival) dynamics. This effectively suggests that coupling such physical processes to reactive systems allows the separation of environments into different milieux and could constitute a simple mechanism to form ecosystem frontiers or elementary interfaces that would protect and favour the development of fragile auto-poietic systems.     

Theory of spatial position and spatial frequency relations in the receptive fields of simple cells in the visual cortex

Striate cells showing linear spatial summation obey very general mathematical inequalities relating the size of their receptive fields to the corresponding spatial frequency and orientation tuning characteristics. The experimental data show that, in the preferred direction of stimulus motion, the spatial response profiles of cells in the simple family are well described by the mathematical form of Gabor elementary signals. The product of the uncertainties in signalling spatial position (x) and spatial frequency (f) has, therefore, a theoretical minimum value of xf=1/2. We examine the implications that these conclusions have for the relationship between the spatial response profiles of simple cells and the characteristics of their spatial frequency tuning curves. Examples of the spatial frequency tuning curves and their associated spatial response profiles are discussed and illustrated. The advantages for the operation of the visual system of different relationships between the spatial response profiles and the characteristics of the spatial frequency tuning curves are examined. Two examples are discussed in detail, one system having a constant receptive field size and the other a constant bandwidth.     

Small-signal analysis of a visual reflex in the locust

Measurement of isometric neck torque of the locust, in response to small sinusoidal motion of visual test patterns with large stripes, shows that displacements of 20 seconds of arc are perceived by the eye. On the other hand, when stripe size is varied, the eye seems not to resolve much detail since no response is elicited by patterns with spatial period less than 3°. It is shown that these two results are not incompatible.Current procedures for comparing geometrical interference phenomena in visual reflexes with the receptor spatial sampling relevant to motion perception are extended to treat the small-signal locust experiment, and shown in general to involve larger confidence limits than usually supposed. Especially, arbitrarily weighted contributions from several ommatidial pair-types in the hexagonal lattice are permissible. Finally, consideration of the effects of receptor and other series nonlinearities on motion-perception experiments of this kind predicts particular test patterns for which visual responses should depend upon phase relations of superposed Fourier spatial components, whether the events of receptor interaction involve correlation or not.Measured effects on the reflex of pattern luminance, contrast, displacement and spatial period form a basis for the small-signal frequency analysis described in the paper which follows this one.     

Differences in Visual-Spatial Input May Underlie Different Compression Properties of Firing Fields for Grid Cell Modules in Medial Entorhinal Cortex

Firing fields of grid cells in medial entorhinal cortex show compression or expansion after manipulations of the location of environmental barriers. This compression or expansion could be selective for individual grid cell modules with particular properties of spatial scaling. We present a model for differences in the response of modules to barrier location that arise from different mechanisms for the influence of visual features on the computation of location that drives grid cell firing patterns. These differences could arise from differences in the position of visual features within the visual field. When location was computed from the movement of visual features on the ground plane (optic flow) in the ventral visual field, this resulted in grid cell spatial firing that was not sensitive to barrier location in modules modeled with small spacing between grid cell firing fields. In contrast, when location was computed from static visual features on walls of barriers, i.e. in the more dorsal visual field, this resulted in grid cell spatial firing that compressed or expanded based on the barrier locations in modules modeled with large spacing between grid cell firing fields. This indicates that different grid cell modules might have differential properties for computing location based on visual cues, or the spatial radius of sensitivity to visual cues might differ between modules.     

Active vision in insects: an analysis of object-directed zig-zag flights in wasps (Odynerus spinipes , Eumenidae)

Ground-nesting wasps (Odynerus spinipes, Eumenidae) perform characteristic zig-zag flight manoeuvres when they encounter a novel object in the vicinity of their nests. We analysed flight parameters and flight control mechanisms and reconstructed the optical flow fields which the wasps generate by these flight manoeuvres. During zig-zag flights, the wasps move sideways and turn to keep the object in their frontal visual field. Their turning speed is controlled by the relative motion between object and background. We find that the wasps adjust their rotational and translational speed in such a way as to produce a specific vortex field of image motion that is centred on the novel object. As a result, differential image motion and changes in the direction of motion vectors are maximal in the vicinity and at the edges of the object. Zig-zag flights thus seem to be a `depth from motion' procedure for the extraction of object-related depth information. Accepted: 31 August 1997