Computation of optical motion by movement detectors

The first part of this paper deals with a system-theoretical approach for the decomposition of multi-input systems into the sum of simpler systems. This approach is applied here to analyse the algorithm which represents the computations underlying the extraction of motion information from the optical environment by biological movement detectors. The second part concentrates on a specific model for motion computation known to be realized by the visual system of insects and of man. These detectors provide the visual system with information on both the velocity and structural properties of a moving pattern. In the third part of this article the properties of two-dimensional arrays of movement detectors are analyzed and their relations to meaningful physiological responses are discussed.     

一种感知视觉运动信息的简化脑模型

视觉运动信息的感知过程,包括从局域运动检测到对模式整体运动的感知过程.我们以蝇视觉系统的图形-背景相对运动分辨的神经回路网络为基本框架,采用初级运动检测器的六角形阵列作为输入层,构造了一种感知视觉运动信息的简化脑模型,模拟了运动信息应该神经计算模型各个层次上的处理.该模型对差分行为实验结果作出了正确预测.本文并对空间生理整合的神经机制作了讨论.     

Figure-ground discrimination by relative movement in the visual system of the fly

The visual system of the fly performs various computations on photoreceptor outputs. The detection and measurement of movement is based on simple nonlinear multiplication-like interactions between adjacent pairs and groups of photoreceptors. The position of a small contrasted object against a uniform background is measured, at least in part, by (formally) 1-input nonlinear flicker detectors. A fly can also detect and discriminate a figure that moves relative to a ground texture. This computation of relative movement relies on a more complex algorithm, one which detects discontinuities in the movement field. The experiments described in this paper indicate that the outputs of neighbouring movement detectors interact in a multiplication-like fashion and then in turn inhibit locally the flicker detectors. The following main characteristic properties (partly a direct consequence of the algorithm's structure) have been established experimentally: a) Coherent motion of figure and ground inhibit the position detectors whereas incoherent motion fails to produce inhibition near the edges of the moving figure (provided the textures of figure and ground are similar). b) The movement detectors underlying this particular computation are direction-insensitive at input frequencies (at the photoreceptor level) above 2.3 Hz. They become increasingly direction-sensitive for lower input frequencies. c) At higher input frequencies the fly cannot discriminate an object against a texture oscillating at the same frequency and amplitude at 0° and 180° phase, whereas 90° or 270° phase shift between figure and ground oscillations yields maximum discrimination. d) Under conditions of coherent movement, strong spatial incoherence is detected by the same mechanism. The algorithm underlying the relative movement computation is further discussed as an example of a coherence measuring process, operating on the outputs of an array of movement detectors. Possible neural correlates are also mentioned.     

Properties of individual movement detectors as derived from behavioural experiments on the visual system of the fly

The performance of the fly's movement detection system is analysed using the visually induced yaw torque generated during tethered flight as a behavioural indicator. In earlier studies usually large parts of the visual field were exposed to the movement stimuli; the fly's response, therefore, represented the spatially pooled output signals of a large number of local movement detectors. Here we examined the responses of individual movement detectors. The stimulus pattern was presented to the fly via small vertical slits, thus, nearly avoiding spatial integration of local movement information along the horizontal axis of the eye. The stimulus consisted of a vertically oriented sine-wave grating which was moved with a constant velocity either clockwise or counterclockwise. In agreement with the theory of movement detectors of the correlation type, the time-course of the detector signal is modulated with the spatial phase of the stimulus pattern. It can even assume negative values for some time during the response cycle and thus signal the wrong direction of motion. By spatially integrating the response over sufficiently large arrays of movement detectors these response modulations disappear. Finally, one obtains a signal of the movement detection system which is constant while the pattern moves in one direction and only changes its sign when the pattern reverses its direction of motion. Spatial integration thus represents a simple means to obtain a meaningful representations of motion information.     

Processing of figure and background motion in the visual system of the fly

The visual system of the fly is able to extract different types of global retinal motion patterns as may be induced on the eyes during different flight maneuvers and to use this information to control visual orientation. The mechanisms underlying these tasks were analyzed by a combination of quantitative behavioral experiments on tethered flying flies (Musca domestica) and model simulations using different conditions of oscillatory large-field motion and relative motion of different segments of the stimulus pattern. Only torque responses about the vertical axis of the animal were determined. The stimulus patterns consisted of random dot textures (Julesz patterns) which could be moved either horizontally or vertically. Horizontal rotatory large-field motion leads to compensatory optomotor turning responses, which under natural conditions would tend to stabilize the retinal image. The response amplitude depends on the oscillation frequency: It is much larger at low oscillation frequencies than at high ones. When an object and its background move relative to each other, the object may, in principle, be discriminated and then induce turning responses of the fly towards the object. However, whether the object is distinguished by the fly depends not only on the phase relationship between object and background motion but also on the oscillation frequency. At all phase relations tested, the object is detected only at high oscillation frequencies. For the patterns used here, the turning responses are only affected by motion along the horizontal axis of the eye. No influences caused by vertical motion could be detected. The experimental data can be explained best by assuming two parallel control systems with different temporal and spatial integration properties: TheLF-system which is most sensitive to coherent rotatory large-field motion and mediates compensatory optomotor responses mainly at low oscillation frequencies. In contrast, theSF-system is tuned to small-field and relative motion and thus specialized to discriminate a moving object from its background; it mediates turning responses towards objects mainly at high oscillation frequencies. The principal organization of the neural networks underlying these control systems could be derived from the characteristic features of the responses to the different stimulus conditions. The input to the model circuits responsible for the characteristic sensitivity of the SF-system to small-field and relative motion is provided by retinotopic arrays of local movement detectors. The movement detectors are integrated by a large-field element, the output cell of the network. The synapses between the detectors and the output cells have nonlinear transmission characteristics. Another type of large-field elements (pool cells) which respond to motion in front of both eyes and have characteristic direction selectivities are assumed to interact with the local movement detector channels by inhibitory synapses of the shunting type, before the movement detectors are integrated by the output cells. The properties of the LF-system can be accounted for by similar model circuits which, however, differ with respect to the transmission characteristic of the synapses between the movement detectors and the output cell; moreover, their pool cells are only monocular. This type of network, however, is not necessary to account for the functional properties of the LF-system. Instead, intrinsic properties of single neurons may be sufficient. Computer simulations of the postulated mechanisms of the SF-and LF-system reveal that these can account for the specific features of the behavioral responses under quite different conditions of coherent large-field motion and relative motion of different pattern segments.     

A two dimensional field theory for motion computation

The local extraction of motion information from brightness patterns by individual movement detectors of the correlation-type is considered in the first part of the paper. A two-dimensional field theory of movement detection is developed by treating the distance between two adjacent photoreceptors as a differential. In the first approximation of the theory we only consider linear terms of the time interval between the reception of a contrast element and its delayed representation by the detector and linear terms of the spatial distances between adjacent photoreceptors. As a result we may neglect terms of higher order than quadratic in a Taylor series development of the brightness pattern. The responses of pairs of individual movement detectors are combined to a local response vector. In the first approximation of the detector field theory the response vector is proportional to the instantaneous pattern velocity vector and linearly dependent on local properties of the moving pattern. The linear dependence on pattern properties is represented by a two by two tensor consisting of elements which are nonlinear, local functional of the moving pattern. Some of the properties of the tensor elements are treated in detail. So, for instance, it is shown that the off-diagonal elements of the tensor disappear when the moving pattern consists of x- and y-dependent separable components. In the second part of the paper the tensor relation leading to the output of a movement detector pair is spatially integrated. The result of the integration is an approximation to a summation of the outputs of an array of detector pairs. The spatially integrated detector tensor relates the translatory motion vector to the resultant output vector. It is shown that the angle between the motion vector and the resultant output vector is always smaller than ±90° whereas the angle between the motion vector and local response vectors, elicited by detector pairs, may cover the entire angular range. In the discussion of the paper the limits of the field theory for motion computation as well as its higher approximations are pointed out in some detail. In a special chapter the dependence of the detector response on the pattern properties is treated and in another chapter questions connected with the so called aperture problem are discussed. Furthermore, properties for compensation of the pattern dependent deviation angle by spatial physiological integration are mentioned in the discussion.     

Dynamic response properties of movement detectors: Theoretical analysis and electrophysiological investigation in the visual system of the fly

Dynamic aspects of the computation of visual motion information are analysed both theoretically and experimentally. The theoretical analysis is based on the type of movement detector which has been proposed to be realized in the visual system of insects (e.g. Hassenstein and Reichardt 1956; Reichardt 1957, 1961; Buchner 1984), but also of man (e.g. van Doorn and Koenderink 1982a, b; van Santen and Sperling 1984; Wilson 1985). The output of both a single movement detector and a one-dimensional array of detectors is formulated mathematically as a function of time. The resulting movement detector theory can be applied to a much wider range of moving stimuli than has been possible on the basis of previous formulations of the detector output. These stimuli comprise one-dimensional smooth detector input functions, i.e. functions which can be expanded into a time-dependent convergent Taylor series for any value of the spatial coordinate.The movement detector response can be represented by a power series. Each term of this series consists of one exclusively time-dependent component and of another component that depends, in addition, on the properties of the pattern. Even the exclusively time-dependent components of the movement detector output are not solely determined by the stimulus velocity. They rather depend in a non-linear way on the weighted sum of the instantaneous velocity and all its higher order time derivatives. The latter point represents another reason — not discussed so far in the literature — that movement detectors of the type analysed here do not represent pure velocity sensors.The significance of this movement detector theory is established for the visual system of the fly. This is done by comparing the spatially integrated movement detector response with the functional properties of the directionally-selective motion-sensitive. Horizontal Cells of the third visual ganglion of the fly's brain.These integrate local motion information over large parts of the visual field. The time course of the spatially integrated movement detector response is about proportional to the velocity of the stimulus pattern only as long as the pattern velocity and its time derivatives are sufficiently small. For large velocities and velocity changes of the stimulus pattern characteristic deviations of the response profiles from being proportional to pattern velocity are predicted on the basis of the detector theory developed here. These deviations are clearly reflected in the response of the wide-field Horizontal Cells, thus, providing very specific evidence that the movement detector theory developed here can be applied to motion detection in the fly. The characteristic dynamic features of the theoretically predicted and the experimentally determined cellular responses are exploited to estimate the time constant of the movement detector filter.     

Correlation versus gradient type motion detectors: the pros and cons

Visual motion contains a wealth of information about self-motion as well as the three-dimensional structure of the environment. Therefore, it is of utmost importance for any organism with eyes. However, visual motion information is not explicitly represented at the photoreceptor level, but rather has to be computed by the nervous system from the changing retinal images as one of the first processing steps. Two prominent models have been proposed to account for this neural computation: the Reichardt detector and the gradient detector. While the Reichardt detector correlates the luminance levels derived from two adjacent image points, the gradient detector provides an estimate of the local retinal image velocity by dividing the spatial and the temporal luminance gradient. As a consequence of their different internal processing structure, both the models differ in a number of functional aspects such as their dependence on the spatial-pattern structure as well as their sensitivity to photon noise. These different properties lead to the proposal that an ideal motion detector should be of Reichardt type at low luminance levels, but of gradient type at high luminance levels. However, experiments on the fly visual systems provided unambiguous evidence in favour of the Reichardt detector under all luminance conditions. Does this mean that the fly nervous system uses suboptimal computations, or is there a functional aspect missing in the optimality criterion? In the following, I will argue in favour of the latter, showing that Reichardt detectors have an automatic gain control allowing them to dynamically adjust their input–output relationships to the statistical range of velocities presented, while gradient detectors do not have this property. As a consequence, Reichardt detectors, but not gradient detectors, always provide a maximum amount of information about stimulus velocity over a large range of velocities. This important property might explain why Reichardt type of computations have been demonstrated to underlie the extraction of motion information in the fly visual system under all luminance levels.     

Computational modelling of motion detectors: responses to two-frame displays

Schemes for motion detection fall into two classes. Reichardt correlators compare spatial luminance patterns at two locations at different times; gradient detectors compare spatial and temporal luminance gradients. Both are candidate operators for biological and machine vision systems. A large body of perceptual data exists, defining the properties of motion detectors used by human observers, which can form a basis for determining which class of detector is appropriate for the human visual system. Plausible versions of each detector were implemented, and their responses to a variety of two-frame stimuli were computed. Results indicated that both detectors can predict most of the data, but on balance gradient detectors offer the best working hypothesis for motion detection by human observers. This conclusion is necessarily limited to the type of stimuli used, and may require modification in the light of responses to continuously moving stimuli.     

Pattern-dependent response modulations in motion-sensitive visual interneurons--a model study

Even if a stimulus pattern moves at a constant velocity across the receptive field of motion-sensitive neurons, such as lobula plate tangential cells (LPTCs) of flies, the response amplitude modulates over time. The amplitude of these response modulations is related to local pattern properties of the moving retinal image. On the one hand, pattern-dependent response modulations have previously been interpreted as 'pattern-noise', because they deteriorate the neuron's ability to provide unambiguous velocity information. On the other hand, these modulations might also provide the system with valuable information about the textural properties of the environment. We analyzed the influence of the size and shape of receptive fields by simulations of four versions of LPTC models consisting of arrays of elementary motion detectors of the correlation type (EMDs). These models have previously been suggested to account for many aspects of LPTC response properties. Pattern-dependent response modulations decrease with an increasing number of EMDs included in the receptive field of the LPTC models, since spatial changes within the visual field are smoothed out by the summation of spatially displaced EMD responses. This effect depends on the shape of the receptive field, being the more pronounced--for a given total size--the more elongated the receptive field is along the direction of motion. Large elongated receptive fields improve the quality of velocity signals. However, if motion signals need to be localized the velocity coding is only poor but the signal provides--potentially useful--local pattern information. These modelling results suggest that motion vision by correlation type movement detectors is subject to uncertainty: you cannot obtain both an unambiguous and a localized velocity signal from the output of a single cell. Hence, the size and shape of receptive fields of motion sensitive neurons should be matched to their potential computational task.     

Fly-like visuomotor responses of a robot using aVLSI motion-sensitive chips

 We explore the use of continuous-time analog very-large-scale-integrated (aVLSI) neuromorphic visual preprocessors together with a robotic platform in generating bio-inspired behaviors. Both the aVLSI motion sensors and the robot behaviors described in this work are inspired by the motion computation in the fly visual system and two different fly behaviors. In most robotic systems, the visual information comes from serially scanned imagers. This restricts the form of computation of the visual image and slows down the input rate to the controller system of the robot, hence increasing the reaction time of the robot. These aVLSI neuromorphic sensors reduce the computational load and power consumption of the robot, thus making it possible to explore continuous-time visuomotor control systems that react in real-time to the environment. The motion sensor provides two outputs: one for the preferred direction and the other for the null direction. These motion outputs are created from the aggregation of six elementary motion detectors that implement a variant of Reichardt's correlation algorithm. The four analog continuous-time outputs from the motion chips go to the control system on the robot which generates a mixture of two behaviors – course stabilization and fixation – from the outputs of these sensors. Since there are only four outputs, the amount of information transmitted to the controller is reduced (as compared to using a CCD sensor), and the reaction time of the robot is greatly decreased. In this work, the robot samples the motion sensors every 3.3 ms during the behavioral experiments. Received: 4 October 1999 / Accepted in revised form: 26 April 2001     

Speed encoding in correlation motion detectors as a consequence of spatial structure

For animals to carry out a wide range of detection, recognition and navigation tasks, visual motion signals are crucial. The encoding of motion information has therefore, attracted much attention in the experimental and computational study of brain function. Two main alternative mechanisms have been proposed on the basis of behavioural and physiological experiments. On one hand, correlation-type and motion energy detectors are simple and efficient in the design of their basic mechanism but are tuned to temporal frequency rather than to speed. On other hand, gradient-type motion detectors directly represent an estimate of speed, but may require more demanding processing mechanisms. We demonstrate here how the temporal frequency dependence observed for sine-wave gratings can disappear for less constrained stimuli, to be replaced by responses reflecting speed for stimuli like square waves when a phase-sensitive detection mechanism is employed. We conclude from these observations that temporal frequency tuning is not necessarily a limitation for motion vision based on correlation detectors, and more generally demonstrate in view of the typical Fourier composition of natural scenes, that correlation detectors operating in such environments can encode image speed. In the context of our results, we discuss the implications of the loss of phase sensitivity inherent in using a linear system approach to describe neural processing.     

Propagation of photon noise and information transfer in visual motion detection

The extraction of the direction of motion from the time varying retinal images is one of the most basic tasks any visual system is confronted with. However, retinal images are severely corrupted by photon noise, in particular at low light levels, thus limiting the performance of motion detection mechanisms of what sort so ever. Here, we study how photon noise propagates through an array of Reichardt-type motion detectors that are commonly believed to underlie fly motion vision. We provide closed-form analytical expressions of the signal and noise spectra at the output of such a motion detector array. We find that Reichardt detectors reveal favorable noise suppression in the frequency range where most of the signal power resides. Most notably, due to inherent adaptive properties, the transmitted information about stimulus velocity remains nearly constant over a large range of velocity entropies. Action editor: Matthew Wiener     

Temporal modulation of luminance adapts time constant of fly movement detectors

The time constant of movement detectors in the fly visual system has been proposed to adapt in response to moving stimuli (de Ruyter van Steveninck et al. 1986). The objective of the present study is to analyse, whether this adaptation can be induced as well, if the luminance of a stationary uniform field is modulated in time. The experiments were done on motion-sensitive wide-field neurones of the lobula plate, the posterior part of the third visual ganglion of the blowfly, calliphora erythrocephala. These cells are assumed to receive input from large retinotopic arrays of movement detectors. In order to demonstrate that our results concern the properties of the movement detectors rather than those of a particular wide-field cell we recorded from two different types of them, the H1- and the HSE-cell. Both cell types respond to a brief movement stimulus in their preferred direction with a transient excitation. This response decays about exponentially. The time constant of this decay reflects, in a first approximation, the time constant of the presynaptic movement detectors. It was determined after prestimulation of the cell by the following stimuli: (a) periodic stationary grating; (b) uniform field, the intensity of which was modulated sinusoidally in time (flicker stimulation); (c) periodic grating moving front-to-back; (d) periodic grating moving back-to-front. The decay of the response is significantly faster not only after movement but also after flicker stimulation as compared with pre-stimulation with a stationary stimulus. This is interpreted as an adaptation of the movement detector's time constant. The finding that flicker stimulation also leads to an adaptation shows that movement is not necessary for this process. Instead the adaptation of the time constant appears to be governed mainly by the temporal modulation (i.e., contrast frequency) of the signal in each visual channel.     

The motion aftereffect as a universal phenomenon in sensory systems involved in spatial orientation. III. Aftereffect of motion adaptation in the somatosensory and vestibular systems

The motion aftereffect may be considered as a consequence of visual illusions of self-motion (vection) and the persistence of sensory information processing. There is ample experimental evidence indicating a uniformity of mechanisms that underlie motion aftereffects in different modalities based on the principle of motion detectors. Currently, there is firm ground to believe that the motion aftereffect is intrinsic to all sensory systems involved in spatial orientation, that motion adaptation in one sensory system elicits changes in another one, and that such adaptation is of great adaptive importance for spatial orientation and motion of an organism. This review seeks to substantiate these ideas.     

蝇视系统的图形—背景分辨和初级模式分辨

本文通过行为实验及计算机模拟进一步证明,蝇视系统的自发模式辨别可以看作是图形—背景分辨的特殊情况.关键在于蝇的模式分辨是由运动检测器实现的.运动检测器不仅对模式速度反应,也对模式的结构特性反应.本文提出,人视系统的模式分辨也可能部分地由运动检测器来实现.     

A Laterally Interconnected Neural Architecture in MST Accounts for Psychophysical Discrimination of Complex Motion Patterns

The complex patterns of visual motion formed across the retina during self-motion, often referred to as optic flow, provide a rich source of information describing our dynamic relationship within the environment. Psychophysical studies indicate the existence of specialized detectors for component motion patterns (radial, circular, planar) that are consistent with the visual motion properties of cells in the medial superior temporal area (MST) of nonhuman primates. Here we use computational modeling and psychophysics to investigate the structural and functional role of these specialized detectors in performing a graded motion pattern (GMP) discrimination task. In the psychophysical task perceptual discrimination varied significantly with the type of motion pattern presented, suggesting perceptual correlates to the preferred motion bias reported in MST. Simulated perceptual discrimination in a population of independent MST-like neural responses showed inconsistent psychophysical performance that varied as a function of the visual motion properties within the population code. Robust psychophysical performance was achieved by fully interconnecting neural populations such that they inhibited nonpreferred units. Taken together, these results suggest that robust processing of the complex motion patterns associated with self-motion and optic flow may be mediated by an inhibitory structure of neural interactions in MST.     

Optic flow representation in the optic lobes of Diptera: modeling the role of T5 directional tuning properties

An evolutionarily conserved system of small retinotopic neurons in dipteran insects, called bushy T-cells, provides information about directional motion to large collator neurons in the lobula plate. Physiological and anatomical features of these cells provide the basis for a model that is used to investigate requirements for generating optic flow selectivity in collators while allowing for evolutionary variations. This account focuses on the role of physiological tuning properties of T5 neurons. Various flow fields are defined as inputs to retinotopic arrays of T5 cells, the responses of which are mapped onto collators using innervation matrices that promote selectivity for flow type and position. Properties known or inferred from physiological and anatomical studies of neurons contributing to motion detection are incorporated into the model: broad tuning to local motion direction and the representation of each visual sampling unit by a quartet of small-field T5-like neurons with orthogonal preferred directions. The model predicts hitherto untested response properties of optic flow selective collators, and predicts that selectivity for a given flow field can be highly sensitive to perturbations in physiological properties of the motion detectors.     

Looking at Op Art from a computational viewpoint

Arts history tells an exciting story about repeated attempts to represent features that are crucial for the understanding of our environment and which, at the same time, go beyond the inherently two-dimensional nature of a flat painting surface: depth and motion. In the twentieth century, Op artists such as Bridget Riley began to experiment with simple black and white patterns that do not represent motion in an artistic way but actually create vivid dynamic illusions in static pictures. The cause of motion illusions in such paintings is still a matter of debate. The role of involuntary eye movements in this phenomenon is studied here with a computational approach. The possible consequences of shifting the retinal image of synthetic wave gratings, dubbed as 'riloids', were analysed by a two-dimensional array of motion detectors (2DMD model), which generates response maps representing the spatial distribution of motion signals generated by such a stimulus. For a two-frame sequence reflecting a saccadic displacement, these motion signal maps contain extended patches in which local directions change only little. These directions, however, do not usually precisely correspond to the direction of pattern displacement that can be expected from the geometry of the curved gratings as an instance of the so-called 'aperture problem'. The patchy structure of the simulated motion detector response to the displacement of riloids resembles the motion illusion, which is not perceived as a coherent shift of the whole pattern but as a wobbling and jazzing of ill-defined regions. Although other explanations are not excluded, this might support the view that the puzzle of Op Art motion illusions could potentially have an almost trivial solution in terms of small involuntary eye movement leading to image shifts that are picked up by well-known motion detectors in the early visual system. This view can have further consequences for our understanding of how the human visual system usually compensates for eye movements, in order to let us perceive a stable world despite continuous image shifts generated by gaze instability.     

Manuo-ocular coordination in target tracking. II. Comparing the model with human behavior

Several studies have shown that humans track a moving visual target with their eyes better if the movement of this target is directly controlled by the observer's hand. The improvement in performance has been attributed to coordination control between the arm motor system and the smooth pursuit (SP) system. In such a task, the SP system shows characteristics that differ from those observed during eye-alone tracking: latency (between the target-arm and the eye motion onsets) is shorter, maximum SP velocity is higher and the maximum target motion frequency at which the SP can function effectively is also higher. The aim of this article is to qualitatively evaluate the behavior of a dynamical model simulating the oculomotor system and the arm motor system when both are involved in tracking visual targets. The evaluation is essentially based on a comparison of the behavior of the model with the behavior of human subjects tracking visual targets under different conditions. The model has been introduced and quantitatively evaluated in a companion paper. The model is based on an exchange of internal information between the two sensorimotor systems, mediated by sensory signals (vision, arm muscle proprioception) and motor signals (arm motor command copy). The exchange is achieved by a specialized structure of the central nervous system, previously identified as a part of the cerebellum. Computer simulation of the model yielded results that fit the behavior of human subjects observed during previously reported experiments, both qualitatively and quantitatively. The parallelism between physiology and human behavior on the one hand, and structure and simulation of the model on the other hand, is discussed. Received: 6 March 1997 / Accepted in revised form: 15 July 1997