Real-time unconstrained object recognition: a processing pipeline based on the mammalian visual system

The mammalian visual system is still the gold standard for recognition accuracy, flexibility, efficiency, and speed. Ongoing advances in our understanding of function and mechanisms in the visual system can now be leveraged to pursue the design of computer vision architectures that will revolutionize the state of the art in computer vision.     

Active manual control of object views facilitates visual recognition

Active exploration of large-scale environments leads to better learning of spatial layout than does passive observation [1] [2] [3]. But active exploration might also help us to remember the appearance of individual objects in a scene. In fact, when we encounter new objects, we often manipulate them so that they can be seen from a variety of perspectives. We present here the first evidence that active control of the visual input in this way facilitates later recognition of objects. Observers who actively rotated novel, three-dimensional objects on a computer screen later showed more efficient visual recognition than observers who passively viewed the exact same sequence of images of these virtual objects. During active exploration, the observers focused mainly on the 'side' or 'front' views of the objects (see also [4] [5] [6]). The results demonstrate that how an object is represented for later recognition is influenced by whether or not one controls the presentation of visual input during learning.     

Making connections in the fly visual system

Understanding the molecular mechanisms that regulate formation of precise patterns of neuronal connections within the central nervous system remains a challenging problem in neurobiology. Genetic studies in worms and flies and molecular studies in vertebrate systems have led to an increasingly sophisticated understanding of how growth cones navigate toward their targets and form topographic maps. Considerably less is known about how growth cones recognize their cellular targets and form synapses with them. Here, we review connection formation in the fly visual system, the methodological approaches used to study it, and recent progress in uncovering the molecular basis of connection specificity.     

fMRI reveals how pain modulates visual object processing in the ventral visual stream

It is well known that pain attracts attention and interferes with cognition. Given that the mechanisms behind this phenomenon are largely unknown, we used functional magnetic resonance imaging and presented visual objects with or without concomitant pain stimuli. To test for the specificity of pain, we compared this modulatory effect with a previously established modulatory effect of working memory on visual object processing. Our data showed a comparable behavioral effect of both types of modulation and identified the lateral occipital complex (LOC) as the site of modulation in the ventral visual stream, for both pain and working memory. However, the sources of these modulatory effects differed for the two processes. Whereas the source of modulation for working memory could be attributed to the parietal cortex, the modulatory effect of pain was observed in the rostral anterior cingulate cortex (rACC), an area ideally suited to link pain perception and attentional control.     

Calcium-dependent control of temporal processing in an auditory interneuron: a computational analysis

Sensitivity to acoustic amplitude modulation in crickets differs between species and depends on carrier frequency (e.g., calling song vs. bat-ultrasound bands). Using computational tools, we explore how Ca²⁺-dependent mechanisms underlying selective attention can contribute to such differences in amplitude modulation sensitivity. For omega neuron 1 (ON1), selective attention is mediated by Ca²⁺-dependent feedback: [Ca²⁺]_internal increases with excitation, activating a Ca²⁺-dependent after-hyperpolarizing current. We propose that Ca²⁺ removal rate and the size of the after-hyperpolarizing current can determine ON1’s temporal modulation transfer function (TMTF). This is tested using a conductance-based simulation calibrated to responses in vivo. The model shows that parameter values that simulate responses to single pulses are sufficient in simulating responses to modulated stimuli: no special modulation-sensitive mechanisms are necessary, as high and low-pass portions of the TMTF are due to Ca²⁺-dependent spike frequency adaptation and post-synaptic potential depression, respectively. Furthermore, variance in the two biophysical parameters is sufficient to produce TMTFs of varying bandwidth, shifting amplitude modulation sensitivity like that in different species and in response to different carrier frequencies. Thus, the hypothesis that the size of after-hyperpolarizing current and the rate of Ca²⁺ removal can affect amplitude modulation sensitivity is computationally validated.     

Dynamic response properties of movement detectors: Theoretical analysis and electrophysiological investigation in the visual system of the fly

Dynamic aspects of the computation of visual motion information are analysed both theoretically and experimentally. The theoretical analysis is based on the type of movement detector which has been proposed to be realized in the visual system of insects (e.g. Hassenstein and Reichardt 1956; Reichardt 1957, 1961; Buchner 1984), but also of man (e.g. van Doorn and Koenderink 1982a, b; van Santen and Sperling 1984; Wilson 1985). The output of both a single movement detector and a one-dimensional array of detectors is formulated mathematically as a function of time. The resulting movement detector theory can be applied to a much wider range of moving stimuli than has been possible on the basis of previous formulations of the detector output. These stimuli comprise one-dimensional smooth detector input functions, i.e. functions which can be expanded into a time-dependent convergent Taylor series for any value of the spatial coordinate.The movement detector response can be represented by a power series. Each term of this series consists of one exclusively time-dependent component and of another component that depends, in addition, on the properties of the pattern. Even the exclusively time-dependent components of the movement detector output are not solely determined by the stimulus velocity. They rather depend in a non-linear way on the weighted sum of the instantaneous velocity and all its higher order time derivatives. The latter point represents another reason — not discussed so far in the literature — that movement detectors of the type analysed here do not represent pure velocity sensors.The significance of this movement detector theory is established for the visual system of the fly. This is done by comparing the spatially integrated movement detector response with the functional properties of the directionally-selective motion-sensitive. Horizontal Cells of the third visual ganglion of the fly's brain.These integrate local motion information over large parts of the visual field. The time course of the spatially integrated movement detector response is about proportional to the velocity of the stimulus pattern only as long as the pattern velocity and its time derivatives are sufficiently small. For large velocities and velocity changes of the stimulus pattern characteristic deviations of the response profiles from being proportional to pattern velocity are predicted on the basis of the detector theory developed here. These deviations are clearly reflected in the response of the wide-field Horizontal Cells, thus, providing very specific evidence that the movement detector theory developed here can be applied to motion detection in the fly. The characteristic dynamic features of the theoretically predicted and the experimentally determined cellular responses are exploited to estimate the time constant of the movement detector filter.     

Efficiency of a visual system under conditions of uncertainty of the appearance of a moving object

The effect of uncertainty of a moving object appearance in the noise field upon the coefficient efficiency, we studied. At short durations of presentation (40-80 msec) and high level of external noise, this effect was maximal: magnified 100 times. The efficiency coefficient dependence on the duration of a moving object presentation was shown to be characterized by two maximums. The position of minimum situated between these two maximums was found to be independent of either presence or absence of uncertainty of a number of parameters: such as initial position of the object the image, time of its appearance, noise level, velocity and direction of the movement, and has a latency approximately 120 sec. A functional model of the observed phenomena, has been proposed.     

The Drosophila larval visual system: high-resolution analysis of a simple visual neuropil

The task of the visual system is to translate light into neuronal encoded information. This translation of photons into neuronal signals is achieved by photoreceptor neurons (PRs), specialized sensory neurons, located in the eye. Upon perception of light the PRs will send a signal to target neurons, which represent a first station of visual processing. Increasing complexity of visual processing stems from the number of distinct PR subtypes and their various types of target neurons that are contacted. The visual system of the fruit fly larva represents a simple visual system (larval optic neuropil, LON) that consists of 12 PRs falling into two classes: blue-senstive PRs expressing Rhodopsin 5 (Rh5) and green-sensitive PRs expressing Rhodopsin 6 (Rh6). These afferents contact a small number of target neurons, including optic lobe pioneers (OLPs) and lateral clock neurons (LNs). We combine the use of genetic markers to label both PR subtypes and the distinct, identifiable sets of target neurons with a serial EM reconstruction to generate a high-resolution map of the larval optic neuropil. We find that the larval optic neuropil shows a clear bipartite organization consisting of one domain innervated by PRs and one devoid of PR axons. The topology of PR projections, in particular the relationship between Rh5 and Rh6 afferents, is maintained from the nerve entering the brain to the axon terminals. The target neurons can be subdivided according to neurotransmitter or neuropeptide they use as well as the location within the brain. We further track the larval optic neuropil through development from first larval instar to its location in the adult brain as the accessory medulla.     

Figure-ground discrimination by relative movement in the visual system of the fly

The visual system of the fly performs various computations on photoreceptor outputs. The detection and measurement of movement is based on simple nonlinear multiplication-like interactions between adjacent pairs and groups of photoreceptors. The position of a small contrasted object against a uniform background is measured, at least in part, by (formally) 1-input nonlinear flicker detectors. A fly can also detect and discriminate a figure that moves relative to a ground texture. This computation of relative movement relies on a more complex algorithm, one which detects discontinuities in the movement field. The experiments described in this paper indicate that the outputs of neighbouring movement detectors interact in a multiplication-like fashion and then in turn inhibit locally the flicker detectors. The following main characteristic properties (partly a direct consequence of the algorithm's structure) have been established experimentally: a) Coherent motion of figure and ground inhibit the position detectors whereas incoherent motion fails to produce inhibition near the edges of the moving figure (provided the textures of figure and ground are similar). b) The movement detectors underlying this particular computation are direction-insensitive at input frequencies (at the photoreceptor level) above 2.3 Hz. They become increasingly direction-sensitive for lower input frequencies. c) At higher input frequencies the fly cannot discriminate an object against a texture oscillating at the same frequency and amplitude at 0° and 180° phase, whereas 90° or 270° phase shift between figure and ground oscillations yields maximum discrimination. d) Under conditions of coherent movement, strong spatial incoherence is detected by the same mechanism. The algorithm underlying the relative movement computation is further discussed as an example of a coherence measuring process, operating on the outputs of an array of movement detectors. Possible neural correlates are also mentioned.     

A visual model for object detection based on active contours and level-set method

A visual model for object detection is proposed. In order to make the detection ability comparable with existing technical methods for object detection, an evolution equation of neurons in the model is derived from the computational principle of active contours. The hierarchical structure of the model emerges naturally from the evolution equation. One drawback involved with initial values of active contours is alleviated by introducing and formulating convexity, which is a visual property. Numerical experiments show that the proposed model detects objects with complex topologies and that it is tolerant of noise. A visual attention model is introduced into the proposed model. Other simulations show that the visual properties of the model are consistent with the results of psychological experiments that disclose the relation between figure–ground reversal and visual attention. We also demonstrate that the model tends to perceive smaller regions as figures, which is a characteristic observed in human visual perception.This work was partially supported by Grants-in-Aid for Scientific Research (#14780254) from Japan Society of Promotion of Science.     

Learning invariant object recognition in the visual system with continuous transformations

The cerebral cortex utilizes spatiotemporal continuity in the world to help build invariant representations. In vision, these might be representations of objects. The temporal continuity typical of objects has been used in an associative learning rule with a short-term memory trace to help build invariant object representations. In this paper, we show that spatial continuity can also provide a basis for helping a system to self-organize invariant representations. We introduce a new learning paradigm “continuous transformation learning” which operates by mapping spatially similar input patterns to the same postsynaptic neurons in a competitive learning system. As the inputs move through the space of possible continuous transforms (e.g. translation, rotation, etc.), the active synapses are modified onto the set of postsynaptic neurons. Because other transforms of the same stimulus overlap with previously learned exemplars, a common set of postsynaptic neurons is activated by the new transforms, and learning of the new active inputs onto the same postsynaptic neurons is facilitated. We demonstrate that a hierarchical model of cortical processing in the ventral visual system can be trained with continuous transform learning, and highlight differences in the learning of invariant representations to those achieved by trace learning.     

Processing of figure and background motion in the visual system of the fly

The visual system of the fly is able to extract different types of global retinal motion patterns as may be induced on the eyes during different flight maneuvers and to use this information to control visual orientation. The mechanisms underlying these tasks were analyzed by a combination of quantitative behavioral experiments on tethered flying flies (Musca domestica) and model simulations using different conditions of oscillatory large-field motion and relative motion of different segments of the stimulus pattern. Only torque responses about the vertical axis of the animal were determined. The stimulus patterns consisted of random dot textures (Julesz patterns) which could be moved either horizontally or vertically. Horizontal rotatory large-field motion leads to compensatory optomotor turning responses, which under natural conditions would tend to stabilize the retinal image. The response amplitude depends on the oscillation frequency: It is much larger at low oscillation frequencies than at high ones. When an object and its background move relative to each other, the object may, in principle, be discriminated and then induce turning responses of the fly towards the object. However, whether the object is distinguished by the fly depends not only on the phase relationship between object and background motion but also on the oscillation frequency. At all phase relations tested, the object is detected only at high oscillation frequencies. For the patterns used here, the turning responses are only affected by motion along the horizontal axis of the eye. No influences caused by vertical motion could be detected. The experimental data can be explained best by assuming two parallel control systems with different temporal and spatial integration properties: TheLF-system which is most sensitive to coherent rotatory large-field motion and mediates compensatory optomotor responses mainly at low oscillation frequencies. In contrast, theSF-system is tuned to small-field and relative motion and thus specialized to discriminate a moving object from its background; it mediates turning responses towards objects mainly at high oscillation frequencies. The principal organization of the neural networks underlying these control systems could be derived from the characteristic features of the responses to the different stimulus conditions. The input to the model circuits responsible for the characteristic sensitivity of the SF-system to small-field and relative motion is provided by retinotopic arrays of local movement detectors. The movement detectors are integrated by a large-field element, the output cell of the network. The synapses between the detectors and the output cells have nonlinear transmission characteristics. Another type of large-field elements (pool cells) which respond to motion in front of both eyes and have characteristic direction selectivities are assumed to interact with the local movement detector channels by inhibitory synapses of the shunting type, before the movement detectors are integrated by the output cells. The properties of the LF-system can be accounted for by similar model circuits which, however, differ with respect to the transmission characteristic of the synapses between the movement detectors and the output cell; moreover, their pool cells are only monocular. This type of network, however, is not necessary to account for the functional properties of the LF-system. Instead, intrinsic properties of single neurons may be sufficient. Computer simulations of the postulated mechanisms of the SF-and LF-system reveal that these can account for the specific features of the behavioral responses under quite different conditions of coherent large-field motion and relative motion of different pattern segments.     

Encoding of motion in real time by the fly visual system.

Direction-selective cells in the fly visual system that have large receptive fields play a decisive role in encoding the time-dependent optic flow the animal encounters during locomotion. Recent experiments on the computations performed by these cells have highlighted the significance of dendritic integration and have addressed the role of spikes versus graded membrane potential changes in encoding optic flow information. It is becoming increasingly clear that the way optic flow is encoded in real time is constrained both by the computational needs of the animal in visually guided behaviour as well as by the specific properties of the underlying neuronal hardware.     

A special class of nonlinear interactions in the visual system of the fly

Flies can detect a small object in front of a randomly contrasted background if the object undergoes small motions. The effect was investigated in fixed flying flies under open-loop conditions. The results suggest that nonlinear inhibitory interactions underly this elementary case of figure-ground discrimination.