Three-dimensional displays and stereo vision

Procedures for three-dimensional image reconstruction that are based on the optical and neural apparatus of human stereoscopic vision have to be designed to work in conjunction with it. The principal methods of implementing stereo displays are described. Properties of the human visual system are outlined as they relate to depth discrimination capabilities and achieving optimal performance in stereo tasks. The concept of depth rendition is introduced to define the change in the parameters of three-dimensional configurations for cases in which the physical disposition of the stereo camera with respect to the viewed object differs from that of the observer's eyes.     

Leaving flatland: Advances in 3D behavioral measurement

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.     

Stereo super-resolution images detection based on multi-scale feature extraction and hierarchical feature fusion

Recently, with most mobile phones coming with dual cameras, stereo image super-resolution is becoming increasingly popular in phones and other modern acquisition devices, leading stereo super-resolution images spread widely on the Internet. However, current image forensics methods are carried out in monocular images, and high false positive rate appears when detecting stereo super-resolution images by these methods. Therefore, it is important to develop stereo super-resolution image detection method. In this paper, a convolutional neural network with multi-scale feature extraction and hierarchical feature fusion is proposed to detect the stereo super-resolution images. Multi-atrous convolutions are employed to extract multi-scale features and be adapt for varying stereo super-resolution images, and hierarchical feature fusion further improve the performance and robustness of the model. Experimental results demonstrate that the proposed network can detect stereo super-resolution images effectively and achieve strong generalization and robustness. To the best of our knowledge, it is the first attempt to investigate the performance of current forensics methods when tested under stereo super-resolution images, and represent the first study of stereo super-resolution images detection. We believe that it can raise the awareness about the security of stereo super-resolution images.     

Computational studies of the extraction of visual spatial information from binocular and motion cues

This paper reviews some of the contributions that work in computational vision has made to the study of biological vision systems. We concentrate on two areas where there has been strong interaction between computational and experimental studies: the use of binocular stereo to recover the distances to surfaces in space, and the recovery of the three-dimensional shape of objects from relative motion in the image. With regard to stereo, we consider models proposed for solving the stereo correspondence problem, focussing on the way in which physical properties of the world constrain possible methods of solution. We also show how critical observations regarding human stereo vision have helped to shape these models. With regard to the recovery of structure from motion, we focus on how the constraint of object rigidity has been used in computational models of this process.     

Using digital images to reconstruct three‐dimensional biological forms: a new tool for morphological studies

We examined two image‐based methods, photogrammetry and stereo vision, used for reconstructing the three‐dimensional form of biological organisms under field conditions. We also developed and tested a third ‘hybrid’ method, which combines the other two techniques. We tested these three methodologies using two different cameras to obtain digital images of museum and field sampled specimens of giant tortoises. Both the precision and repeatability of the methods were assessed statistically on the same specimens by comparing geodesic and Euclidean measurements made on the digital models with linear measurements obtained with caliper and flexible tape. We found no substantial difference between the three methods in measuring the Euclidean distance between landmarks, but spatially denser models (stereo vision and ‘hybrid’) were more accurate for geodesic distances. The use of different digital cameras did not influence the results. Image‐based methods require only inexpensive instruments and appropriate software, and allow reconstruction of the three‐dimensional forms (including their curved surfaces) of organisms sampled in the field. © 2008 The Linnean Society of London, Biological Journal of the Linnean Society, 2008, 95 , 425–436.     

Manuo-ocular coordination in target tracking. II. Comparing the model with human behavior

Several studies have shown that humans track a moving visual target with their eyes better if the movement of this target is directly controlled by the observer's hand. The improvement in performance has been attributed to coordination control between the arm motor system and the smooth pursuit (SP) system. In such a task, the SP system shows characteristics that differ from those observed during eye-alone tracking: latency (between the target-arm and the eye motion onsets) is shorter, maximum SP velocity is higher and the maximum target motion frequency at which the SP can function effectively is also higher. The aim of this article is to qualitatively evaluate the behavior of a dynamical model simulating the oculomotor system and the arm motor system when both are involved in tracking visual targets. The evaluation is essentially based on a comparison of the behavior of the model with the behavior of human subjects tracking visual targets under different conditions. The model has been introduced and quantitatively evaluated in a companion paper. The model is based on an exchange of internal information between the two sensorimotor systems, mediated by sensory signals (vision, arm muscle proprioception) and motor signals (arm motor command copy). The exchange is achieved by a specialized structure of the central nervous system, previously identified as a part of the cerebellum. Computer simulation of the model yielded results that fit the behavior of human subjects observed during previously reported experiments, both qualitatively and quantitatively. The parallelism between physiology and human behavior on the one hand, and structure and simulation of the model on the other hand, is discussed. Received: 6 March 1997 / Accepted in revised form: 15 July 1997     

Quality Index for Stereoscopic Images by Separately Evaluating Adding and Subtracting

The human visual system (HVS) plays an important role in stereo image quality perception. Therefore, it has aroused many people’s interest in how to take advantage of the knowledge of the visual perception in image quality assessment models. This paper proposes a full-reference metric for quality assessment of stereoscopic images based on the binocular difference channel and binocular summation channel. For a stereo pair, the binocular summation map and binocular difference map are computed first by adding and subtracting the left image and right image. Then the binocular summation is decoupled into two parts, namely additive impairments and detail losses. The quality of binocular summation is obtained as the adaptive combination of the quality of detail losses and additive impairments. The quality of binocular summation is computed by using the Contrast Sensitivity Function (CSF) and weighted multi-scale (MS-SSIM). Finally, the quality of binocular summation and binocular difference is integrated into an overall quality index. The experimental results indicate that compared with existing metrics, the proposed metric is highly consistent with the subjective quality assessment and is a robust measure. The result have also indirectly proved hypothesis of the existence of binocular summation and binocular difference channels.     

Manuo-ocular coordination in target tracking. I. A model simulating human performance

During eye tracking of a self-moved target, human subjects' performance differs from eye-alone tracking of an external target. Typical latency between target and eye motion onsets is shorter, ocular smooth pursuit (SP) saturation velocity increases and the maximum target motion frequency at which the SP system functions correctly is higher. Based on a previous qualitative model, a quantitative model of the coordination control between the arm motor system and the SP system is presented and evaluated here. The model structure maintains a high level of parallelism with the physiological system. It contains three main parts: the eye motor control (containing a SP branch and a saccadic branch), the arm motor control and the coordination control. The coordination control is achieved via an exchange of information between the arm and the eye sensorimotor systems, mediated by sensory signals (vision, proprioception) and motor command copy. This cross-talk results in improved SP system performance. The model has been computer simulated and the results have been compared with human subjects' behavior observed during previous experiments. The model performance is seen to quantitatively fit data on human subjects. Received: 6 March 1997 / Accepted in revised form: 15 July 1997     

基于视觉编码的图象处理研究──Ⅰ．原理、算法及实现

提出Ｇａｂｏｒ小波表达的概念及其在图象处理中实现的数学模型和算法框架,主要解决了运算中非正交性问题、收敛性问题,使得该算法及模型可以实现图象的分析和重建,并且可以应用于对视觉编码理论的解释。     

基于视觉编码的图象处理研究──Ⅰ．原理、算法及实现

提出Ｇａｂｏｒ小波表达的概念及其在图象处理中实现的数学模型和算法框架,主要解决了运算中非正交性问题、收敛性问题,使得该算法及模型可以实现图象的分解和重建,并且可以应用于对视觉编码理论的解释。     

Stereoscopic Vision in the Absence of the Lateral Occipital Cortex

Both dorsal and ventral cortical visual streams contain neurons sensitive to binocular disparities, but the two streams may underlie different aspects of stereoscopic vision. Here we investigate stereopsis in the neurological patient D.F., whose ventral stream, specifically lateral occipital cortex, has been damaged bilaterally, causing profound visual form agnosia. Despite her severe damage to cortical visual areas, we report that DF''s stereo vision is strikingly unimpaired. She is better than many control observers at using binocular disparity to judge whether an isolated object appears near or far, and to resolve ambiguous structure-from-motion. DF is, however, poor at using relative disparity between features at different locations across the visual field. This may stem from a difficulty in identifying the surface boundaries where relative disparity is available. We suggest that the ventral processing stream may play a critical role in enabling healthy observers to extract fine depth information from relative disparities within one surface or between surfaces located in different parts of the visual field.     

A Method of Three-Dimensional Recording of Mandibular Movement Based on Two-Dimensional Image Feature Extraction

Background and Objective

To develop a real-time recording system based on computer binocular vision and two-dimensional image feature extraction to accurately record mandibular movement in three dimensions.

Methods

A computer-based binocular vision device with two digital cameras was used in conjunction with a fixed head retention bracket to track occlusal movement. Software was developed for extracting target spatial coordinates in real time based on two-dimensional image feature recognition. A plaster model of a subject’s upper and lower dentition were made using conventional methods. A mandibular occlusal splint was made on the plaster model, and then the occlusal surface was removed. Temporal denture base resin was used to make a 3-cm handle extending outside the mouth connecting the anterior labial surface of the occlusal splint with a detection target with intersecting lines designed for spatial coordinate extraction. The subject''s head was firmly fixed in place, and the occlusal splint was fully seated on the mandibular dentition. The subject was then asked to make various mouth movements while the mandibular movement target locus point set was recorded. Comparisons between the coordinate values and the actual values of the 30 intersections on the detection target were then analyzed using paired t-tests.

Results

The three-dimensional trajectory curve shapes of the mandibular movements were consistent with the respective subject movements. Mean XYZ coordinate values and paired t-test results were as follows: X axis: -0.0037 ± 0.02953, P = 0.502; Y axis: 0.0037 ± 0.05242, P = 0.704; and Z axis: 0.0007 ± 0.06040, P = 0.952. The t-test result showed that the coordinate values of the 30 cross points were considered statistically no significant. (P<0.05)

Conclusions

Use of a real-time recording system of three-dimensional mandibular movement based on computer binocular vision and two-dimensional image feature recognition technology produced a recording accuracy of approximately ± 0.1 mm, and is therefore suitable for clinical application. Certainly, further research is necessary to confirm the clinical applications of the method.     

Binocular fusion in Panum''''s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the "correspondence problem". While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum's limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum's limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum's limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum's limiting case and that the uniqueness constraint is still valid.     

Learning visuomotor transformations for gaze-control and grasping

For reaching to and grasping of an object, visual information about the object must be transformed into motor or postural commands for the arm and hand. In this paper, we present a robot model for visually guided reaching and grasping. The model mimics two alternative processing pathways for grasping, which are also likely to coexist in the human brain. The first pathway directly uses the retinal activation to encode the target position. In the second pathway, a saccade controller makes the eyes (cameras) focus on the target, and the gaze direction is used instead as positional input. For both pathways, an arm controller transforms information on the target’s position and orientation into an arm posture suitable for grasping. For the training of the saccade controller, we suggest a novel staged learning method which does not require a teacher that provides the necessary motor commands. The arm controller uses unsupervised learning: it is based on a density model of the sensor and the motor data. Using this density, a mapping is achieved by completing a partially given sensorimotor pattern. The controller can cope with the ambiguity in having a set of redundant arm postures for a given target. The combined model of saccade and arm controller was able to fixate and grasp an elongated object with arbitrary orientation and at arbitrary position on a table in 94% of trials.     

Learning with slight forgetting optimizes sensorimotor transformation in redundant motor systems

Recent theoretical studies have proposed that the redundant motor system in humans achieves well-organized stereotypical movements by minimizing motor effort cost and motor error. However, it is unclear how this optimization process is implemented in the brain, presumably because conventional schemes have assumed a priori that the brain somehow constructs the optimal motor command, and largely ignored the underlying trial-by-trial learning process. In contrast, recent studies focusing on the trial-by-trial modification of motor commands based on error information suggested that forgetting (i.e., memory decay), which is usually considered as an inconvenient factor in motor learning, plays an important role in minimizing the motor effort cost. Here, we examine whether trial-by-trial error-feedback learning with slight forgetting could minimize the motor effort and error in a highly redundant neural network for sensorimotor transformation and whether it could predict the stereotypical activation patterns observed in primary motor cortex (M1) neurons. First, using a simple linear neural network model, we theoretically demonstrated that: 1) this algorithm consistently leads the neural network to converge at a unique optimal state; 2) the biomechanical properties of the musculoskeletal system necessarily determine the distribution of the preferred directions (PD; the direction in which the neuron is maximally active) of M1 neurons; and 3) the bias of the PDs is steadily formed during the minimization of the motor effort. Furthermore, using a non-linear network model with realistic musculoskeletal data, we demonstrated numerically that this algorithm could consistently reproduce the PD distribution observed in various motor tasks, including two-dimensional isometric torque production, two-dimensional reaching, and even three-dimensional reaching tasks. These results may suggest that slight forgetting in the sensorimotor transformation network is responsible for solving the redundancy problem in motor control.     

Subjective effects of displacement errors in electronically processed stereo-television pictures

Several aspects of the roles of object contours and of rivalry and suppression in binocular vision are considered in a TV engineering context. Three experiments, using 3D b/w stills, were conducted to explore subjective effects of irregular horizontal shifts at object contours (displacement errors), which are expected to be a typical picture impairment problem of future 3D TV multi-viewpoint systems. Performance and rating tasks on a wide range of impairment magnitudes and various picture parameters served to give a quantitative estimate of the influence of displacement errors on: (1) correctness of binocular depth perception; and (2) picture quality. Two experiments (constant vs. variable location of impairments over time) with vertical grating stimuli showed binocular depth perception to withstand levels of up to 90% misplaced contour elements in one part of the stereo pair. Quality assessments were much more critical. They depended both on the proportion of impaired pixels and on the maximum horizontal width of individual impairments. A corresponding stimulus model was found to be valid for pictures with natural content, too. Impairments were less annoying when visible by only one eye instead of both. A specific formulation is given of the influence of contrast and spatial frequency features on performance.     

Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

An inference upon the neural network finding binocular correspondence

Previously, the authors proposed a model of neural network extracting binocular parallax (Hirai and Fukushima, 1975). It is a multilayered network whose final layers consist of neural elements corresponding to binocular depth neurons found in monkey's visual cortex. The binocular depth neuron is selectively sensitive to a binocular stimulus with a specific amount of binocular parallax and does not respond to a monocular one. As described in the last chapter of the previous article (Hirai and Fukushima, 1975), when a binocular pair of input patterns consist of, for example, many vertical bars placed very closely to each other, the binocular depth neurons might respond not only to correct binocular pairs, but also to incorrect ones. Our present study is concentrated upon how the visual system finds correct binocular pairs or binocular correspondence. It is assumed that some neural network is cascaded after the binocular depth neurons and finds out correct binocular correspondence by eliminating the incorrect binocular pairs. In this article a model of such neural network is proposed. The performance of the model has been simulated on a digital computer. The results of the computer simulation show that this model finds binocular correspondence satisfactorily. It has been demonstrated by the computer simulation that this model also explains the mechanism of the hysteresis in the binocular depth perception reported by Fender and Julesz (1967)This work has been done in the NHK Broadcasting Science Research Laboratories     

Binocular fusion in Panum’s limiting case of stereopsis obeys the uniqueness constraint

In the information processing procedure of stereo vision, the uniqueness constraint has been used as one of the constraints to solve the “correspondence problem”. While the uniqueness constraint is valid in most cases, whether it is still valid in some particular stimulus configuration (such as Panum’s limiting case) has been a problem of widespread debate for a long time. To investigate the problem, we adopted the Panum’s limiting case as its basic stimulus configuration, and delved into the phenomenon of binocular fusion from two distinct aspects: visual direction and orientation disparity. The results show that in Panum’s limiting case binocular fusion does not comply with the rules governing regular binocular fusion as far as visual direction and orientation disparity are concerned. This indicates that double fusion does not happen in Panum’s limiting case and that the uniqueness constraint is still valid.     

Developing a multi-Kinect-system for monitoring in dairy cows: object recognition and surface analysis using wavelets

Camera-based systems in dairy cattle were intensively studied over the last years. Different from this study, single camera systems with a limited range of applications were presented, mostly using 2D cameras. This study presents current steps in the development of a camera system comprising multiple 3D cameras (six Microsoft Kinect cameras) for monitoring purposes in dairy cows. An early prototype was constructed, and alpha versions of software for recording, synchronizing, sorting and segmenting images and transforming the 3D data in a joint coordinate system have already been implemented. This study introduced the application of two-dimensional wavelet transforms as method for object recognition and surface analyses. The method was explained in detail, and four differently shaped wavelets were tested with respect to their reconstruction error concerning Kinect recorded depth maps from different camera positions. The images’ high frequency parts reconstructed from wavelet decompositions using the haar and the biorthogonal 1.5 wavelet were statistically analyzed with regard to the effects of image fore- or background and of cows’ or persons’ surface. Furthermore, binary classifiers based on the local high frequencies have been implemented to decide whether a pixel belongs to the image foreground and if it was located on a cow or a person. Classifiers distinguishing between image regions showed high (⩾0.8) values of Area Under reciever operation characteristic Curve (AUC). The classifications due to species showed maximal AUC values of 0.69.