Speech perception: when to put your money where the mouth is

Visual information augments our understanding of auditory speech. New evidence shows that infants' gaze fixations to the mouth and eye region shift predictably with changes in age and language familiarity.     

Figure-ground activity in V1 and guidance of saccadic eye movements.

Every day we shift our gaze about 150.000 times mostly without noticing it. The direction of these gaze shifts are not random but directed by sensory information and internal factors. After each movement the eyes hold still for a brief moment so that visual information at the center of our gaze can be processed in detail. This means that visual information at the saccade target location is sufficient to accurately guide the gaze shift but yet is not sufficiently processed to be fully perceived. In this paper I will discuss the possible role of activity in the primary visual cortex (V1), in particular figure-ground activity, in oculo-motor behavior. Figure-ground activity occurs during the late response period of V1 neurons and correlates with perception. The strength of figure-ground responses predicts the direction and moment of saccadic eye movements. The superior colliculus, a gaze control center that integrates visual and motor signals, receives direct anatomical connections from V1. These projections may convey the perceptual information that is required for appropriate gaze shifts. In conclusion, figure-ground activity in V1 may act as an intermediate component linking visual and motor signals.     

Smaller Is Better: Drift in Gaze Measurements due to Pupil Dynamics

Camera-based eye trackers are the mainstay of eye movement research and countless practical applications of eye tracking. Recently, a significant impact of changes in pupil size on gaze position as measured by camera-based eye trackers has been reported. In an attempt to improve the understanding of the magnitude and population-wise distribution of the pupil-size dependent shift in reported gaze position, we present the first collection of binocular pupil drift measurements recorded from 39 subjects. The pupil-size dependent shift varied greatly between subjects (from 0.3 to 5.2 deg of deviation, mean 2.6 deg), but also between the eyes of individual subjects (0.1 to 3.0 deg difference, mean difference 1.0 deg). We observed a wide range of drift direction, mostly downward and nasal. We demonstrate two methods to partially compensate the pupil-based shift using separate calibrations in pupil-constricted and pupil-dilated conditions, and evaluate an improved method of compensation based on individual look-up-tables, achieving up to 74% of compensation.     

A head view-invariant representation of gaze direction in anterior superior temporal sulcus

Humans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.     

Eye-Head Coordination for Visual Cognitive Processing

We investigated coordinated movements between the eyes and head (“eye-head coordination”) in relation to vision for action. Several studies have measured eye and head movements during a single gaze shift, focusing on the mechanisms of motor control during eye-head coordination. However, in everyday life, gaze shifts occur sequentially and are accompanied by movements of the head and body. Under such conditions, visual cognitive processing influences eye movements and might also influence eye-head coordination because sequential gaze shifts include cycles of visual processing (fixation) and data acquisition (gaze shifts). In the present study, we examined how the eyes and head move in coordination during visual search in a large visual field. Subjects moved their eyes, head, and body without restriction inside a 360° visual display system. We found patterns of eye-head coordination that differed those observed in single gaze-shift studies. First, we frequently observed multiple saccades during one continuous head movement, and the contribution of head movement to gaze shifts increased as the number of saccades increased. This relationship between head movements and sequential gaze shifts suggests eye-head coordination over several saccade-fixation sequences; this could be related to cognitive processing because saccade-fixation cycles are the result of visual cognitive processing. Second, distribution bias of eye position during gaze fixation was highly correlated with head orientation. The distribution peak of eye position was biased in the same direction as head orientation. This influence of head orientation suggests that eye-head coordination is involved in gaze fixation, when the visual system processes retinal information. This further supports the role of eye-head coordination in visual cognitive processing.     

Operant control of human eye movements

Saccade and smooth pursuit are the eye movements used by primates to shift gaze. In this article we review evidence for the effects of reinforcement on several dimensions of these responses such as their latencies, velocities or amplitudes. We propose that these responses are operant behaviours controlled by their consequences on performance of visually guided tasks. Studying the conditions under which particular eye movement patterns might emerge from the cumulative effects of reinforcement provides critical insights about how motor responses are attuned to environmental exigencies.     

Speaking and Listening with the Eyes: Gaze Signaling during Dyadic Interactions

Cognitive scientists have long been interested in the role that eye gaze plays in social interactions. Previous research suggests that gaze acts as a signaling mechanism and can be used to control turn-taking behaviour. However, early research on this topic employed methods of analysis that aggregated gaze information across an entire trial (or trials), which masks any temporal dynamics that may exist in social interactions. More recently, attempts have been made to understand the temporal characteristics of social gaze but little research has been conducted in a natural setting with two interacting participants. The present study combines a temporally sensitive analysis technique with modern eye tracking technology to 1) validate the overall results from earlier aggregated analyses and 2) provide insight into the specific moment-to-moment temporal characteristics of turn-taking behaviour in a natural setting. Dyads played two social guessing games (20 Questions and Heads Up) while their eyes were tracked. Our general results are in line with past aggregated data, and using cross-correlational analysis on the specific gaze and speech signals of both participants we found that 1) speakers end their turn with direct gaze at the listener and 2) the listener in turn begins to speak with averted gaze. Convergent with theoretical models of social interaction, our data suggest that eye gaze can be used to signal both the end and the beginning of a speaking turn during a social interaction. The present study offers insight into the temporal dynamics of live dyadic interactions and also provides a new method of analysis for eye gaze data when temporal relationships are of interest.     

Evidence for on-line visual guidance during saccadic gaze shifts.

Rapid orientating movements of the eyes are believed to be controlled ballistically. The mechanism underlying this control is thought to involve a comparison between the desired displacement of the eye and an estimate of its actual position (obtained from the integration of the eye velocity signal). This study shows, however, that under certain circumstances fast gaze movements may be controlled quite differently and may involve mechanisms which use visual information to guide movements prospectively. Subjects were required to make large gaze shifts in yaw towards a target whose location and motion were unknown prior to movement onset. Six of those tested demonstrated remarkable accuracy when making gaze shifts towards a target that appeared during their ongoing movement. In fact their level of accuracy was not significantly different from that shown when they performed a 'remembered' gaze shift to a known stationary target (F3,15 = 0.15, p > 0.05). The lack of a stereotypical relationship between the skew of the gaze velocity profile and movement duration indicates that on-line modifications were being made. It is suggested that a fast route from the retina to the superior colliculus could account for this behaviour and that models of oculomotor control need to be updated.     

Eye movements: viewing the window of opportunity

When searching with our eyes, parallel programming of successive eye movements ensures that visual information arriving too late to alter the direction of one eye movement can still influence the direction of the next. Paradoxically, we can use random noise to probe the time period over which visual information influences where next to direct our gaze.     

A neural mechanism for coordinate transformation predicts pre-saccadic remapping

Whenever we shift our gaze, any location information encoded in the retinocentric reference frame that is predominant in the visual system is obliterated. How is spatial memory retained across gaze changes? Two different explanations have been proposed: Retinocentric information may be transformed into a gaze-invariant representation through a mechanism consistent with gain fields observed in parietal cortex, or retinocentric information may be updated in anticipation of the shift expected with every gaze change, a proposal consistent with neural observations in LIP. The explanations were considered incompatible with each other, because retinocentric update is observed before the gaze shift has terminated. Here, we show that a neural dynamic mechanism for coordinate transformation can also account for retinocentric updating. Our model postulates an extended mechanism of reference frame transformation that is based on bidirectional mapping between a retinocentric and a body-centered representation and that enables transforming multiple object locations in parallel. The dynamic coupling between the two reference frames generates a shift of the retinocentric representation for every gaze change. We account for the predictive nature of the observed remapping activity by using the same kind of neural mechanism to generate an internal representation of gaze direction that is predictively updated based on corollary discharge signals. We provide evidence for the model by accounting for a series of behavioral and neural experimental observations.     

Fixational Eye Movement Correction of Blink-Induced Gaze Position Errors

Our eyes move continuously. Even when we attempt to fix our gaze, we produce “fixational” eye movements including microsaccades, drift and tremor. The potential role of microsaccades versus drifts in the control of eye position has been debated for decades and remains in question today. Here we set out to determine the corrective functions of microsaccades and drifts on gaze-position errors due to blinks in non-human primates (Macaca mulatta) and humans. Our results show that blinks contribute to the instability of gaze during fixation, and that microsaccades, but not drifts, correct fixation errors introduced by blinks. These findings provide new insights about eye position control during fixation, and indicate a more general role of microsaccades in fixation correction than thought previously.     

Cues for early social skills: direct gaze modulates newborns' recognition of talking faces

Previous studies showed that, from birth, speech and eye gaze are two important cues in guiding early face processing and social cognition. These studies tested the role of each cue independently; however, infants normally perceive speech and eye gaze together. Using a familiarization-test procedure, we first familiarized newborn infants (n = 24) with videos of unfamiliar talking faces with either direct gaze or averted gaze. Newborns were then tested with photographs of the previously seen face and of a new one. The newborns looked longer at the face that previously talked to them, but only in the direct gaze condition. These results highlight the importance of both speech and eye gaze as socio-communicative cues by which infants identify others. They suggest that gaze and infant-directed speech, experienced together, are powerful cues for the development of early social skills.     

Coordinates of Human Visual and Inertial Heading Perception

Heading estimation involves both inertial and visual cues. Inertial motion is sensed by the labyrinth, somatic sensation by the body, and optic flow by the retina. Because the eye and head are mobile these stimuli are sensed relative to different reference frames and it remains unclear if a perception occurs in a common reference frame. Recent neurophysiologic evidence has suggested the reference frames remain separate even at higher levels of processing but has not addressed the resulting perception. Seven human subjects experienced a 2s, 16 cm/s translation and/or a visual stimulus corresponding with this translation. For each condition 72 stimuli (360° in 5° increments) were delivered in random order. After each stimulus the subject identified the perceived heading using a mechanical dial. Some trial blocks included interleaved conditions in which the influence of ±28° of gaze and/or head position were examined. The observations were fit using a two degree-of-freedom population vector decoder (PVD) model which considered the relative sensitivity to lateral motion and coordinate system offset. For visual stimuli gaze shifts caused shifts in perceived head estimates in the direction opposite the gaze shift in all subjects. These perceptual shifts averaged 13 ± 2° for eye only gaze shifts and 17 ± 2° for eye-head gaze shifts. This finding indicates visual headings are biased towards retina coordinates. Similar gaze and head direction shifts prior to inertial headings had no significant influence on heading direction. Thus inertial headings are perceived in body-centered coordinates. Combined visual and inertial stimuli yielded intermediate results.     

The neural selection and control of saccades by the frontal eye field

Recent research has provided new insights into the neural processes that select the target for and control the production of a shift of gaze. Being a key node in the network that subserves visual processing and saccade production, the frontal eye field (FEF) has been an effective area in which to monitor these processes. Certain neurons in the FEF signal the location of conspicuous or meaningful stimuli that may be the targets for saccades. Other neurons control whether and when the gaze shifts. The existence of distinct neural processes for visual selection and saccade production is necessary to explain the flexibility of visually guided behaviour.     

The composition of central programs subserving horizontal eye movements in man

A hypothesis is presented which describes, in biomechanical terms, the central programs underlying horizontal eye movements in man. It is suggested that eye movements are produced by means of programmed shifts of the so-called invariant muscle characteristics (static force vs angle of gaze). These shifts lead to a change of the equilibrium point resulting from the interaction of agnnist and antagonist muscles and, as a consequence, to movement and the attainment of a new position of gaze. A reciprocal or a coactivation command to agonist and antagonist muscles occurs when their characteristics shift with respect to the coordinate in the same or opposite directions, respectively. It is proposed that during pursuit and saccadic eye movements a supperposition of the both central commands occurs. During a saccade, the reciprocal command develops evenly up to a certain level. The initial and final levels of the reciprocal command dictate the respective position of gaze and therefore the size of the saccade. The coactivation command develops to a maximum level and is slowly switched off when the new position of gaze has been achieved. The magnitude of the coactivation command seems to be not connected with an absolute position of gaze. It provides probably a stability of the movement and, in particular, prevents overshoot and oscillation during the saccade. The same timing of these commands occurs during pursuit movements, but the magnitude of the coactivation command and the rates of the development of the both commands are less in this case and correlate with the velocity of the movement. This hypothesis enables the tension changes in the muscle during saccadic and pursuit movements to be simulated in qualitative accordance with unique experimental data obtained by Collins et al. (1975). The functional significance of superposition of these motor commands and similarity in the efferent organization of eye and limb movements are discussed.     

眼睛注视线索对物体认知加工的影响及其机制

眼睛注视作为一种重要的非语言社会线索,不仅可以传达他人丰富的注意方向信息,并且能够诱发独特的社会性注意行为.近年来,研究者利用改编的社会性注意任务发现,眼睛注视线索还可以进一步影响我们对各种不同种类物体(用具、符号、面孔等)的感知觉加工,以及主观评价、记忆等其他高级认知加工过程.眼睛注视线索对物体加工的这种影响受到诸多因素的调节,如面孔属性、数量以及注视模式等.特别地,眼睛注视线索对物体加工的这一调制作用能够在无意识水平发生,具有一定的特异性.此外,针对这一调制作用背后机制的研究暗示心理理论和观点采择可能参与其中,但仍有待进一步探究.眼睛注视对物体加工影响的研究有助于我们深入理解社会互动的方式以及人类与环境的交互过程,因此具有重要的理论意义和社会应用价值.     

Young infants' neural processing of objects is affected by eye gaze direction and emotional expression

Eye gaze is an important social cue which is used to determine another person's focus of attention and intention to communicate. In combination with a fearful facial expression eye gaze can also signal threat in the environment. The ability to detect and understand others' social signals is essential in order to avoid danger and enable social evaluation. It has been a matter of debate when infants are able to use gaze cues and emotional facial expressions in reference to external objects. Here we demonstrate that by 3 months of age the infant brain differentially responds to objects as a function of how other people are reacting to them. Using event-related electrical brain potentials (ERPs), we show that an indicator of infants' attention is enhanced by an adult's expression of fear toward an unfamiliar object. The infant brain showed an increased Negative central (Nc) component toward objects that had been previously cued by an adult's eye gaze and frightened facial expression. Our results further suggest that infants' sensitivity cannot be due to a general arousal elicited by a frightened face with eye gaze directed at an object. The neural attention system of 3 month old infants is sensitive to an adult's eye gaze direction in combination with a fearful expression. This early capacity may lay the foundation for the development of more sophisticated social skills such as social referencing, language, and theory of mind.     

Vision: a brake on the speed of sight

We move our eyes more often than our heart beats. Our brain seems to cope effortlessly with the consequences of these rapid visual alterations, but a new study shows that similar scene changes in the absence of eye movements delay the speed of information processing. So are there costs in constantly shifting our focus of gaze?     

眼－头运动协同中头部运动的双重模式控制

本文通过目标运动引起的眼－头运动协同的实验，测量和分析了头部运动的动态特性来探讨其头部运动的控制机制。研究结果揭示了眼－头协同的注视运动中头部运动的双重模式控制机制：在小幅度运动范围是线性比例控制，在大幅度运动范围是使用最大作用力的Ｂａｎｇ－Ｂａｎｇ开关控制。