Audiovisual integration of speech falters under high attention demands

One of the most commonly cited examples of human multisensory integration occurs during exposure to natural speech, when the vocal and the visual aspects of the signal are integrated in a unitary percept. Audiovisual association of facial gestures and vocal sounds has been demonstrated in nonhuman primates and in prelinguistic children, arguing for a general basis for this capacity. One critical question, however, concerns the role of attention in such multisensory integration. Although both behavioral and neurophysiological studies have converged on a preattentive conceptualization of audiovisual speech integration, this mechanism has rarely been measured under conditions of high attentional load, when the observers' attention resources are depleted. We tested the extent to which audiovisual integration was modulated by the amount of available attentional resources by measuring the observers' susceptibility to the classic McGurk illusion in a dual-task paradigm. The proportion of visually influenced responses was severely, and selectively, reduced if participants were concurrently performing an unrelated visual or auditory task. In contrast with the assumption that crossmodal speech integration is automatic, our results suggest that these multisensory binding processes are subject to attentional demands.     

Effects of Multimodal Load on Spatial Monitoring as Revealed by ERPs

While the role of selective attention in filtering out irrelevant information has been extensively studied, its characteristics and neural underpinnings when multiple environmental stimuli have to be processed in parallel are much less known. Building upon a dual-task paradigm that induced spatial awareness deficits for contralesional hemispace in right hemisphere-damaged patients, we investigated the electrophysiological correlates of multimodal load during spatial monitoring in healthy participants. The position of appearance of briefly presented, lateralized targets had to be reported either in isolation (single task) or together with a concurrent task, visual or auditory, which recruited additional attentional resources (dual-task). This top-down manipulation of attentional load, without any change of the sensory stimulation, modulated the amplitude of the first positive ERP response (P1) and shifted its neural generators, with a suppression of the signal in the early visual areas during both visual and auditory dual tasks. Furthermore, later N2 contralateral components elicited by left targets were particularly influenced by the concurrent visual task and were related to increased activation of the supramarginal gyrus. These results suggest that the right hemisphere is particularly affected by load manipulations, and confirm its crucial role in subtending automatic orienting of spatial attention and in monitoring both hemispaces.     

Neuronal mechanisms of voice control are affected by implicit expectancy of externally triggered perturbations in auditory feedback

Accurate vocal production relies on several factors including sensory feedback and the ability to predict future challenges to the control processes. Repetitive patterns of perturbations in sensory feedback by themselves elicit implicit expectations in the vocal control system regarding the timing, quality and direction of perturbations. In the present study, the predictability of voice pitch-shifted auditory feedback was experimentally manipulated. A block of trials where all pitch-shift stimuli were upward, and therefore predictable was contrasted against an unpredictable block of trials in which the stimulus direction was randomized between upward and downward pitch-shifts. It was found that predictable perturbations in voice auditory feedback led to a reduction in the proportion of compensatory vocal responses, which might be indicative of a reduction in vocal control. The predictable perturbations also led to a reduction in the magnitude of the N1 component of cortical Event Related Potentials (ERP) that was associated with the reflexive compensations to the perturbations. We hypothesize that formation of expectancy in our study is accompanied by involuntary allocation of attentional resources occurring as a result of habituation or learning, that in turn trigger limited and controlled exploration-related motor variability in the vocal control system.     

Sensory Attenuation of Self-Produced Feedback: The Lombard Effect Revisited

The Lombard effect describes the automatic and involuntary increase in vocal intensity that speakers exhibit in a noisy environment. Previous studies of the Lombard effect have typically focused on the relationship between speaking and hearing. Automatic and involuntary increases in motor output have also been noted in studies of finger force production, an effect attributed to mechanisms of sensory attenuation. The present study tested the hypothesis that sensory attenuation mechanisms also underlie expression of the Lombard effect. Participants vocalized phonemes in time with a metronome, while auditory and visual feedback of their performance were manipulated or removed during the course of the trial. We demonstrate that providing a visual reference to calibrate somatosensory-based judgments of current vocal intensity resulted in reduced expression of the Lombard effect. Our results suggest that sensory attenuation effects typically seen in fingertip force production play an important role in the control of speech volume.     

Sensory-Motor Interactions for Vocal Pitch Monitoring in Non-Primary Human Auditory Cortex

The neural mechanisms underlying processing of auditory feedback during self-vocalization are poorly understood. One technique used to study the role of auditory feedback involves shifting the pitch of the feedback that a speaker receives, known as pitch-shifted feedback. We utilized a pitch shift self-vocalization and playback paradigm to investigate the underlying neural mechanisms of audio-vocal interaction. High-resolution electrocorticography (ECoG) signals were recorded directly from auditory cortex of 10 human subjects while they vocalized and received brief downward (−100 cents) pitch perturbations in their voice auditory feedback (speaking task). ECoG was also recorded when subjects passively listened to playback of their own pitch-shifted vocalizations. Feedback pitch perturbations elicited average evoked potential (AEP) and event-related band power (ERBP) responses, primarily in the high gamma (70–150 Hz) range, in focal areas of non-primary auditory cortex on superior temporal gyrus (STG). The AEPs and high gamma responses were both modulated by speaking compared with playback in a subset of STG contacts. From these contacts, a majority showed significant enhancement of high gamma power and AEP responses during speaking while the remaining contacts showed attenuated response amplitudes. The speaking-induced enhancement effect suggests that engaging the vocal motor system can modulate auditory cortical processing of self-produced sounds in such a way as to increase neural sensitivity for feedback pitch error detection. It is likely that mechanisms such as efference copies may be involved in this process, and modulation of AEP and high gamma responses imply that such modulatory effects may affect different cortical generators within distinctive functional networks that drive voice production and control.     

A Lightweight,Headphones-based System for Manipulating Auditory Feedback in Songbirds

Experimental manipulations of sensory feedback during complex behavior have provided valuable insights into the computations underlying motor control and sensorimotor plasticity¹. Consistent sensory perturbations result in compensatory changes in motor output, reflecting changes in feedforward motor control that reduce the experienced feedback error. By quantifying how different sensory feedback errors affect human behavior, prior studies have explored how visual signals are used to recalibrate arm movements^2,3 and auditory feedback is used to modify speech production^4-7. The strength of this approach rests on the ability to mimic naturalistic errors in behavior, allowing the experimenter to observe how experienced errors in production are used to recalibrate motor output.Songbirds provide an excellent animal model for investigating the neural basis of sensorimotor control and plasticity^8,9. The songbird brain provides a well-defined circuit in which the areas necessary for song learning are spatially separated from those required for song production, and neural recording and lesion studies have made significant advances in understanding how different brain areas contribute to vocal behavior^9-12. However, the lack of a naturalistic error-correction paradigm - in which a known acoustic parameter is perturbed by the experimenter and then corrected by the songbird - has made it difficult to understand the computations underlying vocal learning or how different elements of the neural circuit contribute to the correction of vocal errors¹³.The technique described here gives the experimenter precise control over auditory feedback errors in singing birds, allowing the introduction of arbitrary sensory errors that can be used to drive vocal learning. Online sound-processing equipment is used to introduce a known perturbation to the acoustics of song, and a miniaturized headphones apparatus is used to replace a songbird''s natural auditory feedback with the perturbed signal in real time. We have used this paradigm to perturb the fundamental frequency (pitch) of auditory feedback in adult songbirds, providing the first demonstration that adult birds maintain vocal performance using error correction¹⁴. The present protocol can be used to implement a wide range of sensory feedback perturbations (including but not limited to pitch shifts) to investigate the computational and neurophysiological basis of vocal learning.     

Remapping auditory-motor representations in voice production

Evidence regarding visually guided limb movements suggests that the motor system learns and maintains neural maps between motor commands and sensory feedback. Such systems are hypothesized to be used in a feed-forward control strategy that permits precision and stability without the delays of direct feedback control. Human vocalizations involve precise control over vocal and respiratory muscles. However, little is known about the sensorimotor representations underlying speech production. Here, we manipulated the heard fundamental frequency of the voice during speech to demonstrate learning of auditory-motor maps. Mandarin speakers repeatedly produced words with specific pitch patterns (tone categories). On each successive utterance, the frequency of their auditory feedback was increased by 1/100 of a semitone until they heard their feedback one full semitone above their true pitch. Subjects automatically compensated for these changes by lowering their vocal pitch. When feedback was unexpectedly returned to normal, speakers significantly increased the pitch of their productions beyond their initial baseline frequency. This adaptation was found to generalize to the production of another tone category. However, results indicate that a more robust adaptation was produced for the tone that was spoken during feedback alteration. The immediate aftereffects suggest a global remapping of the auditory-motor relationship after an extremely brief training period. However, this learning does not represent a complete transformation of the mapping; rather, it is in part target dependent.     

Attentional load modulates responses of human primary visual cortex to invisible stimuli

Visual neuroscience has long sought to determine the extent to which stimulus-evoked activity in visual cortex depends on attention and awareness. Some influential theories of consciousness maintain that the allocation of attention is restricted to conscious representations [1, 2]. However, in the load theory of attention [3], competition between task-relevant and task-irrelevant stimuli for limited-capacity attention does not depend on conscious perception of the irrelevant stimuli. The critical test is whether the level of attentional load in a relevant task would determine unconscious neural processing of invisible stimuli. Human participants were scanned with high-field fMRI while they performed a foveal task of low or high attentional load. Irrelevant, invisible monocular stimuli were simultaneously presented peripherally and were continuously suppressed by a flashing mask in the other eye [4]. Attentional load in the foveal task strongly modulated retinotopic activity evoked in primary visual cortex (V1) by the invisible stimuli. Contrary to traditional views [1, 2, 5, 6], we found that availability of attentional capacity determines neural representations related to unconscious processing of continuously suppressed stimuli in human primary visual cortex. Spillover of attention to cortical representations of invisible stimuli (under low load) cannot be a sufficient condition for their awareness.     

Attention and time estimation in 5- and 8-year-old children: a dual-task procedure

This experiment tested the effect of a dual-task on time reproduction in 5- and 8-year-olds. Children had to reproduce a stimulus duration lasting for 6 or 12 s, during which they either did or did not perform a concurrent non-temporal task (i.e. picture naming) both in low (LA) and high (HA) attentional demand conditions. The results showed that children reproduced shorter durations in the dual-task than in the single-task condition, whatever the duration value used. However, this shortening effect was greater in the 5-year-olds than in the 8-year-olds. Furthermore, in the 5-year-olds, temporal reproductions were significantly shorter in both dual-tasks (LA or HA) than in the single-task, whereas, in the 8-year-olds, differences reached significance only between the HA dual-task and the single-task. In the non-temporal task, the proportion of naming errors was also greater in the dual-task than in the single-task, especially under high attentional demand, but it did not significantly differ between the two age groups tested.     

Selective attention to task-irrelevant emotional distractors is unaffected by the perceptual load associated with a foreground task

A number of studies have shown that emotionally arousing stimuli are preferentially processed in the human brain. Whether or not this preference persists under increased perceptual load associated with a task at hand remains an open question. Here we manipulated two possible determinants of the attentional selection process, perceptual load associated with a foreground task and the emotional valence of concurrently presented task-irrelevant distractors. As a direct measure of sustained attentional resource allocation in early visual cortex we used steady-state visual evoked potentials (SSVEPs) elicited by distinct flicker frequencies of task and distractor stimuli. Subjects either performed a detection (low load) or discrimination (high load) task at a centrally presented symbol stream that flickered at 8.6 Hz while task-irrelevant neutral or unpleasant pictures from the International Affective Picture System (IAPS) flickered at a frequency of 12 Hz in the background of the stream. As reflected in target detection rates and SSVEP amplitudes to both task and distractor stimuli, unpleasant relative to neutral background pictures more strongly withdrew processing resources from the foreground task. Importantly, this finding was unaffected by the factor 'load' which turned out to be a weak modulator of attentional processing in human visual cortex.     

Auditory Processing under Cross-Modal Visual Load Investigated with Simultaneous EEG-fMRI

Cognitive task demands in one sensory modality (T1) can have beneficial effects on a secondary task (T2) in a different modality, due to reduced top-down control needed to inhibit the secondary task, as well as crossmodal spread of attention. This contrasts findings of cognitive load compromising a secondary modality’s processing. We manipulated cognitive load within one modality (visual) and studied the consequences of cognitive demands on secondary (auditory) processing. 15 healthy participants underwent a simultaneous EEG-fMRI experiment. Data from 8 participants were obtained outside the scanner for validation purposes. The primary task (T1) was to respond to a visual working memory (WM) task with four conditions, while the secondary task (T2) consisted of an auditory oddball stream, which participants were asked to ignore. The fMRI results revealed fronto-parietal WM network activations in response to T1 task manipulation. This was accompanied by significantly higher reaction times and lower hit rates with increasing task difficulty which confirmed successful manipulation of WM load. Amplitudes of auditory evoked potentials, representing fundamental auditory processing showed a continuous augmentation which demonstrated a systematic relation to cross-modal cognitive load. With increasing WM load, primary auditory cortices were increasingly deactivated while psychophysiological interaction results suggested the emergence of auditory cortices connectivity with visual WM regions. These results suggest differential effects of crossmodal attention on fundamental auditory processing. We suggest a continuous allocation of resources to brain regions processing primary tasks when challenging the central executive under high cognitive load.     

How do we select perceptions and actions? Human brain imaging studies.

The selective nature of human perception and action implies a modulatory interaction between sensorimotor processes and attentional processes. This paper explores the use of functional imaging in humans to explore the mechanisms of perceptual selection and the fate of irrelevant stimuli that are not selected. Experiments with positron emission tomography show that two qualitatively different patterns of modulation of cerebral blood flow can be observed in experiments where non-spatial visual attention and auditory attention are manipulated. These patterns of modulation of cerebral blood flow modulation can be described as gain control and bias signal mechanisms. In visual and auditory cortex, the dominant change in cerebral blood flow associated with attention to either modality is related to a bias signal. The relation of these patterns of modulation to attentional effects that have been observed in single neurons is discussed. The existence of mechanisms for selective perception raises the more general question of whether irrelevant ignored stimuli are nevertheless perceived. Lavie''s theory of attention proposes that the degree to which ignored stimuli are processed varies depending on the perceptual load of the current task. Evidence from behavioural and functional magnetic resonance imaging studies of ignored visual motion processing is presented in support of this proposal.     

Effects of attention and slow potential shifts on self-regulation of event-related potentials

Research on the effects of self-regulation of slow potentials (SP) and event-related potentials (ERP) has failed to look at the possible interactions of these two kinds of brain potentials. The present study investigated such interactions by recording both ERP and SP potential changes in an operant ERP conditioning paradigm. Ten subjects participated in two conditions that were designed to differentially manipulate attention to the stimuli. In the operant conditioning task, subjects received auditory feedback as they attempted to increase the ERP amplitude at 180 msec poststimulus (P180), which was elicited by a subpainful shock stimulus to the forearm over 250 trials. In the distraction task, subjects were instructed not to attend to stimuli or feedback tones, but rather received and were tested on reading materials. Attention, as manipulated by these tasks, was not a determinant of changes in ERP amplitude since there were no significant differences in the size of P180 between attention conditions. While no significant change in the mean ERP amplitude occurred, subjects were able to produce ERPs above criterion threshold significantly more often during trials in the conditioning task than in the reading task. Thus, there was evidence of some learning. The difference in wave forms between hit and miss trials indicates a latency shift (with misses having a later ERP peak). This may indicate that latency, rather than, or in addition to, amplitude, is shaped during conditioning procedures. In addition, the CNV that developed between the shock stimulus and the feedback signal during conditioning was significantly larger in amplitude than in the distraction condition. This is taken as evidence of increased attention during conditioning. Since hit trials demonstrated larger contingent negative variation (CNV) amplitudes, production of CNVs may be instrumental in mediating hits. Therefore, attentional mechanisms may play a role in successful ERP self-regulation. No correlations were found involving P180, CNVs, or tonic slow potential shifts. Changes in tonic DC levels showed a suggestive trend between conditions. Although both conditions began with a negative shift, during conditioning the negativity increased, while during distraction the tonic level went to positivity. These trends support the hypothesis that attention and arousal increased during conditioning. The possible reasons for the lack of significant correlations between ERP and tonic or phasic slow potential changes in this paradigm are discussed.     

Dancers and fastball sports athletes have different spatial visual attention styles

Physical exercise and the training effects of repeated practice of skills over an extended period of time may have additive effects on brain networks and functions. Various motor skills and attentional styles can be developed by athletes engaged in different sports. In this study, the effects of fast ball sports and dance training on attention were investigated by event related potentials (ERP). ERP were recorded in auditory and visual tasks in professional dancer, professional fast ball sports athlete (FBSA) and healthy control volunteer groups consisting of twelve subjects each. In the auditory task both dancer and FBSA groups have faster N200 (N2) and P300 (P3) latencies than the controls. In the visual task FBSA have faster latencies of P3 than the dancers and controls. They also have higher P100 (P1) amplitudes to non-target stimuli than the dancers and controls. On the other hand, dancers have faster latencies of P1 and higher N100 (N1) amplitude to non-target stimuli and they also have higher P3 amplitudes than the FBSA and controls. Overall exercise has positive effects on cognitive processing speed as reflected on the faster auditory N2 and P3 latencies. However, FBSA and dancers differed on attentional styles in the visual task. Dancers displayed predominantly endogenous/top down features reflected by increased N1 and P3 amplitudes, decreased P1 amplitude and shorter P1 latency. On the other hand, FBSA showed predominantly exogenous/bottom up processes revealed by increased P1 amplitude. The controls were in between the two groups.     

Cross-modal distortion of time perception: demerging the effects of observed and performed motion

Temporal information is often contained in multi-sensory stimuli, but it is currently unknown how the brain combines e.g. visual and auditory cues into a coherent percept of time. The existing studies of cross-modal time perception mainly support the "modality appropriateness hypothesis", i.e. the domination of auditory temporal cues over visual ones because of the higher precision of audition for time perception. However, these studies suffer from methodical problems and conflicting results. We introduce a novel experimental paradigm to examine cross-modal time perception by combining an auditory time perception task with a visually guided motor task, requiring participants to follow an elliptic movement on a screen with a robotic manipulandum. We find that subjective duration is distorted according to the speed of visually observed movement: The faster the visual motion, the longer the perceived duration. In contrast, the actual execution of the arm movement does not contribute to this effect, but impairs discrimination performance by dual-task interference. We also show that additional training of the motor task attenuates the interference, but does not affect the distortion of subjective duration. The study demonstrates direct influence of visual motion on auditory temporal representations, which is independent of attentional modulation. At the same time, it provides causal support for the notion that time perception and continuous motor timing rely on separate mechanisms, a proposal that was formerly supported by correlational evidence only. The results constitute a counterexample to the modality appropriateness hypothesis and are best explained by Bayesian integration of modality-specific temporal information into a centralized "temporal hub".     

Attentional Switches and Dual-Task Interference

In four experiments, we studied the time course of interference between detection of an oddball orientation target (OT) in an 8-item circular search display, and identification of a letter target (LT) in a central stream of distractor letters. Dual-task performance for different temporal lags between targets was compared to single-task performance. When the LT preceded the OT, dual-task performance levels were reduced at short inter-target intervals of 0 and 166 ms; when the OT preceded the LT, the dual-task interference was unexpectedly stronger and lasted for up to 500 ms. Resource competition due to temporally overlapping target processing cannot account for this result, because the feature search task is easier than the letter identification task, and therefore would have generated less interference when presented first. Two alternative explanations were explored. First, by manipulating the spatial inter-target distance, we investigated to what degree there is a penalty associated with directing the attentional window from a large object (the search display) to a smaller object (the central letter stream). Second, by varying the duration of the OT and subsequent mask, we studied whether the interference was caused by the difficulty of disengaging attention from the search display. Results support this second explanation and thus indicate that switching attention to the letter stream is hampered by the continuing presence of (masked) search display items. This result shows that attentional effects may play a major role in dual-task execution and can easily obscure interference due to other factors such as resource competition.     

Effects of Auditory Stimuli in the Horizontal Plane on Audiovisual Integration: An Event-Related Potential Study

This article aims to investigate whether auditory stimuli in the horizontal plane, particularly originating from behind the participant, affect audiovisual integration by using behavioral and event-related potential (ERP) measurements. In this study, visual stimuli were presented directly in front of the participants, auditory stimuli were presented at one location in an equidistant horizontal plane at the front (0°, the fixation point), right (90°), back (180°), or left (270°) of the participants, and audiovisual stimuli that include both visual stimuli and auditory stimuli originating from one of the four locations were simultaneously presented. These stimuli were presented randomly with equal probability; during this time, participants were asked to attend to the visual stimulus and respond promptly only to visual target stimuli (a unimodal visual target stimulus and the visual target of the audiovisual stimulus). A significant facilitation of reaction times and hit rates was obtained following audiovisual stimulation, irrespective of whether the auditory stimuli were presented in the front or back of the participant. However, no significant interactions were found between visual stimuli and auditory stimuli from the right or left. Two main ERP components related to audiovisual integration were found: first, auditory stimuli from the front location produced an ERP reaction over the right temporal area and right occipital area at approximately 160–200 milliseconds; second, auditory stimuli from the back produced a reaction over the parietal and occipital areas at approximately 360–400 milliseconds. Our results confirmed that audiovisual integration was also elicited, even though auditory stimuli were presented behind the participant, but no integration occurred when auditory stimuli were presented in the right or left spaces, suggesting that the human brain might be particularly sensitive to information received from behind than both sides.     

Increased Early Processing of Task-Irrelevant Auditory Stimuli in Older Adults

The inhibitory deficit hypothesis of cognitive aging posits that older adults’ inability to adequately suppress processing of irrelevant information is a major source of cognitive decline. Prior research has demonstrated that in response to task-irrelevant auditory stimuli there is an age-associated increase in the amplitude of the N1 wave, an ERP marker of early perceptual processing. Here, we tested predictions derived from the inhibitory deficit hypothesis that the age-related increase in N1 would be 1) observed under an auditory-ignore, but not auditory-attend condition, 2) attenuated in individuals with high executive capacity (EC), and 3) augmented by increasing cognitive load of the primary visual task. ERPs were measured in 114 well-matched young, middle-aged, young-old, and old-old adults, designated as having high or average EC based on neuropsychological testing. Under the auditory-ignore (visual-attend) task, participants ignored auditory stimuli and responded to rare target letters under low and high load. Under the auditory-attend task, participants ignored visual stimuli and responded to rare target tones. Results confirmed an age-associated increase in N1 amplitude to auditory stimuli under the auditory-ignore but not auditory-attend task. Contrary to predictions, EC did not modulate the N1 response. The load effect was the opposite of expectation: the N1 to task-irrelevant auditory events was smaller under high load. Finally, older adults did not simply fail to suppress the N1 to auditory stimuli in the task-irrelevant modality; they generated a larger response than to identical stimuli in the task-relevant modality. In summary, several of the study’s findings do not fit the inhibitory-deficit hypothesis of cognitive aging, which may need to be refined or supplemented by alternative accounts.     

Perceiving a Stranger's Voice as Being One's Own: A ‘Rubber Voice’ Illusion?

We describe an illusion in which a stranger's voice, when presented as the auditory concomitant of a participant's own speech, is perceived as a modified version of their own voice. When the congruence between utterance and feedback breaks down, the illusion is also broken. Compared to a baseline condition in which participants heard their own voice as feedback, hearing a stranger's voice induced robust changes in the fundamental frequency (F0) of their production. Moreover, the shift in F0 appears to be feedback dependent, since shift patterns depended reliably on the relationship between the participant's own F0 and the stranger-voice F0. The shift in F0 was evident both when the illusion was present and after it was broken, suggesting that auditory feedback from production may be used separately for self-recognition and for vocal motor control. Our findings indicate that self-recognition of voices, like other body attributes, is malleable and context dependent.     

Task attention facilitates learning of task-irrelevant stimuli

Attention plays a fundamental role in visual learning and memory. One highly established principle of visual attention is that the harder a central task is, the more attentional resources are used to perform the task and the smaller amount of attention is allocated to peripheral processing because of limited attention capacity. Here we show that this principle holds true in a dual-task setting but not in a paradigm of task-irrelevant perceptual learning. In Experiment 1, eight participants were asked to identify either bright or dim number targets at the screen center and to remember concurrently presented scene backgrounds. Their recognition performances for scenes paired with dim/hard targets were worse than those for scenes paired with bright/easy targets. In Experiment 2, eight participants were asked to identify either bright or dim letter targets at the screen center while a task-irrelevant coherent motion was concurrently presented in the background. After five days of training on letter identification, participants improved their motion sensitivity to the direction paired with hard/dim targets improved but not to the direction paired with easy/bright targets. Taken together, these results suggest that task-irrelevant stimuli are not subject to the attentional control mechanisms that task-relevant stimuli abide.