An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex

Research on the neural basis of speech-reading implicates a network of auditory language regions involving inferior frontal cortex, premotor cortex and sites along superior temporal cortex. In audiovisual speech studies, neural activity is consistently reported in posterior superior temporal Sulcus (pSTS) and this site has been implicated in multimodal integration. Traditionally, multisensory interactions are considered high-level processing that engages heteromodal association cortices (such as STS). Recent work, however, challenges this notion and suggests that multisensory interactions may occur in low-level unimodal sensory cortices. While previous audiovisual speech studies demonstrate that high-level multisensory interactions occur in pSTS, what remains unclear is how early in the processing hierarchy these multisensory interactions may occur. The goal of the present fMRI experiment is to investigate how visual speech can influence activity in auditory cortex above and beyond its response to auditory speech. In an audiovisual speech experiment, subjects were presented with auditory speech with and without congruent visual input. Holding the auditory stimulus constant across the experiment, we investigated how the addition of visual speech influences activity in auditory cortex. We demonstrate that congruent visual speech increases the activity in auditory cortex.     

Functional imaging reveals numerous fields in the monkey auditory cortex

Anatomical studies propose that the primate auditory cortex contains more fields than have actually been functionally confirmed or described. Spatially resolved functional magnetic resonance imaging (fMRI) with carefully designed acoustical stimulation could be ideally suited to extend our understanding of the processing within these fields. However, after numerous experiments in humans, many auditory fields remain poorly characterized. Imaging the macaque monkey is of particular interest as these species have a richer set of anatomical and neurophysiological data to clarify the source of the imaged activity. We functionally mapped the auditory cortex of behaving and of anesthetized macaque monkeys with high resolution fMRI. By optimizing our imaging and stimulation procedures, we obtained robust activity throughout auditory cortex using tonal and band-passed noise sounds. Then, by varying the frequency content of the sounds, spatially specific activity patterns were observed over this region. As a result, the activity patterns could be assigned to many auditory cortical fields, including those whose functional properties were previously undescribed. The results provide an extensive functional tessellation of the macaque auditory cortex and suggest that 11 fields contain neurons tuned for the frequency of sounds. This study provides functional support for a model where three fields in primary auditory cortex are surrounded by eight neighboring “belt” fields in non-primary auditory cortex. The findings can now guide neurophysiological recordings in the monkey to expand our understanding of the processing within these fields. Additionally, this work will improve fMRI investigations of the human auditory cortex.     

Experience-dependent modulation of tonotopic neural responses in human auditory cortex.

Experience-dependent plasticity of receptive fields in the auditory cortex has been demonstrated by electrophysiological experiments in animals. In the present study we used PET neuroimaging to measure regional brain activity in volunteer human subjects during discriminatory classical conditioning of high (8000 Hz) or low (200 Hz) frequency tones by an aversive 100 dB white noise burst. Conditioning-related, frequency-specific modulation of tonotopic neural responses in the auditory cortex was observed. The modulated regions of the auditory cortex positively covaried with activity in the amygdala, basal forebrain and orbitofrontal cortex, and showed context-specific functional interactions with the medial geniculate nucleus. These results accord with animal single-unit data and support neurobiological models of auditory conditioning and value-dependent neural selection.     

Competing neural responses for auditory and visual decisions

Why is it hard to divide attention between dissimilar activities, such as reading and listening to a conversation? We used functional magnetic resonance imaging (fMRI) to study interference between simple auditory and visual decisions, independently of motor competition. Overlapping activity for auditory and visual tasks performed in isolation was found in lateral prefrontal regions, middle temporal cortex and parietal cortex. When the visual stimulus occurred during the processing of the tone, its activation in prefrontal and middle temporal cortex was suppressed. Additionally, reduced activity was seen in modality-specific visual cortex. These results paralleled impaired awareness of the visual event. Even without competing motor responses, a simple auditory decision interferes with visual processing on different neural levels, including prefrontal cortex, middle temporal cortex and visual regions.     

Effects of L-dopa during Auditory Instrumental Learning in Humans

The dopaminergic neurotransmitter system is critically involved in promoting plasticity in auditory cortex. We combined functional magnetic resonance imaging (fMRI) and a pharmacological manipulation to investigate dopaminergic modulation of neural activity in auditory cortex during instrumental learning. Volunteers either received 100 mg L-dopa (Madopar) or placebo in an appetitive, differential instrumental conditioning paradigm, which involved learning that a specific category of frequency modulated tones predicts a monetary reward when fast responses were made in a subsequent reaction time task. The other category of frequency modulated tones was not related to a reward. Our behavioral data provides evidence that dopaminergic stimulation differentially impacts on the speed of instrumental responding in rewarded and unrewarded trials. L-dopa increased neural BOLD activity in left auditory cortex to tones in rewarded and unrewarded trials. This increase was related to plasma L-dopa levels and learning rate. Our data thus provides evidence for dopaminergic modulation of neural activity in auditory cortex, which occurs for both auditory stimuli related to a later reward and those not related to a reward.     

The sensory consequences of speaking: parametric neural cancellation during speech in auditory cortex

When we speak, we provide ourselves with auditory speech input. Efficient monitoring of speech is often hypothesized to depend on matching the predicted sensory consequences from internal motor commands (forward model) with actual sensory feedback. In this paper we tested the forward model hypothesis using functional Magnetic Resonance Imaging. We administered an overt picture naming task in which we parametrically reduced the quality of verbal feedback by noise masking. Presentation of the same auditory input in the absence of overt speech served as listening control condition. Our results suggest that a match between predicted and actual sensory feedback results in inhibition of cancellation of auditory activity because speaking with normal unmasked feedback reduced activity in the auditory cortex compared to listening control conditions. Moreover, during self-generated speech, activation in auditory cortex increased as the feedback quality of the self-generated speech decreased. We conclude that during speaking early auditory cortex is involved in matching external signals with an internally generated model or prediction of sensory consequences, the locus of which may reside in auditory or higher order brain areas. Matching at early auditory cortex may provide a very sensitive monitoring mechanism that highlights speech production errors at very early levels of processing and may efficiently determine the self-agency of speech input.     

Tinnitus Intensity Dependent Gamma Oscillations of the Contralateral Auditory Cortex

Background

Non-pulsatile tinnitus is considered a subjective auditory phantom phenomenon present in 10 to 15% of the population. Tinnitus as a phantom phenomenon is related to hyperactivity and reorganization of the auditory cortex. Magnetoencephalography studies demonstrate a correlation between gamma band activity in the contralateral auditory cortex and the presence of tinnitus. The present study aims to investigate the relation between objective gamma-band activity in the contralateral auditory cortex and subjective tinnitus loudness scores.

Methods and Findings

In unilateral tinnitus patients (N = 15; 10 right, 5 left) source analysis of resting state electroencephalographic gamma band oscillations shows a strong positive correlation with Visual Analogue Scale loudness scores in the contralateral auditory cortex (max r = 0.73, p<0.05).

Conclusion

Auditory phantom percepts thus show similar sound level dependent activation of the contralateral auditory cortex as observed in normal audition. In view of recent consciousness models and tinnitus network models these results suggest tinnitus loudness is coded by gamma band activity in the contralateral auditory cortex but might not, by itself, be responsible for tinnitus perception.     

Spontaneous high-gamma band activity reflects functional organization of auditory cortex in the awake macaque

In the absence of sensory stimuli, spontaneous activity in the brain has been shown to exhibit organization at multiple spatiotemporal scales. In the macaque auditory cortex, responses to acoustic stimuli are tonotopically organized within multiple, adjacent frequency maps aligned in a caudorostral direction on the supratemporal plane (STP) of the lateral sulcus. Here, we used chronic microelectrocorticography to investigate the correspondence between sensory maps and spontaneous neural fluctuations in the auditory cortex. We first mapped tonotopic organization across 96 electrodes spanning approximately two centimeters along the primary and higher auditory cortex. In separate sessions, we then observed that spontaneous activity at the same sites exhibited spatial covariation that reflected the tonotopic map of the STP. This observation demonstrates a close relationship between functional organization and spontaneous neural activity in the sensory cortex of the awake monkey.     

Human auditory cortical activation during self-vocalization

During speaking, auditory feedback is used to adjust vocalizations. The brain systems mediating this integrative ability have been investigated using a wide range of experimental strategies. In this report we examined how vocalization alters speech-sound processing within auditory cortex by directly recording evoked responses to vocalizations and playback stimuli using intracranial electrodes implanted in neurosurgery patients. Several new findings resulted from these high-resolution invasive recordings in human subjects. Suppressive effects of vocalization were found to occur only within circumscribed areas of auditory cortex. In addition, at a smaller number of sites, the opposite pattern was seen; cortical responses were enhanced during vocalization. This increase in activity was reflected in high gamma power changes, but was not evident in the averaged evoked potential waveforms. These new findings support forward models for vocal control in which efference copies of premotor cortex activity modulate sub-regions of auditory cortex.     

Learning-induced plasticity in animal and human auditory cortex

Recent data on learning-related changes in animal and human auditory cortex indicate functions beyond mere stimulus representation and simple recognition memory for stimuli. Rather, auditory cortex seems to process and represent stimuli in a task-dependent fashion. This implies plasticity in neural processing, which can be observed at the level of single neuron firing and the level of spatiotemporal activity patterns in cortical areas. Auditory cortex is a structure in which behaviorally relevant aspects of stimulus processing are highly developed because of the fugitive nature of auditory stimuli.     

Fear conditioning induces guinea pig auditory cortex activation by foot shock alone

The present study used an optical imaging paradigm to investigate plastic changes in the auditory cortex induced by fear conditioning, in which a sound (conditioned stimulus, CS) was paired with an electric foot-shock (unconditioned stimulus, US). We report that, after conditioning, auditory information could be retrieved on the basis of an electric foot-shock alone. Before conditioning, the auditory cortex showed no response to a foot-shock presented in the absence of sound. In contrast, after conditioning, the mere presentation of a foot-shock without any sound succeeded in eliciting activity in the auditory cortex. Additionally, the magnitude of the optical response in the auditory cortex correlated with variation in the electrocardiogram (correlation coefficient: −0.68). The area activated in the auditory cortex, in response to the electric foot-shock, statistically significantly had a larger cross-correlation value for tone response to the CS sound (12 kHz) compared to the non-CS sounds in normal conditioning group. These results suggest that integration of different sensory modalities in the auditory cortex was established by fear conditioning.     

Processing of complex stimuli and natural scenes in the auditory cortex

Neuronal responses in auditory cortex show a fascinating mixture of characteristics that span the range from almost perfect copies of physical aspects of the stimuli to extremely complex context-dependent responses. Fast, highly stimulus-specific adaptation and slower plastic mechanisms work together to constantly adjust neuronal response properties to the statistics of the auditory scene. Evidence with converging implications suggests that the neuronal activity in primary auditory cortex represents sounds in terms of auditory objects rather than in terms of invariant acoustic features.     

Positron Emission Tomography Imaging Reveals Auditory and Frontal Cortical Regions Involved with Speech Perception and Loudness Adaptation

Considerable progress has been made in the treatment of hearing loss with auditory implants. However, there are still many implanted patients that experience hearing deficiencies, such as limited speech understanding or vanishing perception with continuous stimulation (i.e., abnormal loudness adaptation). The present study aims to identify specific patterns of cerebral cortex activity involved with such deficiencies. We performed O-15-water positron emission tomography (PET) in patients implanted with electrodes within the cochlea, brainstem, or midbrain to investigate the pattern of cortical activation in response to speech or continuous multi-tone stimuli directly inputted into the implant processor that then delivered electrical patterns through those electrodes. Statistical parametric mapping was performed on a single subject basis. Better speech understanding was correlated with a larger extent of bilateral auditory cortex activation. In contrast to speech, the continuous multi-tone stimulus elicited mainly unilateral auditory cortical activity in which greater loudness adaptation corresponded to weaker activation and even deactivation. Interestingly, greater loudness adaptation was correlated with stronger activity within the ventral prefrontal cortex, which could be up-regulated to suppress the irrelevant or aberrant signals into the auditory cortex. The ability to detect these specific cortical patterns and differences across patients and stimuli demonstrates the potential for using PET to diagnose auditory function or dysfunction in implant patients, which in turn could guide the development of appropriate stimulation strategies for improving hearing rehabilitation. Beyond hearing restoration, our study also reveals a potential role of the frontal cortex in suppressing irrelevant or aberrant activity within the auditory cortex, and thus may be relevant for understanding and treating tinnitus.     

Auditory cortex basal activity modulates cochlear responses in chinchillas

Background

The auditory efferent system has unique neuroanatomical pathways that connect the cerebral cortex with sensory receptor cells. Pyramidal neurons located in layers V and VI of the primary auditory cortex constitute descending projections to the thalamus, inferior colliculus, and even directly to the superior olivary complex and to the cochlear nucleus. Efferent pathways are connected to the cochlear receptor by the olivocochlear system, which innervates outer hair cells and auditory nerve fibers. The functional role of the cortico-olivocochlear efferent system remains debated. We hypothesized that auditory cortex basal activity modulates cochlear and auditory-nerve afferent responses through the efferent system.

Methodology/Principal Findings

Cochlear microphonics (CM), auditory-nerve compound action potentials (CAP) and auditory cortex evoked potentials (ACEP) were recorded in twenty anesthetized chinchillas, before, during and after auditory cortex deactivation by two methods: lidocaine microinjections or cortical cooling with cryoloops. Auditory cortex deactivation induced a transient reduction in ACEP amplitudes in fifteen animals (deactivation experiments) and a permanent reduction in five chinchillas (lesion experiments). We found significant changes in the amplitude of CM in both types of experiments, being the most common effect a CM decrease found in fifteen animals. Concomitantly to CM amplitude changes, we found CAP increases in seven chinchillas and CAP reductions in thirteen animals. Although ACEP amplitudes were completely recovered after ninety minutes in deactivation experiments, only partial recovery was observed in the magnitudes of cochlear responses.

Conclusions/Significance

These results show that blocking ongoing auditory cortex activity modulates CM and CAP responses, demonstrating that cortico-olivocochlear circuits regulate auditory nerve and cochlear responses through a basal efferent tone. The diversity of the obtained effects suggests that there are at least two functional pathways from the auditory cortex to the cochlea.     

Auditory filters, features, and redundant representations

Responses in auditory cortex tend to be weaker, more phasic, and noisier than those of auditory brainstem and midbrain nuclei. Is the activity in cortex therefore merely a "degraded echo" of lower-level neural representations? In this issue of Neuron, Chechik and colleagues show that, while cortical responses indeed convey less sensory information than auditory midbrain neurons, their responses are also much less redundant.     

Neural responses in human superior temporal cortex support coding of voice representations

The ability to recognize abstract features of voice during auditory perception is an intricate feat of human audition. For the listener, this occurs in near-automatic fashion to seamlessly extract complex cues from a highly variable auditory signal. Voice perception depends on specialized regions of auditory cortex, including superior temporal gyrus (STG) and superior temporal sulcus (STS). However, the nature of voice encoding at the cortical level remains poorly understood. We leverage intracerebral recordings across human auditory cortex during presentation of voice and nonvoice acoustic stimuli to examine voice encoding at the cortical level in 8 patient-participants undergoing epilepsy surgery evaluation. We show that voice selectivity increases along the auditory hierarchy from supratemporal plane (STP) to the STG and STS. Results show accurate decoding of vocalizations from human auditory cortical activity even in the complete absence of linguistic content. These findings show an early, less-selective temporal window of neural activity in the STG and STS followed by a sustained, strongly voice-selective window. Encoding models demonstrate divergence in the encoding of acoustic features along the auditory hierarchy, wherein STG/STS responses are best explained by voice category and acoustics, as opposed to acoustic features of voice stimuli alone. This is in contrast to neural activity recorded from STP, in which responses were accounted for by acoustic features. These findings support a model of voice perception that engages categorical encoding mechanisms within STG and STS to facilitate feature extraction.

Voice perception occurs via specialized networks in higher order auditory cortex, but how voice features are encoded remains a central unanswered question. Using human intracerebral recordings of auditory cortex, this study provides evidence for categorical encoding of voice.     

Functional Imaging of Auditory Cortex in Adult Cats using High-field fMRI

Current knowledge of sensory processing in the mammalian auditory system is mainly derived from electrophysiological studies in a variety of animal models, including monkeys, ferrets, bats, rodents, and cats. In order to draw suitable parallels between human and animal models of auditory function, it is important to establish a bridge between human functional imaging studies and animal electrophysiological studies. Functional magnetic resonance imaging (fMRI) is an established, minimally invasive method of measuring broad patterns of hemodynamic activity across different regions of the cerebral cortex. This technique is widely used to probe sensory function in the human brain, is a useful tool in linking studies of auditory processing in both humans and animals and has been successfully used to investigate auditory function in monkeys and rodents. The following protocol describes an experimental procedure for investigating auditory function in anesthetized adult cats by measuring stimulus-evoked hemodynamic changes in auditory cortex using fMRI. This method facilitates comparison of the hemodynamic responses across different models of auditory function thus leading to a better understanding of species-independent features of the mammalian auditory cortex.     

Cholecystokinin from the entorhinal cortex enables neural plasticity in the auditory cortex

Patients with damage to the medial temporal lobe show deficits in forming new declarative memories but can still recall older memories, suggesting that the medial temporal lobe is necessary for encoding memories in the neocortex. Here, we found that cortical projection neurons in the perirhinal and entorhinal cortices were mostly immunopositive for cholecystokinin (CCK). Local infusion of CCK in the auditory cortex of anesthetized rats induced plastic changes that enabled cortical neurons to potentiate their responses or to start responding to an auditory stimulus that was paired with a tone that robustly triggered action potentials. CCK infusion also enabled auditory neurons to start responding to a light stimulus that was paired with a noise burst. In vivo intracellular recordings in the auditory cortex showed that synaptic strength was potentiated after two pairings of presynaptic and postsynaptic activity in the presence of CCK. Infusion of a CCK_B antagonist in the auditory cortex prevented the formation of a visuo-auditory association in awake rats. Finally, activation of the entorhinal cortex potentiated neuronal responses in the auditory cortex, which was suppressed by infusion of a CCK_B antagonist. Together, these findings suggest that the medial temporal lobe influences neocortical plasticity via CCK-positive cortical projection neurons in the entorhinal cortex.     

Development and plasticity of cortical areas and networks

The development of cortical layers, areas and networks is mediated by a combination of factors that are present in the cortex and are influenced by thalamic input. Electrical activity of thalamocortical afferents has a progressive role in shaping cortex. For early thalamic innervation and patterning, the presence of activity might be sufficient; for features that develop later, such as intracortical networks that mediate emergent responses of cortex, the spatiotemporal pattern of activity often has an instructive role. Experiments that route projections from the retina to the auditory pathway alter the pattern of activity in auditory thalamocortical afferents at a very early stage and reveal the progressive influence of activity on cortical development. Thus, cortical features such as layers and thalamocortical innervation are unaffected, whereas features that develop later, such as intracortical connections, are affected significantly. Surprisingly, the behavioural role of 'rewired' cortex is also influenced profoundly, indicating the importance of patterned activity for this key aspect of cortical function.     

Thalamic activation modulates the responses of neurons in rat primary auditory cortex: an in vivo intracellular recording study

Auditory cortical plasticity can be induced through various approaches. The medial geniculate body (MGB) of the auditory thalamus gates the ascending auditory inputs to the cortex. The thalamocortical system has been proposed to play a critical role in the responses of the auditory cortex (AC). In the present study, we investigated the cellular mechanism of the cortical activity, adopting an in vivo intracellular recording technique, recording from the primary auditory cortex (AI) while presenting an acoustic stimulus to the rat and electrically stimulating its MGB. We found that low-frequency stimuli enhanced the amplitudes of sound-evoked excitatory postsynaptic potentials (EPSPs) in AI neurons, whereas high-frequency stimuli depressed these auditory responses. The degree of this modulation depended on the intensities of the train stimuli as well as the intervals between the electrical stimulations and their paired sound stimulations. These findings may have implications regarding the basic mechanisms of MGB activation of auditory cortical plasticity and cortical signal processing.