Acoustic versus phonetic representation of speech as reflected by the mismatch negativity event-related potential

The concept of categorical perception of speech and speech-like sounds has been central to models of speech perception for decades. Event-related potentials (ERPs) provide a neurophysiologic perspective of this important phenomenon. In the present experiment the mismatch negativity (MMN) event-related potential, which is sensitive to fine acoustic differences, was recorded in adults. Of interest was whether the MMN reflects the acoustic or categorical perception of speech.The MMN was elicited by stimulus pairs (along a continuum varying in place of articulation from /da/ to /ga/) which had been identified as the same phoneme /da/ (within category condition) and as different phonemes /da/ and /ga/ (across categories condition). The acoustic differences between these two pairs of stimuli were equivalent.The MMN was observed in all subjects both in the within and across category conditions. Furthermore, the MMN did not differ in latency, amplitude or area within and across categories. That is, the MMN indicated equal discrimination both across and within categories. These results suggest that the MMN appears to reflect the processing of acoustic aspects of the speech stimulus, but not phonetic processing into categories. The MMN appears to be an extremely sensitive electrophysiologic index of minimal acoustic differences in speech stimuli.     

Decoding Speech Perception by Native and Non-Native Speakers Using Single-Trial Electrophysiological Data

Brain-computer interfaces (BCIs) are systems that use real-time analysis of neuroimaging data to determine the mental state of their user for purposes such as providing neurofeedback. Here, we investigate the feasibility of a BCI based on speech perception. Multivariate pattern classification methods were applied to single-trial EEG data collected during speech perception by native and non-native speakers. Two principal questions were asked: 1) Can differences in the perceived categories of pairs of phonemes be decoded at the single-trial level? 2) Can these same categorical differences be decoded across participants, within or between native-language groups? Results indicated that classification performance progressively increased with respect to the categorical status (within, boundary or across) of the stimulus contrast, and was also influenced by the native language of individual participants. Classifier performance showed strong relationships with traditional event-related potential measures and behavioral responses. The results of the cross-participant analysis indicated an overall increase in average classifier performance when trained on data from all participants (native and non-native). A second cross-participant classifier trained only on data from native speakers led to an overall improvement in performance for native speakers, but a reduction in performance for non-native speakers. We also found that the native language of a given participant could be decoded on the basis of EEG data with accuracy above 80%. These results indicate that electrophysiological responses underlying speech perception can be decoded at the single-trial level, and that decoding performance systematically reflects graded changes in the responses related to the phonological status of the stimuli. This approach could be used in extensions of the BCI paradigm to support perceptual learning during second language acquisition.     

Developmental Changes in Mismatch Responses to Mandarin Consonants and Lexical Tones from Early to Middle Childhood

The purpose of this study was to use mismatch responses (MMRs) to explore the dynamic changes of Mandarin speech perception abilities from early to middle childhood. Twenty preschoolers, 18 school-aged children, and 26 adults participated in this study. Two sets of synthesized speech stimuli varying in Mandarin consonant (alveolo-palatal affricate vs. fricative) and lexical tone features (rising vs. contour tone) were used to examine the developmental course of speech perception abilities. The results indicated that only the adult group demonstrated typical early mismatch negativity (MMN) responses, suggesting that the ability to discriminate specific speech cues in Mandarin consonant and lexical tone is a continuing process in preschool- and school-aged children. Additionally, distinct MMR patterns provided evidence indicating diverse developmental courses to different speech characteristics. By incorporating data from the two speech conditions, we propose using MMR profiles consisting of mismatch negativity (MMN), positive mismatch response (p-MMR), and late discriminative negativity (LDN) as possible brain indices to investigate speech perception development.     

Neural Correlates of Speech Processing in Prelingually Deafened Children and Adolescents with Cochlear Implants

Prelingually deafened children with cochlear implants stand a good chance of developing satisfactory speech performance. Nevertheless, their eventual language performance is highly variable and not fully explainable by the duration of deafness and hearing experience. In this study, two groups of cochlear implant users (CI groups) with very good basic hearing abilities but non-overlapping speech performance (very good or very bad speech performance) were matched according to hearing age and age at implantation. We assessed whether these CI groups differed with regard to their phoneme discrimination ability and auditory sensory memory capacity, as suggested by earlier studies. These functions were measured behaviorally and with the Mismatch Negativity (MMN). Phoneme discrimination ability was comparable in the CI group of good performers and matched healthy controls, which were both better than the bad performers. Source analyses revealed larger MMN activity (155–225 ms) in good than in bad performers, which was generated in the frontal cortex and positively correlated with measures of working memory. For the bad performers, this was followed by an increased activation of left temporal regions from 225 to 250 ms with a focus on the auditory cortex. These results indicate that the two CI groups developed different auditory speech processing strategies and stress the role of phonological functions of auditory sensory memory and the prefrontal cortex in positively developing speech perception and production.     

Mismatch negativity in school-age children to speech stimuli that are just perceptibly different

The mismatch negativity event-related potential (MMN) was elicited in normal school-age children in response to just perceptibly different variants of the speech phoneme /da/. A significant MMN was measured in each subject tested. Child and adult MMNs were similar with respect to peak latency and duration. Measures of MMN magnitude (peak-to-peak amplitude and area) were significantly larger in children than in adults. The results of the present study indicate that the MMN can be elicited in response to minimal acoustic stimulus differences in complex speech signals in school-age children. The results support the feasibility of using the MMN as a tool in the study of deficient auditory perception in children.     

Evidence for the late MMN as a neurophysiological endophenotype for dyslexia

Dyslexia affects 5-10% of school-aged children and is therefore one of the most common learning disorders. Research on auditory event related potentials (AERP), particularly the mismatch negativity (MMN) component, has revealed anomalies in individuals with dyslexia to speech stimuli. Furthermore, candidate genes for this disorder were found through molecular genetic studies. A current challenge for dyslexia research is to understand the interaction between molecular genetics and brain function, and to promote the identification of relevant endophenotypes for dyslexia. The present study examines MMN, a neurophysiological correlate of speech perception, and its potential as an endophenotype for dyslexia in three groups of children. The first group of children was clinically diagnosed with dyslexia, whereas the second group of children was comprised of their siblings who had average reading and spelling skills and were therefore "unaffected" despite having a genetic risk for dyslexia. The third group consisted of control children who were not related to the other groups and were also unaffected. In total, 225 children were included in the study. All children showed clear MMN activity to/da/-/ba/contrasts that could be separated into three distinct MMN components. Whilst the first two MMN components did not differentiate the groups, the late MMN component (300-700 ms) revealed significant group differences. The mean area of the late MMN was attenuated in both the dyslexic children and their unaffected siblings in comparison to the control children. This finding is indicative of analogous alterations of neurophysiological processes in children with dyslexia and those with a genetic risk for dyslexia, without a manifestation of the disorder. The present results therefore further suggest that the late MMN might be a potential endophenotype for dyslexia.     

Relationships between Categorical Perception of Phonemes,Phoneme Awareness,and Visual Attention Span in Developmental Dyslexia

We tested the hypothesis that the categorical perception deficit of speech sounds in developmental dyslexia is related to phoneme awareness skills, whereas a visual attention (VA) span deficit constitutes an independent deficit. Phoneme awareness tasks, VA span tasks and categorical perception tasks of phoneme identification and discrimination using a d/t voicing continuum were administered to 63 dyslexic children and 63 control children matched on chronological age. Results showed significant differences in categorical perception between the dyslexic and control children. Significant correlations were found between categorical perception skills, phoneme awareness and reading. Although VA span correlated with reading, no significant correlations were found between either categorical perception or phoneme awareness and VA span. Mediation analyses performed on the whole dyslexic sample suggested that the effect of categorical perception on reading might be mediated by phoneme awareness. This relationship was independent of the participants’ VA span abilities. Two groups of dyslexic children with a single phoneme awareness or a single VA span deficit were then identified. The phonologically impaired group showed lower categorical perception skills than the control group but categorical perception was similar in the VA span impaired dyslexic and control children. The overall findings suggest that the link between categorical perception, phoneme awareness and reading is independent from VA span skills. These findings provide new insights on the heterogeneity of developmental dyslexia. They suggest that phonological processes and VA span independently affect reading acquisition.     

The mismatch negativity as an index of the perception of speech sounds by the human brain

The present article outlines the contribution of the mismatch negativity (MMN), and its magnetic equivalent MMNm, to our understanding of the perception of speech sounds in the human brain. MMN data indicate that each sound, both speech and non-speech, develops its neural representation corresponding to the percept of this sound in the neurophysiological substrate of auditory sensory memory. The accuracy of this representation, determining the accuracy of the discrimination between different sounds, can be probed with MMN separately for any auditory feature or stimulus type such as phonemes. Furthermore, MMN data show that the perception of phonemes, and probably also of larger linguistic units (syllables and words), is based on language-specific phonetic traces developed in the posterior part of the left-hemisphere auditory cortex. These traces serve as recognition models for the corresponding speech sounds in listening to speech.     

Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex

How natural speech is represented in the auditory cortex constitutes a major challenge for cognitive neuroscience. Although many single-unit and neuroimaging studies have yielded valuable insights about the processing of speech and matched complex sounds, the mechanisms underlying the analysis of speech dynamics in human auditory cortex remain largely unknown. Here, we show that the phase pattern of theta band (4-8 Hz) responses recorded from human auditory cortex with magnetoencephalography (MEG) reliably tracks and discriminates spoken sentences and that this discrimination ability is correlated with speech intelligibility. The findings suggest that an approximately 200 ms temporal window (period of theta oscillation) segments the incoming speech signal, resetting and sliding to track speech dynamics. This hypothesized mechanism for cortical speech analysis is based on the stimulus-induced modulation of inherent cortical rhythms and provides further evidence implicating the syllable as a computational primitive for the representation of spoken language.     

Infant Auditory Processing and Event-related Brain Oscillations

Rapid auditory processing and acoustic change detection abilities play a critical role in allowing human infants to efficiently process the fine spectral and temporal changes that are characteristic of human language. These abilities lay the foundation for effective language acquisition; allowing infants to hone in on the sounds of their native language. Invasive procedures in animals and scalp-recorded potentials from human adults suggest that simultaneous, rhythmic activity (oscillations) between and within brain regions are fundamental to sensory development; determining the resolution with which incoming stimuli are parsed. At this time, little is known about oscillatory dynamics in human infant development. However, animal neurophysiology and adult EEG data provide the basis for a strong hypothesis that rapid auditory processing in infants is mediated by oscillatory synchrony in discrete frequency bands. In order to investigate this, 128-channel, high-density EEG responses of 4-month old infants to frequency change in tone pairs, presented in two rate conditions (Rapid: 70 msec ISI and Control: 300 msec ISI) were examined. To determine the frequency band and magnitude of activity, auditory evoked response averages were first co-registered with age-appropriate brain templates. Next, the principal components of the response were identified and localized using a two-dipole model of brain activity. Single-trial analysis of oscillatory power showed a robust index of frequency change processing in bursts of Theta band (3 - 8 Hz) activity in both right and left auditory cortices, with left activation more prominent in the Rapid condition. These methods have produced data that are not only some of the first reported evoked oscillations analyses in infants, but are also, importantly, the product of a well-established method of recording and analyzing clean, meticulously collected, infant EEG and ERPs. In this article, we describe our method for infant EEG net application, recording, dynamic brain response analysis, and representative results.     

Temporal Feature Perception in Cochlear Implant Users

For the perception of timbre of a musical instrument, the attack time is known to hold crucial information. The first 50 to 150 ms of sound onset reflect the excitation mechanism, which generates the sound. Since auditory processing and music perception in particular are known to be hampered in cochlear implant (CI) users, we conducted an electroencephalography (EEG) study with an oddball paradigm to evaluate the processing of small differences in musical sound onset. The first 60 ms of a cornet sound were manipulated in order to examine whether these differences are detected by CI users and normal-hearing controls (NH controls), as revealed by auditory evoked potentials (AEPs). Our analysis focused on the N1 as an exogenous component known to reflect physical stimuli properties as well as on the P2 and the Mismatch Negativity (MMN). Our results revealed different N1 latencies as well as P2 amplitudes and latencies for the onset manipulations in both groups. An MMN could be elicited only in the NH control group. Together with additional findings that suggest an impact of musical training on CI users’ AEPs, our findings support the view that impaired timbre perception in CI users is at partly due to altered sound onset feature detection.     

Phonetic learning as a pathway to language: new data and native language magnet theory expanded (NLM-e)

Infants' speech perception skills show a dual change towards the end of the first year of life. Not only does non-native speech perception decline, as often shown, but native language speech perception skills show improvement, reflecting a facilitative effect of experience with native language. The mechanism underlying change at this point in development, and the relationship between the change in native and non-native speech perception, is of theoretical interest. As shown in new data presented here, at the cusp of this developmental change, infants' native and non-native phonetic perception skills predict later language ability, but in opposite directions. Better native language skill at 7.5 months of age predicts faster language advancement, whereas better non-native language skill predicts slower advancement. We suggest that native language phonetic performance is indicative of neural commitment to the native language, while non-native phonetic performance reveals uncommitted neural circuitry. This paper has three goals: (i) to review existing models of phonetic perception development, (ii) to present new event-related potential data showing that native and non-native phonetic perception at 7.5 months of age predicts language growth over the next 2 years, and (iii) to describe a revised version of our previous model, the native language magnet model, expanded (NLM-e). NLM-e incorporates five new principles. Specific testable predictions for future research programmes are described.     

Auditory Processing in High-Functioning Adolescents with Autism Spectrum Disorder

Autism Spectrum Disorder (ASD) is a pervasive developmental disorder including abnormalities in perceptual processing. We measure perception in a battery of tests across speech (filtering, phoneme categorization, multisensory integration) and music (pitch memory, meter categorization, harmonic priming). We found that compared to controls, the ASD group showed poorer filtering, less audio-visual integration, less specialization for native phonemic and metrical categories, and a higher instance of absolute pitch. No group differences were found in harmonic priming. Our results are discussed in a developmental framework where culture-specific knowledge acquired early compared to late in development is most impaired, perhaps because of early-accelerated brain growth in ASD. These results suggest that early auditory remediation is needed for good communication and social functioning.     

A sparse neural code for some speech sounds but not for others

The precise neural mechanisms underlying speech sound representations are still a matter of debate. Proponents of 'sparse representations' assume that on the level of speech sounds, only contrastive or otherwise not predictable information is stored in long-term memory. Here, in a passive oddball paradigm, we challenge the neural foundations of such a 'sparse' representation; we use words that differ only in their penultimate consonant ("coronal" [t] vs. "dorsal" [k] place of articulation) and for example distinguish between the German nouns Latz ([lats]; bib) and Lachs ([laks]; salmon). Changes from standard [t] to deviant [k] and vice versa elicited a discernible Mismatch Negativity (MMN) response. Crucially, however, the MMN for the deviant [lats] was stronger than the MMN for the deviant [laks]. Source localization showed this difference to be due to enhanced brain activity in right superior temporal cortex. These findings reflect a difference in phonological 'sparsity': Coronal [t] segments, but not dorsal [k] segments, are based on more sparse representations and elicit less specific neural predictions; sensory deviations from this prediction are more readily 'tolerated' and accordingly trigger weaker MMNs. The results foster the neurocomputational reality of 'representationally sparse' models of speech perception that are compatible with more general predictive mechanisms in auditory perception.     

Pre-stimulus spectral EEG patterns and the visual evoked response

The relationship between the latencies and amplitudes of the N1 and P2 components of the visual evoked potential (VEP) and the psychophysiological state of the brain immediately preceding the time of the stimulus has been investigated in 7 male subjects. Power spectral measures in the delta, theta, alpha and beta bands of the 1 sec pre-stimulus EEG were used to assess the brain state, and low intensity flashes, delivered randomly between 2 and 6 whole seconds, were used as the stimuli. Trials were ranked separately according to the relative amounts of pre-stimulus power in each EEG band and were partitioned into groups by an equal pre-stimulus spectral power criterion. Averaged EPs were computed from these groups and multiple regression analysis was used to relate pre-stimulus spectral power values to EP features. Five of the 7 subjects displayed consistent increases in N1-P2 amplitude as a function of increasing pre-stimulus relative alpha power. The between-subjects effect of pre-stimulus EEG on N1 latency was small, but was moderate for P2 latency (both significant). Both N1 and P2 latency were found to decrease with increasing amounts of pre-stimulus relative delta and theta power.     

Oscillatory Underpinnings of Mismatch Negativity and Their Relationship with Cognitive Function in Patients with Schizophrenia

Background

Impairments in mismatch negativity (MMN) generation have been consistently reported in patients with schizophrenia. However, underlying oscillatory activity of MMN deficits in schizophrenia and the relationship with cognitive impairments have not been investigated in detail. Time-frequency power and phase analyses can provide more detailed measures of brain dynamics of MMN deficits in schizophrenia.

Method

21 patients with schizophrenia and 21 healthy controls were tested with a roving frequency paradigm to generate MMN. Time-frequency domain power and phase-locking (PL) analysis was performed on all trials using short-time Fourier transforms with Hanning window tapering. A comprehensive battery (CANTAB) was used to assess neurocognitive functioning.

Results

Mean MMN amplitude was significantly lower in patients with schizophrenia (95% CI 0.18 - 0.77). Patients showed significantly lower EEG power (95% CI -1.02 - -0.014) in the ~4-7 Hz frequency range (theta band) between 170 and 210 ms. Patients with schizophrenia showed cognitive impairment in multiple domains of CANTAB. However, MMN impairments in amplitude and power were not correlated with clinical measures, medication dose, social functioning or neurocognitive performance.

Conclusion

The findings from this study suggested that while MMN may be a useful marker to probe NMDA receptor mediated mechanisms and associated impairments in gain control and perceptual changes, it may not be a useful marker in association with clinical or cognitive changes. Trial-by-trial EEG power analysis can be used as a measure of brain dynamics underlying MMN deficits which also can have implications for the use of MMN as a biomarker for drug discovery.     

Perception features of emotional intonation of short pseudowords

Reaction time and recognition accuracy of speech emotional intonations in short meaningless words that differed only in one phoneme with background noise and without it were studied in 49 adults of 20-79 years old. The results were compared with the same parameters of emotional intonations in intelligent speech utterances under similar conditions. Perception of emotional intonations at different linguistic levels (phonological and lexico-semantic) was found to have both common features and certain peculiarities. Recognition characteristics of emotional intonations depending on gender and age of listeners appeared to be invariant with regard to linguistic levels of speech stimuli. Phonemic composition of pseudowords was found to influence the emotional perception, especially against the background noise. The most significant stimuli acoustic characteristic responsible for the perception of speech emotional prosody in short meaningless words under the two experimental conditions, i.e. with and without background noise, was the fundamental frequency variation.     

Anatomical correlates of learning novel speech sounds

We examined the relationship between brain anatomy and the ability to learn nonnative speech sounds, as well as rapidly changing and steady-state nonlinguistic sounds, using voxel-based morphometry in 59 healthy adults. Faster phonetic learners appeared to have more white matter in parietal regions, especially in the left hemisphere. The pattern of results was similar for the rapidly changing but not for the steady-state nonlinguistic stimuli, suggesting that morphological correlates of phonetic learning are related to the ability to process rapid temporal variation. Greater asymmetry in the amount of white matter in faster learners may be related to greater myelination allowing more efficient neural processing, which is critical for the ability to process certain speech sounds.     

Integration of consonant and pitch processing as revealed by the absence of additivity in mismatch negativity

Consonants, unlike vowels, are thought to be speech specific and therefore no interactions would be expected between consonants and pitch, a basic element for musical tones. The present study used an electrophysiological approach to investigate whether, contrary to this view, there is integrative processing of consonants and pitch by measuring additivity of changes in the mismatch negativity (MMN) of evoked potentials. The MMN is elicited by discriminable variations occurring in a sequence of repetitive, homogeneous sounds. In the experiment, event-related potentials (ERPs) were recorded while participants heard frequently sung consonant-vowel syllables and rare stimuli deviating in either consonant identity only, pitch only, or in both dimensions. Every type of deviation elicited a reliable MMN. As expected, the two single-deviant MMNs had similar amplitudes, but that of the double-deviant MMN was also not significantly different from them. This absence of additivity in the double-deviant MMN suggests that consonant and pitch variations are processed, at least at a pre-attentive level, in an integrated rather than independent way. Domain-specificity of consonants may depend on higher-level processes in the hierarchy of speech perception.     

Effect of alexithymia on the event-related theta-synchronization of human EEG during reception of emotional visual stimuli

The 62-channel EEG was recorded while control non-alexithymic (n = 21) and alexithymic (n = 20) participants viewed sequentially presented neutral, pleasant and unpleasant pictures and subjectively rated them after each presentation. The event-related synchronization (ERS) to these stimuli was assessed in the theta-1 (4-6 Hr) and theta-2 (6-8 Hz) frequency bands. The obtained findings indicate that alexithymia influences perception of only emotional stimuli. In the upper theta over anterior cortical regions alexithymia vs control individuals in response to both pleasant and unpleasant stimuli manifested decreased left hemisphere ERS in the early test period of 0-200 ms along with enhanced ERS in response to negative vs positive and neutral stimuli in the right hemisphere at 200-600 ms after stimulus onset. The findings provide the first EEG evidence that alexithymia construct, associated with a cognitive deficit in initial evaluation of emotion, is indexed by disrupted early frontal synchronization in the upper theta band that can be best interpreted to reflect disregulation during appraisal of emotional stimuli.