High-Resolution,Non-Invasive Imaging of Upper Vocal Tract Articulators Compatible with Human Brain Recordings

A complete neurobiological understanding of speech motor control requires determination of the relationship between simultaneously recorded neural activity and the kinematics of the lips, jaw, tongue, and larynx. Many speech articulators are internal to the vocal tract, and therefore simultaneously tracking the kinematics of all articulators is nontrivial—especially in the context of human electrophysiology recordings. Here, we describe a noninvasive, multi-modal imaging system to monitor vocal tract kinematics, demonstrate this system in six speakers during production of nine American English vowels, and provide new analysis of such data. Classification and regression analysis revealed considerable variability in the articulator-to-acoustic relationship across speakers. Non-negative matrix factorization extracted basis sets capturing vocal tract shapes allowing for higher vowel classification accuracy than traditional methods. Statistical speech synthesis generated speech from vocal tract measurements, and we demonstrate perceptual identification. We demonstrate the capacity to predict lip kinematics from ventral sensorimotor cortical activity. These results demonstrate a multi-modal system to non-invasively monitor articulator kinematics during speech production, describe novel analytic methods for relating kinematic data to speech acoustics, and provide the first decoding of speech kinematics from electrocorticography. These advances will be critical for understanding the cortical basis of speech production and the creation of vocal prosthetics.     

Are Articulatory Settings Mechanically Advantageous for Speech Motor Control?

We address the hypothesis that postures adopted during grammatical pauses in speech production are more “mechanically advantageous” than absolute rest positions for facilitating efficient postural motor control of vocal tract articulators. We quantify vocal tract posture corresponding to inter-speech pauses, absolute rest intervals as well as vowel and consonant intervals using automated analysis of video captured with real-time magnetic resonance imaging during production of read and spontaneous speech by 5 healthy speakers of American English. We then use locally-weighted linear regression to estimate the articulatory forward map from low-level articulator variables to high-level task/goal variables for these postures. We quantify the overall magnitude of the first derivative of the forward map as a measure of mechanical advantage. We find that postures assumed during grammatical pauses in speech as well as speech-ready postures are significantly more mechanically advantageous than postures assumed during absolute rest. Further, these postures represent empirical extremes of mechanical advantage, between which lie the postures assumed during various vowels and consonants. Relative mechanical advantage of different postures might be an important physical constraint influencing planning and control of speech production.     

Masculine Men Articulate Less Clearly

In previous research, acoustic characteristics of the male voice have been shown to signal various aspects of mate quality and threat potential. But the human voice is also a medium of linguistic communication. The present study explores whether physical and vocal indicators of male mate quality and threat potential are linked to effective communicative behaviors such as vowel differentiation and use of more salient phonetic variants of consonants. We show that physical and vocal indicators of male threat potential, height and formant position, are negatively linked to vowel space size, and that height and levels of circulating testosterone are negatively linked to the use of the aspirated variant of the alveolar stop consonant /t/. Thus, taller, more masculine men display less clarity in their speech and prefer phonetic variants that may be associated with masculine attributes such as toughness. These findings suggest that vocal signals of men’s mate quality and/or dominance are not confined to the realm of voice acoustics but extend to other aspects of communicative behavior, even if this means a trade-off with speech patterns that are considered communicatively advantageous, such as clarity and indexical cues to higher social class.     

Sensitive period for sensorimotor integration during vocal motor learning

Sensory experience during sensitive periods in development may direct the organization of neural substrates, thereby permanently influencing subsequent adult behavior. We report a sensitive period during the imitative motor learning phase of sensorimotor integration in birdsong development. By temporarily and reversibly blocking efference to the vocal muscles, we disrupted vocal motor practice during selected stages of song development. Motor disruption during prolonged periods early in development, which allows recovery of vocal control prior to the onset of adult song, has no effect on adult song production. However, song disruption late in development, during the emergence of adult song, results in permanent motor defects in adult song production. These results reveal a decreased ability to compensate for interference with motor function when disturbances occur during the terminal stage of vocal motor development. Temporary disruption of syringeal motor control in adults does not produce permanent changes in song production. Permanent vocal aberrations in juveniles are evident exclusively in learned song elements rather than nonlearned calls, suggesting that the sensitive period is associated with motor learning.     

Songbirds use pulse tone register in two voices to generate low-frequency sound

The principal physical mechanism of sound generation is similar in songbirds and humans, despite large differences in their vocal organs. Whereas vocal fold dynamics in the human larynx are well characterized, the vibratory behaviour of the sound-generating labia in the songbird vocal organ, the syrinx, is unknown. We present the first high-speed video records of the intact syrinx during induced phonation. The syrinx of anaesthetized crows shows a vibration pattern of the labia similar to that of the human vocal fry register. Acoustic pulses result from short opening of the labia, and pulse generation alternates between the left and right sound sources. Spontaneously calling crows can also generate similar pulse characteristics with only one sound generator. Airflow recordings in zebra finches and starlings show that pulse tone sounds can be generated unilaterally, synchronously or by alternating between the two sides. Vocal fry-like dynamics therefore represent a common production mechanism for low-frequency sounds in songbirds. These results also illustrate that complex vibration patterns can emerge from the mechanical properties of the coupled sound generators in the syrinx. The use of vocal fry-like dynamics in the songbird syrinx extends the similarity to this unusual vocal register with mammalian sound production mechanisms.     

Prosthetic avian vocal organ controlled by a freely behaving bird based on a low dimensional model of the biomechanical periphery

Because of the parallels found with human language production and acquisition, birdsong is an ideal animal model to study general mechanisms underlying complex, learned motor behavior. The rich and diverse vocalizations of songbirds emerge as a result of the interaction between a pattern generator in the brain and a highly nontrivial nonlinear periphery. Much of the complexity of this vocal behavior has been understood by studying the physics of the avian vocal organ, particularly the syrinx. A mathematical model describing the complex periphery as a nonlinear dynamical system leads to the conclusion that nontrivial behavior emerges even when the organ is commanded by simple motor instructions: smooth paths in a low dimensional parameter space. An analysis of the model provides insight into which parameters are responsible for generating a rich variety of diverse vocalizations, and what the physiological meaning of these parameters is. By recording the physiological motor instructions elicited by a spontaneously singing muted bird and computing the model on a Digital Signal Processor in real-time, we produce realistic synthetic vocalizations that replace the bird's own auditory feedback. In this way, we build a bio-prosthetic avian vocal organ driven by a freely behaving bird via its physiologically coded motor commands. Since it is based on a low-dimensional nonlinear mathematical model of the peripheral effector, the emulation of the motor behavior requires light computation, in such a way that our bio-prosthetic device can be implemented on a portable platform.     

Temporal recalibration in vocalization induced by adaptation of delayed auditory feedback

Background

We ordinarily perceive our voice sound as occurring simultaneously with vocal production, but the sense of simultaneity in vocalization can be easily interrupted by delayed auditory feedback (DAF). DAF causes normal people to have difficulty speaking fluently but helps people with stuttering to improve speech fluency. However, the underlying temporal mechanism for integrating the motor production of voice and the auditory perception of vocal sound remains unclear. In this study, we investigated the temporal tuning mechanism integrating vocal sensory and voice sounds under DAF with an adaptation technique.

Methods and Findings

Participants produced a single voice sound repeatedly with specific delay times of DAF (0, 66, 133 ms) during three minutes to induce ‘Lag Adaptation’. They then judged the simultaneity between motor sensation and vocal sound given feedback. We found that lag adaptation induced a shift in simultaneity responses toward the adapted auditory delays. This indicates that the temporal tuning mechanism in vocalization can be temporally recalibrated after prolonged exposure to delayed vocal sounds. Furthermore, we found that the temporal recalibration in vocalization can be affected by averaging delay times in the adaptation phase.

Conclusions

These findings suggest vocalization is finely tuned by the temporal recalibration mechanism, which acutely monitors the integration of temporal delays between motor sensation and vocal sound.     

Response-Locked Brain Dynamics of Word Production

The cortical regions involved in the different stages of speech production are relatively well-established, but their spatio-temporal dynamics remain poorly understood. In particular, the available studies have characterized neural events with respect to the onset of the stimulus triggering a verbal response. The core aspect of language production, however, is not perception but action. In this context, the most relevant question may not be how long after a stimulus brain events happen, but rather how long before the production act do they occur. We investigated speech production-related brain activity time-locked to vocal onset, in addition to the common stimulus-locked approach. We report the detailed temporal interplay between medial and left frontal activities occurring shortly before vocal onset. We interpret those as reflections of, respectively, word selection and word production processes. This medial-lateral organization is in line with that described in non-linguistic action control, suggesting that similar processes are at play in word production and non-linguistic action production. This novel view of the brain dynamics underlying word production provides a useful background for future investigations of the spatio-temporal brain dynamics that lead to the production of verbal responses.     

Specialized Motor-Driven dusp1 Expression in the Song Systems of Multiple Lineages of Vocal Learning Birds

Mechanisms for the evolution of convergent behavioral traits are largely unknown. Vocal learning is one such trait that evolved multiple times and is necessary in humans for the acquisition of spoken language. Among birds, vocal learning is evolved in songbirds, parrots, and hummingbirds. Each time similar forebrain song nuclei specialized for vocal learning and production have evolved. This finding led to the hypothesis that the behavioral and neuroanatomical convergences for vocal learning could be associated with molecular convergence. We previously found that the neural activity-induced gene dual specificity phosphatase 1 (dusp1) was up-regulated in non-vocal circuits, specifically in sensory-input neurons of the thalamus and telencephalon; however, dusp1 was not up-regulated in higher order sensory neurons or motor circuits. Here we show that song motor nuclei are an exception to this pattern. The song nuclei of species from all known vocal learning avian lineages showed motor-driven up-regulation of dusp1 expression induced by singing. There was no detectable motor-driven dusp1 expression throughout the rest of the forebrain after non-vocal motor performance. This pattern contrasts with expression of the commonly studied activity-induced gene egr1, which shows motor-driven expression in song nuclei induced by singing, but also motor-driven expression in adjacent brain regions after non-vocal motor behaviors. In the vocal non-learning avian species, we found no detectable vocalizing-driven dusp1 expression in the forebrain. These findings suggest that independent evolutions of neural systems for vocal learning were accompanied by selection for specialized motor-driven expression of the dusp1 gene in those circuits. This specialized expression of dusp1 could potentially lead to differential regulation of dusp1-modulated molecular cascades in vocal learning circuits.     

Songbird: a unique animal model for studying the molecular basis of disorders of vocal development and communication

Like humans, songbirds are one of the few animal groups that learn vocalization. Vocal learning requires coordination of auditory input and vocal output using auditory feedback to guide one’s own vocalizations during a specific developmental stage known as the critical period. Songbirds are good animal models for understand the neural basis of vocal learning, a complex form of imitation, because they have many parallels to humans with regard to the features of vocal behavior and neural circuits dedicated to vocal learning. In this review, we will summarize the behavioral, neural, and genetic traits of birdsong. We will also discuss how studies of birdsong can help us understand how the development of neural circuits for vocal learning and production is driven by sensory input (auditory information) and motor output (vocalization).     

Motor mechanisms of a vocal mimic: implications for birdsong production

The diverse vocal performances of oscine songbirds are produced by the independent but coordinated patterns of activity in muscles controlling separate sound generators on the left and right sides of their duplex vocal organ, the syrinx. Species with different song styles use the two sides of their syrinx in different ways to produce their species-typical songs. Understanding how a vocal mimic copies another species' song may provide an insight into whether there are alternative motor mechanisms for generating the model's song and what parts of his song are most difficult to produce. We show here that when a vocal mimic, the northern mockingbird, accurately copies the song of another species it also uses the vocal motor pattern employed by the model species. Deviations from the model's production mechanism result in predictable differences in the mockingbird's song. Species-specific acoustic features of the model seem most difficult to copy, suggesting that they have been exposed to the strongest selective pressure to maximize their performance.     

Sensory-motor interactions modulate a primate vocal behavior: antiphonal calling in common marmosets

A fundamental issue in neuroscience pertains to how different cortical systems interact to generate behavior. One of the most direct ways to address this issue is to investigate how sensory information is encoded and used to produce a motor response. Antiphonal calling is a natural vocal behavior that involves individuals producing their species-specific long distance vocalization in response to hearing the same call and engages both the auditory and motor systems, as well as the cognitive neural systems involved in decision making and categorization. Here we present results from a series of behavioral experiments investigating the auditory–vocal interactions during antiphonal calling in the common marmoset (Callithrix jacchus). We manipulated sensory input by placing subjects in different social contexts and found that the auditory input had a significant effect on call timing and propensity to call. Playback experiments tested the significance of the timing of vocal production in antiphonal calling and showed that a short latency between antiphonal calls was necessary to maintain reciprocal vocal interactions. Overall, this study shows that sensory-motor interactions can be experimentally induced and manipulated in a natural primate vocal behavior. Antiphonal calling represents a promising model system to examine these issues in non-human primates at both the behavioral and neural levels.     

Convergent differential regulation of parvalbumin in the brains of vocal learners

Spoken language and learned song are complex communication behaviors found in only a few species, including humans and three groups of distantly related birds--songbirds, parrots, and hummingbirds. Despite their large phylogenetic distances, these vocal learners show convergent behaviors and associated brain pathways for vocal communication. However, it is not clear whether this behavioral and anatomical convergence is associated with molecular convergence. Here we used oligo microarrays to screen for genes differentially regulated in brain nuclei necessary for producing learned vocalizations relative to adjacent brain areas that control other behaviors in avian vocal learners versus vocal non-learners. A top candidate gene in our screen was a calcium-binding protein, parvalbumin (PV). In situ hybridization verification revealed that PV was expressed significantly higher throughout the song motor pathway, including brainstem vocal motor neurons relative to the surrounding brain regions of all distantly related avian vocal learners. This differential expression was specific to PV and vocal learners, as it was not found in avian vocal non-learners nor for control genes in learners and non-learners. Similar to the vocal learning birds, higher PV up-regulation was found in the brainstem tongue motor neurons used for speech production in humans relative to a non-human primate, macaques. These results suggest repeated convergent evolution of differential PV up-regulation in the brains of vocal learners separated by more than 65-300 million years from a common ancestor and that the specialized behaviors of learned song and speech may require extra calcium buffering and signaling.     

A neural network model of speech acquisition and motor equivalent speech production

This article describes a neural network model that addresses the acquisition of speaking skills by infants and subsequent motor equivalent production of speech sounds. The model learns two mappings during a babbling phase. A phonetic-to-orosensory mapping specifies a vocal tract target for each speech sound; these targets take the form of convex regions in orosensory coordinates defining the shape of the vocal tract. The babbling process wherein these convex region targets are formed explains how an infant can learn phoneme-specific and language-specific limits on acceptable variability of articulator movements. The model also learns an orosensory-to-articulatory mapping wherein cells coding desired movement directions in orosensory space learn articulator movements that achieve these orosensory movement directions. The resulting mapping provides a natural explanation for the formation of coordinative structures. This mapping also makes efficient use of redundancy in the articulator system, thereby providing the model with motor equivalent capabilities. Simulations verify the model's ability to compensate for constraints or perturbations applied to the articulators automatically and without new learning and to explain contextual variability seen in human speech production.Supported in part by AFOSR F49620-92-J-0499     

Of Mice,Birds, and Men: The Mouse Ultrasonic Song System Has Some Features Similar to Humans and Song-Learning Birds

Humans and song-learning birds communicate acoustically using learned vocalizations. The characteristic features of this social communication behavior include vocal control by forebrain motor areas, a direct cortical projection to brainstem vocal motor neurons, and dependence on auditory feedback to develop and maintain learned vocalizations. These features have so far not been found in closely related primate and avian species that do not learn vocalizations. Male mice produce courtship ultrasonic vocalizations with acoustic features similar to songs of song-learning birds. However, it is assumed that mice lack a forebrain system for vocal modification and that their ultrasonic vocalizations are innate. Here we investigated the mouse song system and discovered that it includes a motor cortex region active during singing, that projects directly to brainstem vocal motor neurons and is necessary for keeping song more stereotyped and on pitch. We also discovered that male mice depend on auditory feedback to maintain some ultrasonic song features, and that sub-strains with differences in their songs can match each other''s pitch when cross-housed under competitive social conditions. We conclude that male mice have some limited vocal modification abilities with at least some neuroanatomical features thought to be unique to humans and song-learning birds. To explain our findings, we propose a continuum hypothesis of vocal learning.     

Temporal patterning of song production: Participation of nucleus uvaeformis of the thalamus

Birdsong is a learned vocal behavior used in intraspecific communication. The motor pathway serving learned vocalizations includes the forebrain nuclei NIf, HVC, and RA; RA projects to midbrain and brain stem areas that control the temporal and acoustic features of song. Nucleus Uvaeformis of the thalamus (Uva) sends input to two of these forebrain nuclei (NIf and HVC) but has not been thought to be important for song production. We used three experimental approaches to reexamine Uva's function in adult male zebra finches. (1) Electrical stimulation applied to Uva activated HVC and the vocal motor pathway, including tracheosyringeal motor neurons that innervate the bird's vocal organ. (2) Bilateral lesions of Uva including the dorso-medial portion of the nucleus affected the normal temporal organization of song. (3) Chronic multiunit recordings from Uva during normal song and calls show bursts of premotor activity that lead the onset of some song components, and also larger bursts that mark the end of complete song motifs. These results implicate Uva in the production of learned vocalizations, and further suggest that Uva contributes more to the temporal structure than to the acoustic characteristics of song. © 1993 John Wiley & Sons, Inc.     

A Lightweight,Headphones-based System for Manipulating Auditory Feedback in Songbirds

Experimental manipulations of sensory feedback during complex behavior have provided valuable insights into the computations underlying motor control and sensorimotor plasticity¹. Consistent sensory perturbations result in compensatory changes in motor output, reflecting changes in feedforward motor control that reduce the experienced feedback error. By quantifying how different sensory feedback errors affect human behavior, prior studies have explored how visual signals are used to recalibrate arm movements^2,3 and auditory feedback is used to modify speech production^4-7. The strength of this approach rests on the ability to mimic naturalistic errors in behavior, allowing the experimenter to observe how experienced errors in production are used to recalibrate motor output.Songbirds provide an excellent animal model for investigating the neural basis of sensorimotor control and plasticity^8,9. The songbird brain provides a well-defined circuit in which the areas necessary for song learning are spatially separated from those required for song production, and neural recording and lesion studies have made significant advances in understanding how different brain areas contribute to vocal behavior^9-12. However, the lack of a naturalistic error-correction paradigm - in which a known acoustic parameter is perturbed by the experimenter and then corrected by the songbird - has made it difficult to understand the computations underlying vocal learning or how different elements of the neural circuit contribute to the correction of vocal errors¹³.The technique described here gives the experimenter precise control over auditory feedback errors in singing birds, allowing the introduction of arbitrary sensory errors that can be used to drive vocal learning. Online sound-processing equipment is used to introduce a known perturbation to the acoustics of song, and a miniaturized headphones apparatus is used to replace a songbird''s natural auditory feedback with the perturbed signal in real time. We have used this paradigm to perturb the fundamental frequency (pitch) of auditory feedback in adult songbirds, providing the first demonstration that adult birds maintain vocal performance using error correction¹⁴. The present protocol can be used to implement a wide range of sensory feedback perturbations (including but not limited to pitch shifts) to investigate the computational and neurophysiological basis of vocal learning.     

Calls out of chaos: the adaptive significance of nonlinear phenomena in mammalian vocal production

Recent work on human vocal production demonstrates that certain irregular phenomena seen in human pathological voices and baby crying result from nonlinearities in the vocal production system. Equivalent phenomena are quite common in nonhuman mammal vocal repertoires. In particular, bifurcations and chaos are ubiquitous aspects of the normal adult repertoire in many primate species. Here we argue that these phenomena result from properties inherent in the peripheral production mechanism, which allows individuals to generate highly complex and unpredictable vocalizations without requiring equivalently complex neural control mechanisms. We provide examples from the vocal repertoire of rhesus macaques, Macaca mulatta, and other species illustrating the different classes of nonlinear phenomena, and review the concepts from nonlinear dynamics that explicate these calls. Finally, we discuss the evolutionary significance of nonlinear vocal phenomena. We suggest that nonlinear phenomena may subserve individual recognition and the estimation of size or fluctuating asymmetry from vocalizations. Furthermore, neurally ‘cheap’ unpredictability may serve the valuable adaptive function of making chaotic calls difficult to predict and ignore. While noting that nonlinear phenomena are in some cases probably nonadaptive by-products of the physics of the sound-generating mechanism, we suggest that these functional hypotheses provide at least a partial explanation for the ubiquity of nonlinear calls in nonhuman vocal repertoires.     

Modeling Consonant-Vowel Coarticulation for Articulatory Speech Synthesis

A central challenge for articulatory speech synthesis is the simulation of realistic articulatory movements, which is critical for the generation of highly natural and intelligible speech. This includes modeling coarticulation, i.e., the context-dependent variation of the articulatory and acoustic realization of phonemes, especially of consonants. Here we propose a method to simulate the context-sensitive articulation of consonants in consonant-vowel syllables. To achieve this, the vocal tract target shape of a consonant in the context of a given vowel is derived as the weighted average of three measured and acoustically-optimized reference vocal tract shapes for that consonant in the context of the corner vowels /a/, /i/, and /u/. The weights are determined by mapping the target shape of the given context vowel into the vowel subspace spanned by the corner vowels. The model was applied for the synthesis of consonant-vowel syllables with the consonants /b/, /d/, /g/, /l/, /r/, /m/, /n/ in all combinations with the eight long German vowels. In a perception test, the mean recognition rate for the consonants in the isolated syllables was 82.4%. This demonstrates the potential of the approach for highly intelligible articulatory speech synthesis.