Hearing and seeing meaning in speech and gesture: insights from brain and behaviour

As we speak, we use not only the arbitrary form–meaning mappings of the speech channel but also motivated form–meaning correspondences, i.e. iconic gestures that accompany speech (e.g. inverted V-shaped hand wiggling across gesture space to demonstrate walking). This article reviews what we know about processing of semantic information from speech and iconic gestures in spoken languages during comprehension of such composite utterances. Several studies have shown that comprehension of iconic gestures involves brain activations known to be involved in semantic processing of speech: i.e. modulation of the electrophysiological recording component N400, which is sensitive to the ease of semantic integration of a word to previous context, and recruitment of the left-lateralized frontal–posterior temporal network (left inferior frontal gyrus (IFG), medial temporal gyrus (MTG) and superior temporal gyrus/sulcus (STG/S)). Furthermore, we integrate the information coming from both channels recruiting brain areas such as left IFG, posterior superior temporal sulcus (STS)/MTG and even motor cortex. Finally, this integration is flexible: the temporal synchrony between the iconic gesture and the speech segment, as well as the perceived communicative intent of the speaker, modulate the integration process. Whether these findings are special to gestures or are shared with actions or other visual accompaniments to speech (e.g. lips) or other visual symbols such as pictures are discussed, as well as the implications for a multimodal view of language.     

Acoustic and auditory phonetics: the adaptive design of speech sound systems

Speech perception is remarkably robust. This paper examines how acoustic and auditory properties of vowels and consonants help to ensure intelligibility. First, the source-filter theory of speech production is briefly described, and the relationship between vocal-tract properties and formant patterns is demonstrated for some commonly occurring vowels. Next, two accounts of the structure of preferred sound inventories, quantal theory and dispersion theory, are described and some of their limitations are noted. Finally, it is suggested that certain aspects of quantal and dispersion theories can be unified in a principled way so as to achieve reasonable predictive accuracy.     

Functional imaging of the auditory processing applied to speech sounds

In this paper, we describe domain-general auditory processes that we believe are prerequisite to the linguistic analysis of speech. We discuss biological evidence for these processes and how they might relate to processes that are specific to human speech and language. We begin with a brief review of (i) the anatomy of the auditory system and (ii) the essential properties of speech sounds. Section 4 describes the general auditory mechanisms that we believe are applied to all communication sounds, and how functional neuroimaging is being used to map the brain networks associated with domain-general auditory processing. Section 5 discusses recent neuroimaging studies that explore where such general processes give way to those that are specific to human speech and language.     

The sensory consequences of speaking: parametric neural cancellation during speech in auditory cortex

When we speak, we provide ourselves with auditory speech input. Efficient monitoring of speech is often hypothesized to depend on matching the predicted sensory consequences from internal motor commands (forward model) with actual sensory feedback. In this paper we tested the forward model hypothesis using functional Magnetic Resonance Imaging. We administered an overt picture naming task in which we parametrically reduced the quality of verbal feedback by noise masking. Presentation of the same auditory input in the absence of overt speech served as listening control condition. Our results suggest that a match between predicted and actual sensory feedback results in inhibition of cancellation of auditory activity because speaking with normal unmasked feedback reduced activity in the auditory cortex compared to listening control conditions. Moreover, during self-generated speech, activation in auditory cortex increased as the feedback quality of the self-generated speech decreased. We conclude that during speaking early auditory cortex is involved in matching external signals with an internally generated model or prediction of sensory consequences, the locus of which may reside in auditory or higher order brain areas. Matching at early auditory cortex may provide a very sensitive monitoring mechanism that highlights speech production errors at very early levels of processing and may efficiently determine the self-agency of speech input.     

Auditory perception and the evolution of speech

Among topics related to the evolution of language, the evolution of speech is particularly fascinating. Early theorists believed that it was the ability to produce articulate speech that set the stage for the evolution of the «special» speech processing abilities that exist in modern-day humans. Prior to the evolution of speech production, speech processing abilities were presumed not to exist. The data reviewed here support a different view. Two lines of evidence, one from young human infants and the other from infrahuman species, neither of whom can produce articulate speech, show that in the absence of speech production capabilities, the perception of speech sounds is robust and sophisticated. Human infants and non-human animals evidence auditory perceptual categories that conform to those defined by the phonetic categories of language. These findings suggest the possibility that in evolutionary history the ability to perceive rudimentary speech categories preceded the ability to produce articulate speech. This in turn suggests that it may be audition that structured, at least initially, the formation of phonetic categories.     

Functional topography of cat primary auditory cortex: response latencies

Minimum onset latency (L_min) of single- and multiple-unit responses were mapped in the primary auditory cortex (AI) of barbiturate-anesthetized cats. Contralateral L_min for multiple units was non-homogeneously distributed along the dorso-ventral/isofrequency axis of the AI. Responses with shorter latencies were more often located in the central, more sharply tuned region while longer latencies were more frequently encountered in the dorsal and ventral portions of the AI. For single units, a large scatter of L_min values was found throughout the extent of the AI including cortical depth. The relationship between L_min and previously reported spectral, intensity and temporal parameters was analyzed and revealed statistically significant correlations between minimum onset latency and the following response properties in some but not all studied animals: sharpness of tuning of a frequency response area 10 dB above threshold, broadband transient response, strongest response level, monotonicity of rate/level functions, dynamic range, and preferred frequency modulation sweep direction. This analysis suggests that L_min is determined by several independent factors and that the prediction of L_min based on relationships with other spectral and temporal response properties is inherently weak. The spatial distribution and the functional relationship between these response parameters may provide an important aspect of the time-based cortical representation of specific features in the animal's natural environment. Accepted: 13 August 1997     

Representations of specific acoustic patterns in the auditory cortex and hippocampus

Previous behavioural studies have shown that repeated presentation of a randomly chosen acoustic pattern leads to the unsupervised learning of some of its specific acoustic features. The objective of our study was to determine the neural substrate for the representation of freshly learnt acoustic patterns. Subjects first performed a behavioural task that resulted in the incidental learning of three different noise-like acoustic patterns. During subsequent high-resolution functional magnetic resonance imaging scanning, subjects were then exposed again to these three learnt patterns and to others that had not been learned. Multi-voxel pattern analysis was used to test if the learnt acoustic patterns could be ‘decoded’ from the patterns of activity in the auditory cortex and medial temporal lobe. We found that activity in planum temporale and the hippocampus reliably distinguished between the learnt acoustic patterns. Our results demonstrate that these structures are involved in the neural representation of specific acoustic patterns after they have been learnt.     

Processing of spectral localization-informative changes in sound signals by neurons of inferior colliculus and auditory cortex of the house mouse Mus musculus

Comparative analysis was performed of sensitivity of three populations of neurons of the inferior colliculus central nucleus and of neurons of the auditory cortex A1 and AAF fields of the house mouse Mus musculus to series of signals of wideband noise with spectral notch shifting along the frequency axis and to series of the band noise signals with shifting band. Sensitivity to spectral notches in noise was estimated from a change of impulse activity depending on notch location on the frequency axis (modulation coefficients were determined as the normalized difference between the maximal and minimal spike number in neuronal responses to all noises with notch exposed in the series). It was shown that the highest modulation coefficient values and accordingly the highest frequency-dependent sensitivity to spectral notches in the noise were peculiar to inhibition-dependent inferior colliculus neurons. Statistical analysis confirmed that distribution of modulation coefficients for the group of the inhibition-dependent neurons differed statistically significantly from the distribution for groups of primary-like and V-shaped inferior colliculus neurons as well as of cortical neurons (U-test, p < 0.0001). The lowest sensitivity to spectral notches was revealed in the V-shaped inferior colliculus neurons and cortical neurons; in these groups, distribution of modulation coefficients did not differed statistically significantly (p > 0.3). Thus, although a part of cortical neurons does have the frequency-dependent selectivity to spectral localizationally informative changes in sound signals, its formation needs participation of the inferior colliculus and its inhibition-dependent neurons. Selectivity to direction of the shift of spectral changes in noise signals in neurons of the inferior colliculus and auditory cortex was similar and was manifested mainly as shift along the frequency axis of dependences of the spike number in the neuronal responses and latent periods on central frequency of notch in noise (the noise band).     

青年猫与老年猫初级听皮层GABA_A受体免疫表达的比较研究

目的对5只老年猫(12岁,3-3.5kg)与5只青年猫(2岁,3-3.5kg)初级听皮层(AI)γ-氨基丁酸(gamma-aminobutyric acid,GABA)A受体神经元进行免疫表达比较研究,探索老年猫与青年猫初级听皮层(AI)GABAA受体年龄性变化及产生可影响的生理作用。方法运用免疫组织化学反应与免疫印迹相结合的方法对不同年龄组动物(AI)组织进行染色。光学显微镜下观察、拍照;免疫组织化学阳性反应示GABAA R-IR(GABAA receptor-immunoreaction)神经元形态、密度及分布;免疫印迹示GABAA受体蛋白含量变化。结果老年猫的AI区GABAA R-IR神经元密度比青年猫的GABAA R-IR下降了29.19%,阳性反应强度减弱了20.7%,老年猫阳性反应细胞占神经元总数百分比比青年猫的减少了5.32%;老年猫的GABAA受体蛋白表达量比青年猫的下降了23.16%。结论初级听皮层GABAA受体细胞及受体表达下调可能是老年个体听觉功能减退的重要原因。     

Levels of processing during non-conscious perception: a critical review of visual masking

Understanding the extent and limits of non-conscious processing is an important step on the road to a thorough understanding of the cognitive and cerebral correlates of conscious perception. In this article, we present a critical review of research on subliminal perception during masking and other related experimental conditions. Although initially controversial, the possibility that a broad variety of processes can be activated by a non-reportable stimulus is now well established. Behavioural findings of subliminal priming indicate that a masked word or digit can have an influence on perceptual, lexical and semantic levels, while neuroimaging directly visualizes the brain activation that it evokes in several cortical areas. This activation is often attenuated under subliminal presentation conditions compared to consciously reportable conditions, but there are sufficiently many exceptions, in paradigms such as the attentional blink, to indicate that high activation, per se, is not a sufficient condition for conscious access to occur. We conclude by arguing that for a stimulus to reach consciousness, two factors are jointly needed: (i) the input stimulus must have enough strength (which can be prevented by masking) and (ii) it must receive top-down attention (which can be prevented by drawing attention to another stimulus or task). This view leads to a distinction between two types of non-conscious processes, which we call subliminal and preconscious. According to us, maintaining this distinction is essential in order to make sense of the growing neuroimaging data on the neural correlates of consciousness.     

Pollution going multimodal: the complex impact of the human-altered sensory environment on animal perception and performance

Anthropogenic sensory pollution is affecting ecosystems worldwide. Human actions generate acoustic noise, emanate artificial light and emit chemical substances. All of these pollutants are known to affect animals. Most studies on anthropogenic pollution address the impact of pollutants in unimodal sensory domains. High levels of anthropogenic noise, for example, have been shown to interfere with acoustic signals and cues. However, animals rely on multiple senses, and pollutants often co-occur. Thus, a full ecological assessment of the impact of anthropogenic activities requires a multimodal approach. We describe how sensory pollutants can co-occur and how covariance among pollutants may differ from natural situations. We review how animals combine information that arrives at their sensory systems through different modalities and outline how sensory conditions can interfere with multimodal perception. Finally, we describe how sensory pollutants can affect the perception, behaviour and endocrinology of animals within and across sensory modalities. We conclude that sensory pollution can affect animals in complex ways due to interactions among sensory stimuli, neural processing and behavioural and endocrinal feedback. We call for more empirical data on covariance among sensory conditions, for instance, data on correlated levels in noise and light pollution. Furthermore, we encourage researchers to test animal responses to a full-factorial set of sensory pollutants in the presence or the absence of ecologically important signals and cues. We realize that such approach is often time and energy consuming, but we think this is the only way to fully understand the multimodal impact of sensory pollution on animal performance and perception.     

Apolipoprotein-D expression is increased during development and maturation of the human prefrontal cortex

Apolipoprotein D (apoD) is a lipid binding protein expressed in the brain where its function is largely unknown. Based on changes in lipid metabolism and deposition that occur in the human brain during postnatal development, we investigated changes in apoD expression in the prefrontal cortex in 69 normal cases ranging in age from 40 days to 49 years utilizing gene microarray, quantitative PCR and western blotting methods. In contrast to the high expression of apolipoprotein E ( APOE ), low-density lipoprotein receptor-related protein 8 ( LRP8 ) and 3-hydroxy-3-methyl-glutaryl-CoA reducatase ( HMGCR ) (genes that play a role in lipid-related pathways in brain development) early in life, apoD expression was low in neonates and increased in expression throughout life resulting in six- to eight-fold higher levels at the mRNA and protein levels in adults. Recent studies suggest that apoD has a novel antioxidant function in the brain and we found that the increased apoD expression throughout development and into adulthood was correlated with the expression of antioxidant genes superoxide dismutase 1 ( SOD1 ) and glutathione peroxidase 3 ( GPX3 ) as well as proteins that were modified by the lipid peroxidation end-product 4-hydroxynonenal. These studies reveal that apoD expression is increased throughout life in the human prefrontal cortex and that this is correlated with genetic and biochemical markers of oxidative stress.     

Morphological and physiological differences of the auditory system in three related bushcrickets (Orthoptera: Phaneropteridae, Poecilimon)

Abstract. The auditory system of three closely related bushcrickets was investigated with respect to morphological and physiological differences. The size of the acoustic vesicle in the prothorax cavity and the size of the acoustic spiracle were compared to differences in auditory tuning of the tympanic nerve and differences in the directionality. The results indicate that a small auditory vesicle and auditory spiracle provide reduced sensitivity in the high frequency range (above 10—15 kHz), but increase sensitivity at low frequencies (below 10 kHz). The directionality of the hearing system deteriorates at frequencies between 10 and 25 kHz in species with a small spiracle and trachea. The evolutionary implications of these differences of the auditory systems are discussed. They are considered to be influenced more by ecological factors than bioacoustical ones.     

Perceptual learning in children during perception of the emotional component of speech utterances: The influence of age and gender

Perceptual learning was examined during perception of the emotional component of speech when a valid signal was presented against a background of noise to 7-to 17-year-old boys and girls. Accuracy of emotion recognition (AR) and response time (RT) were recorded in two consecutive test series. Analysis of variance of the results demonstrated a high significance of the series order for both the RT and the AR. The study revealed the ontogenetic features of the characteristics of perceptual learning under these conditions. The most notable changes were observed in the time of discriminating emotions. Mathematically, perceptual learning was modeled exponentially. Model parameters such as the rate of perceptual learning and the initial level of perceptual performance were estimated. It was found that the rate of learning, calculated for both the AR and the RT, had close values and similar trends of changes in both the male and the female samples. This suggests that the rate of perceptual learning is an invariant characteristic of perception of the emotional component of speech in a child. The initial level of perceptual performance increased as a function of age, more notably in the boys than in the girls. The initial level of the RT decreased with age in the girls and almost did not depend on age in the boys. However, the integral AR suggests that the age-related changes in the initial level of the integral perception of emotions are identical in girls and boys.     

Neural evolution in the bat-free habitat of Tahiti: partial regression in an anti-predator auditory system

Noctuid moths endemic to the mountains of Tahiti have evolved in an environment without bats and these insects have lost a defensive behaviour against these predators, the acoustic startle response (ASR). The ASR in noctuid moths is presumed to be activated by a single auditory receptor neuron (A2 cell) and we report that while this cell still exists in endemic species and possesses similar sensitivity thresholds compared to the A2 cell of recently introduced species, it exhibits reduced firing activity to ASR-evoking sounds. This partial neural regression suggests that the evolutionary disappearance of the ASR in these insects is incomplete and that sensoribehavioural integration decays gradually following the removal of stabilizing selective forces.     

Auditory neurones in the brain of the cricket Gryllus bimaculatus (De Geer): Ascending interneurones

Two types of auditory interneurone which ascend from the prothoracic ganglion to the brain in the cricket Gryllus bimaculatus (De Geer) are described. Intracellular recordings were made from the axons of the neurones in the brain under closed-field stimulus conditions and the recorded cells then stained with either cobalt or Lucifer Yellow. Both neurone types—the Plurisegmental ascending low frequency neurone 1 (PALF1), and the Plurisegmental ascending high frequency neurone 1 (PAHF1)—show response characteristics which would make them well suited to encoding the conspecific calling and courtship songs respectively. Further, the projection areas of both neurone types in the brain overlap those of previously identified intraganglionic interneurones, particularly in the anterior-ventral protocerebrum, and it is suggested that an auditory neuropile may exist in this region.     

Postglacial colonization of Europe by the barbastelle bat: agreement between molecular data and past predictive modelling

The barbastelle (Barbastella barbastellus) is a rare forest bat with a wide distribution in Europe. Here, we combine results from the analysis of two mtDNA fragments with species distribution modelling to determine glacial refugia and postglacial colonization routes. We also investigated whether niche conservatism occurs in this species. Glacial refugia were identified in the three southern European peninsulas: Iberia, Italy and the Balkans. These latter two refugia played a major role in the postglacial colonization process, with their populations expanding to England and central Europe, respectively. Palaeo‐distribution models predicted that suitable climatic conditions existed in the inferred refugia during the last glacial maximum (LGM). Nevertheless, the overlap between the current and the LGM distributions was almost inexistent in Italy and in the Balkans, meaning that B. barbastellus populations were forced to shift range between glacial and interglacial periods, a process that probably caused some local extinctions. In contrast, Iberian populations showed a ‘refugia within refugium’ pattern, with two unconnected areas containing stable populations (populations that subsisted during both glacial and interglacial phases). Moreover, the match between LGM models and the refugial areas determined by molecular analysis supported the hypothesis of niche conservatism in B. barbastellus. We argue that geographic patterns of genetic structuring, altogether with the modelling results, indicate the existence of four management units for conservation: Morocco, Iberia, Italy and UK, and Balkans and central Europe. In addition, all countries sampled possessed unique gene pools, thus stressing the need for the conservation of local populations.     

The body beyond the body: expectation of a sensory event is enough to induce ownership over a fake hand

More than 100 papers have been published on the rubber hand illusion since its discovery 14 years ago. The illusion has been proposed as a demonstration that the body is distinguished from other objects by its participation in specific forms of intermodal perceptual correlation. Here, we radically challenge this view by claiming that perceptual correlation is not necessary to produce the experience of this body as mine. Each of 15 participants was seated with his/her right arm resting upon a table just below another smaller table. Thus, the real hand was hidden from the participant''s view and a life-sized rubber model of a right hand was placed on the small table in front of the participant. The participant observed the experimenter''s hand while approaching—without touching—the rubber hand. Phenomenology of the illusion was measured by means of skin conductance response and questionnaire. Both measures indicated that participants experienced the illusion that the experimenter''s hand was about to touch their hidden hand rather than the rubber hand, as if the latter replaced their own hand. This did not occur when the rubber hand was rotated by 180° or replaced by a piece of wood. This illusion indicates that our brain does not build a sense of self in a merely reactive way, via perceptual correlations; rather it generates predictions on what may or may not belong to itself.     

Central projection of auditory receptors in the prothoracic ganglion of the bushcricket Psorodonotus illyricus (tettigoniidae): Computer-aided analysis of the end branch pattern

The projection patterns of morphologically and functionally identified auditory and auditory-vibratory receptor cells of receptor organs (the crista acustica and the intermediate organ) in the foreleg of the tettigoniid Psorodonotus illyricus, were investigated with combined recording and staining techniques, and subsequent histological examination and morphometric measurements. With the application of a computer program (AutoCAD), three-dimensional reconstructions of the axon end branches of receptor cells within the neuropile of the anterior Ring Tract (aRT) were made, in order to determine, the entire shape of each, the pattern and density of the end branches, and the positions of the target areas within the auditory neuropile. Clear differences for different functional types of receptors were found. 1994 John Wiley & Sons, Inc.