Implicit multisensory associations influence voice recognition

Natural objects provide partially redundant information to the brain through different sensory modalities. For example, voices and faces both give information about the speech content, age, and gender of a person. Thanks to this redundancy, multimodal recognition is fast, robust, and automatic. In unimodal perception, however, only part of the information about an object is available. Here, we addressed whether, even under conditions of unimodal sensory input, crossmodal neural circuits that have been shaped by previous associative learning become activated and underpin a performance benefit. We measured brain activity with functional magnetic resonance imaging before, while, and after participants learned to associate either sensory redundant stimuli, i.e. voices and faces, or arbitrary multimodal combinations, i.e. voices and written names, ring tones, and cell phones or brand names of these cell phones. After learning, participants were better at recognizing unimodal auditory voices that had been paired with faces than those paired with written names, and association of voices with faces resulted in an increased functional coupling between voice and face areas. No such effects were observed for ring tones that had been paired with cell phones or names. These findings demonstrate that brief exposure to ecologically valid and sensory redundant stimulus pairs, such as voices and faces, induces specific multisensory associations. Consistent with predictive coding theories, associative representations become thereafter available for unimodal perception and facilitate object recognition. These data suggest that for natural objects effective predictive signals can be generated across sensory systems and proceed by optimization of functional connectivity between specialized cortical sensory modules.     

Adaptation Aftereffects in Vocal Emotion Perception Elicited by Expressive Faces and Voices

The perception of emotions is often suggested to be multimodal in nature, and bimodal as compared to unimodal (auditory or visual) presentation of emotional stimuli can lead to superior emotion recognition. In previous studies, contrastive aftereffects in emotion perception caused by perceptual adaptation have been shown for faces and for auditory affective vocalization, when adaptors were of the same modality. By contrast, crossmodal aftereffects in the perception of emotional vocalizations have not been demonstrated yet. In three experiments we investigated the influence of emotional voice as well as dynamic facial video adaptors on the perception of emotion-ambiguous voices morphed on an angry-to-happy continuum. Contrastive aftereffects were found for unimodal (voice) adaptation conditions, in that test voices were perceived as happier after adaptation to angry voices, and vice versa. Bimodal (voice + dynamic face) adaptors tended to elicit larger contrastive aftereffects. Importantly, crossmodal (dynamic face) adaptors also elicited substantial aftereffects in male, but not in female participants. Our results (1) support the idea of contrastive processing of emotions (2), show for the first time crossmodal adaptation effects under certain conditions, consistent with the idea that emotion processing is multimodal in nature, and (3) suggest gender differences in the sensory integration of facial and vocal emotional stimuli.     

Auditory-visual crossmodal integration in perception of face gender

Whereas extensive neuroscientific and behavioral evidence has confirmed a role of auditory-visual integration in representing space [1-6], little is known about the role of auditory-visual integration in object perception. Although recent neuroimaging results suggest integrated auditory-visual object representations [7-11], substantiating behavioral evidence has been lacking. We demonstrated auditory-visual integration in the perception of face gender by using pure tones that are processed in low-level auditory brain areas and that lack the spectral components that characterize human vocalization. When androgynous faces were presented together with pure tones in the male fundamental-speaking-frequency range, faces were more likely to be judged as male, whereas when faces were presented with pure tones in the female fundamental-speaking-frequency range, they were more likely to be judged as female. Importantly, when participants were explicitly asked to attribute gender to these pure tones, their judgments were primarily based on relative pitch and were uncorrelated with the male and female fundamental-speaking-frequency ranges. This perceptual dissociation of absolute-frequency-based crossmodal-integration effects from relative-pitch-based explicit perception of the tones provides evidence for a sensory integration of auditory and visual signals in representing human gender. This integration probably develops because of concurrent neural processing of visual and auditory features of gender.     

Adaptor Identity Modulates Adaptation Effects in Familiar Face Identification and Their Neural Correlates

Adaptation-related aftereffects (AEs) show how face perception can be altered by recent perceptual experiences. Along with contrastive behavioural biases, modulations of the early event-related potentials (ERPs) were typically reported on categorical levels. Nevertheless, the role of the adaptor stimulus per se for face identity-specific AEs is not completely understood and was therefore investigated in the present study. Participants were adapted to faces (S1s) varying systematically on a morphing continuum between pairs of famous identities (identities A and B), or to Fourier phase-randomized faces, and had to match the subsequently presented ambiguous faces (S2s; 50/50% identity A/B) to one of the respective original faces. We found that S1s identical with or near to the original identities led to strong contrastive biases with more identity B responses following A adaptation and vice versa. In addition, the closer S1s were to the 50/50% S2 on the morphing continuum, the smaller the magnitude of the AE was. The relation between S1s and AE was, however, not linear. Additionally, stronger AEs were accompanied by faster reaction times. Analyses of the simultaneously recorded ERPs revealed categorical adaptation effects starting at 100 ms post-stimulus onset, that were most pronounced at around 125–240 ms for occipito-temporal sites over both hemispheres. S1-specific amplitude modulations were found at around 300–400 ms. Response-specific analyses of ERPs showed reduced voltages starting at around 125 ms when the S1 biased perception in a contrastive way as compared to when it did not. Our results suggest that face identity AEs do not only depend on physical differences between S1 and S2, but also on perceptual factors, such as the ambiguity of S1. Furthermore, short-term plasticity of face identity processing might work in parallel to object-category processing, and is reflected in the first 400 ms of the ERP.     

Impairments of Biological Motion Perception in Congenital Prosopagnosia

Prosopagnosia is a deficit in recognizing people from their faces. Acquired prosopagnosia results after brain damage, developmental or congenital prosopagnosia (CP) is not caused by brain lesion, but has presumably been present from early childhood onwards. Since other sensory, perceptual, and cognitive abilities are largely spared, CP is considered to be a stimulus-specific deficit, limited to face processing. Given that recent behavioral and imaging studies indicate a close relationship of face and biological-motion perception in healthy adults, we hypothesized that biological motion processing should be impaired in CP. Five individuals with CP and ten matched healthy controls were tested with diverse biological-motion stimuli and tasks. Four of the CP individuals showed severe deficits in biological-motion processing, while one performed within the lower range of the controls. A discriminant analysis classified all participants correctly with a very high probability for each participant. These findings demonstrate that in CP, impaired perception of faces can be accompanied by impaired biological-motion perception. We discuss implications for dedicated and shared mechanisms involved in the perception of faces and biological motion.     

The many faces of research on face perception

Face perception is fundamental to human social interaction. Many different types of important information are visible in faces and the processes and mechanisms involved in extracting this information are complex and can be highly specialized. The importance of faces has long been recognized by a wide range of scientists. Importantly, the range of perspectives and techniques that this breadth has brought to face perception research has, in recent years, led to many important advances in our understanding of face processing. The articles in this issue on face perception each review a particular arena of interest in face perception, variously focusing on (i) the social aspects of face perception (attraction, recognition and emotion), (ii) the neural mechanisms underlying face perception (using brain scanning, patient data, direct stimulation of the brain, visual adaptation and single-cell recording), and (iii) comparative aspects of face perception (comparing adult human abilities with those of chimpanzees and children). Here, we introduce the central themes of the issue and present an overview of the articles.     

Ignored Faces Produce Figural Face Aftereffects

Following adaptation to faces with contracted (or expanded) internal features, faces previously perceived as normal appear distorted in the opposite direction. This figural face aftereffect suggests face-coding mechanisms adapt to changes in the spatial relations of features and/or the global structure of faces. Here, we investigated whether the figural aftereffect requires spatial attention. Participants ignored a distorted adapting face and performed a highly demanding letter-count task. Before and after adaptation, participants rated the normality of morphed distorted faces ranging from 50% contracted through undistorted to 50% expanded. A robust aftereffect was observed. These results suggest that the figural face aftereffect can occur in the absence of spatial attention, even when the attentional demands of the relevant task are high.     

面孔分类中空间高低频表征的神经机制：一个颅内脑电研究

来自多方面的研究表明,面孔的分类和识别位于特定脑区．同时,已有行为实验研究表明,图像的空间高低频特征在面孔分类的不同范畴中起不同的贡献,例如身份更多被低频信号传递,性别被高低频共同传递,而表情更多被高频传递．然而,空间频率在面孔分类中的贡献,其表征和神经机制目前相关研究很少．利用特定癫痫患者植入颅内电极的监控期,呈现不同类型面孔图像,同时记录其颅内脑电,用事件相关电位方法考察了据认为是面孔特定成分的相关电位的潜伏期在170 ms的波形(N170波形)的变化;用电极反应显著性分析考察了空间频率在不同分类特征上的贡献．结果表明,空间高频(HSF)图像的N170潜伏期显著延迟．只呈现空间低频(LSF)图像,N170的潜伏期对普通人面孔会延迟,而对熟悉的名人则没有这个差异．女性面孔诱发的N170在HSF条件下潜伏期明显晚于LSF条件,而男性面孔诱发的波形则不存在这个差异．表情在N170上没有体现出任何差异．但是基于电极的显著性分析表明,有更多的额叶电极参与了表情的加工;身份特征加工有更多电极在空间低频上表现出差异,而性别加工则空间高低频比较平衡．与以往行为结果不同的是,表情加工也有更多低频贡献,而且表情的差异可以在早达114 ms的时候就发生．这符合表情信息在颞枕区域有一个快速基本加工,再传递到其他脑区的认知模型．因此,空间高低频信息在身份和性别上的贡献,可能发生在经典的面孔加工脑区,由N170表达,表情信息不由N170表达,而是在颞枕较广泛的范围内快速加工再传递到别的脑区,如额叶．这是首次利用颅内脑电就空间频率在面孔分类中的贡献的神经机制进行研究,为深入理解脑内面孔各种特征加工的动态过程提供了一个新的切入点．     

Auditory and action semantic features activate sensory-specific perceptual brain regions

Traditionally, concepts were considered propositional, amodal, and verbal in nature (for review, see ). Recent findings, however, suggest that conceptual knowledge is divisible into different types (L. Wu and L.W. Barsalou, personal communication, ) and that each type may be linked to specific sensory and motor processes. This implies that sensory processing regions of the brain may also process concepts. In fact, there is some neuroimaging evidence that conceptual information does activate perceptual brain regions and that there is a correspondence between knowledge type and the region being activated. In the following experiment, using a training technique developed in previous studies, participants verbally learned associations between novel objects and conceptual features. The objective was to create objects that were associated with features from only one knowledge type, something that does not occur with common objects. During a visual task that did not require retrieval of learned associations, the superior temporal gyrus, which responds well to sounds, was preferentially activated by objects associated with auditory features (e.g., buzzes). Likewise, the posterior superior temporal sulcus, which responds well to motion, was preferentially activated by objects associated with "action" features (e.g., hops). These findings support the theory that knowledge is grounded in perception.     

Visual Adaptation to Thin and Fat Bodies Transfers across Identity

Visual perception is highly variable and can be influenced by the surrounding world. Previous research has revealed that body perception can be biased due to adaptation to thin or fat body shapes. The aim of the present study was to show that adaptation to certain body shapes and the resulting perceptual biases transfer across different identities of adaptation and test stimuli. We designed two similar adaptation experiments in which healthy female participants adapted to pictures of either thin or fat bodies and subsequently compared more or less distorted pictures of their own body to their actual body shape. In the first experiment (n = 16) the same identity was used as adaptation and test stimuli (i.e. pictures of the participant’s own body) while in the second experiment (n = 16) we used pictures of unfamiliar thin or fat bodies as adaptation stimuli. We found comparable adaptation effects in both experiments: After adaptation to a thin body, participants rated a thinner than actual body picture to be the most realistic and vice versa. We therefore assume that adaptation to certain body shapes transfers across different identities. These results raise the questions of whether some type of natural adaptation occurs in everyday life. Natural and predominant exposure to certain bodily features like body shape – especially the thin ideal in Western societies – could bias perception for these features. In this regard, further research might shed light on aspects of body dissatisfaction and the development of body image disturbances in terms of eating disorders.     

Laying eyes on headlights: eye movements suggest facial features in cars

Humans' proneness to see faces even in inanimate structures such as cars has long been noticed, yet empirical evidence is scarce. To examine this tendency of anthropomorphism, participants were asked to compare specific features (such as the eyes) of a face and a car front presented next to each other. Eye movement patterns indicated on which visual information participants relied to solve the task and clearly revealed the perception of facial features in cars, such as headlights as eyes or grille as nose. Most importantly, a predominance of headlights was found in attracting and guiding people's gaze irrespective of the feature they were asked to compare--equivalent to the role of the eyes during face perception. This response to abstract configurations is interpreted as an adaptive bias of the respective inherent mechanism for face perception and is evolutionarily reasonable with regard to a "better safe than sorry" strategy.     

Human face processing is tuned to sexual age preferences

Human faces can motivate nurturing behaviour or sexual behaviour when adults see a child or an adult face, respectively. This suggests that face processing is tuned to detecting age cues of sexual maturity to stimulate the appropriate reproductive behaviour: either caretaking or mating. In paedophilia, sexual attraction is directed to sexually immature children. Therefore, we hypothesized that brain networks that normally are tuned to mature faces of the preferred gender show an abnormal tuning to sexual immature faces in paedophilia. Here, we use functional magnetic resonance imaging (fMRI) to test directly for the existence of a network which is tuned to face cues of sexual maturity. During fMRI, participants sexually attracted to either adults or children were exposed to various face images. In individuals attracted to adults, adult faces activated several brain regions significantly more than child faces. These brain regions comprised areas known to be implicated in face processing, and sexual processing, including occipital areas, the ventrolateral prefrontal cortex and, subcortically, the putamen and nucleus caudatus. The same regions were activated in paedophiles, but with a reversed preferential response pattern.     

Auditory adaptation in voice perception

Perceptual aftereffects following adaptation to simple stimulus attributes (e.g., motion, color) have been studied for hundreds of years. A striking recent discovery was that adaptation also elicits contrastive aftereffects in visual perception of complex stimuli and faces [1-6]. Here, we show for the first time that adaptation to nonlinguistic information in voices elicits systematic auditory aftereffects. Prior adaptation to male voices causes a voice to be perceived as more female (and vice versa), and these auditory aftereffects were measurable even minutes after adaptation. By contrast, crossmodal adaptation effects were absent, both when male or female first names and when silently articulating male or female faces were used as adaptors. When sinusoidal tones (with frequencies matched to male and female voice fundamental frequencies) were used as adaptors, no aftereffects on voice perception were observed. This excludes explanations for the voice aftereffect in terms of both pitch adaptation and postperceptual adaptation to gender concepts and suggests that contrastive voice-coding mechanisms may routinely influence voice perception. The role of adaptation in calibrating properties of high-level voice representations indicates that adaptation is not confined to vision but is a ubiquitous mechanism in the perception of nonlinguistic social information from both faces and voices.     

Looking the part: social status cues shape race perception

It is commonly believed that race is perceived through another's facial features, such as skin color. In the present research, we demonstrate that cues to social status that often surround a face systematically change the perception of its race. Participants categorized the race of faces that varied along White-Black morph continua and that were presented with high-status or low-status attire. Low-status attire increased the likelihood of categorization as Black, whereas high-status attire increased the likelihood of categorization as White; and this influence grew stronger as race became more ambiguous (Experiment 1). When faces with high-status attire were categorized as Black or faces with low-status attire were categorized as White, participants' hand movements nevertheless revealed a simultaneous attraction to select the other race-category response (stereotypically tied to the status cue) before arriving at a final categorization. Further, this attraction effect grew as race became more ambiguous (Experiment 2). Computational simulations then demonstrated that these effects may be accounted for by a neurally plausible person categorization system, in which contextual cues come to trigger stereotypes that in turn influence race perception. Together, the findings show how stereotypes interact with physical cues to shape person categorization, and suggest that social and contextual factors guide the perception of race.     

Imagine Jane and identify John: face identity aftereffects induced by imagined faces

It is not known whether prolonged exposure to perceived and imagined complex visual images produces similar shifts in subsequent perception through selective adaptation. This question is important because a positive finding would suggest that perception and imagery of visual stimuli are mediated by shared neural networks. In this study, we used a selective adaptation procedure designed to induce high-level face-identity aftereffects--a phenomenon in which extended exposure to a particular face facilitates recognition of subsequent faces with opposite features while impairing recognition of all other faces. We report here that adaptation to either real or imagined faces produces a similar shift in perception and that identity boundaries represented in real and imagined faces are equivalent. Together, our results show that identity information contained in imagined and real faces produce similar behavioral outcomes. Our findings of high-level visual aftereffects induced by imagined stimuli can be taken as evidence for the involvement of shared neural networks that mediate perception and imagery of complex visual stimuli.     

Non-invasive evaluation of face and motion perception in humans

The neural mechanisms for the perception of face and motion were studied using psychophysical threshold measurements, event-related potentials (ERPs), and functional magnetic resonance imaging (fMRI). A face-specific ERP component, N170, was recorded over the posterior temporal cortex. Removal of the high-spatial-frequency components of the face altered the perception of familiar faces significantly, and familiarity can facilitate the cortico-cortical processing of facial perceptions. Similarly, the high-spatial-frequency components of the face seemed to be crucial for the recognition of facial expressions. Aging and visuospatial impairments affected motion perception significantly. Two distinct components of motion ERPs, N170 and P200, were recorded over the parietal region. The former was related to horizontal motion perception while the latter reflected the perception of radial optic flow motion. The results of fMRI showed that horizontal movements of objects and radial optic flow motion were perceived differently in the V5/MT and superior parietal lobe. We conclude that an integrated approach can provide useful information on spatial and temporal processing of face and motion non-invasively.     

The role of the 'face-cell' area in the discrimination and recognition of faces by monkeys.

Cortical neurons that are selectively sensitive to faces, parts of faces and particular facial expressions are concentrated in the banks and floor of the superior temporal sulcus in macaque monkeys. Their existence has prompted suggestions that it is damage to such a region in the human brain that leads to prosopagnosia: the inability to recognize faces or to discriminate between faces. This was tested by removing the face-cell area in a group of monkeys. The animals learned to discriminate between pictures of faces or inanimate objects, to select the odd face from a group, to inspect a face then select the matching face from a pair of faces after a variable delay, to discriminate between novel and familiar faces, and to identify specific faces. Removing the face-cell area produced no or little impairment which in the latter case was not specific for faces. In contrast, several prosopagnosic patients were impaired at several of these tasks. The animals were less able than before to discern the angle of regard in pictures of faces, suggesting that this area of the brain may be concerned with the perception of facial expression and bearing, which are important social signals in primates.     

A Cross‐Cultural Comparison of Population‐Specific Face Shape Preferences (Homo sapiens)

A substantial number of studies indicate that human facial features such as symmetry and averageness are preferred in human mate choice. Little is known, however, about face preferences in intrapopulation or between‐cultural settings as only a few studies have addressed this topic. One hypothesis is that people prefer faces that resemble faces from their own population as a result of visual adaptation. Here, we present results of an internet‐based experiment in which male and female participants from two European nations, Poland and Scotland, were asked to choose between ‘Polish‐modified’ and ‘Scottish‐modified’ faces (i.e. face portraits modified along one of ten different dimensions to resemble faces of the respective populations) of both sexes while being unaware of the origins of the portraits. The results show that the preferences of Polish judges for Polish‐shaped faces were significantly stronger than the preferences of Scottish judges for the Scottish‐shaped faces. Moreover, both Polish and Scottish judges preferred Polish‐shaped faces to Scottish‐shaped faces. These results suggest that natural visual experience acquired in everyday interpersonal interactions may have limited influence on aesthetic preferences in these particular intrapopulation settings.     

Spontaneous gender categorization in masking and priming studies: key for distinguishing Jane from John Doe but not Madonna from Sinatra

Facial recognition is key to social interaction, however with unfamiliar faces only generic information, in the form of facial stereotypes such as gender and age is available. Therefore is generic information more prominent in unfamiliar versus familiar face processing? In order to address the question we tapped into two relatively disparate stages of face processing. At the early stages of encoding, we employed perceptual masking to reveal that only perception of unfamiliar face targets is affected by the gender of the facial masks. At the semantic end; using a priming paradigm, we found that while to-be-ignored unfamiliar faces prime lexical decisions to gender congruent stereotypic words, familiar faces do not. Our findings indicate that gender is a more salient dimension in unfamiliar relative to familiar face processing, both in early perceptual stages as well as later semantic stages of person construal.