The origins of gestures and language: history,current advances and proposed theories

Investigating in depth the mechanisms underlying human and non‐human primate intentional communication systems (involving gestures, vocalisations, facial expressions and eye behaviours) can shed light on the evolutionary roots of language. Reports on non‐human primates, particularly great apes, suggest that gestural communication would have been a crucial prerequisite for the emergence of language, mainly based on the evidence of large communication repertoires and their associated multifaceted nature of intentionality that are key properties of language. Such research fuels important debates on the origins of gestures and language. We review here three non‐mutually exclusive processes that can explain mainly great apes' gestural acquisition and development: phylogenetic ritualisation, ontogenetic ritualisation, and learning via social negotiation. We hypothesise the following scenario for the evolutionary origins of gestures: gestures would have appeared gradually through evolution via signal ritualisation following the principle of derived activities, with the key involvement of emotional expression and processing. The increasing level of complexity of socioecological lifestyles and associated daily manipulative activities might then have enabled the acquisition and development of different interactional strategies throughout the life cycle. Many studies support a multimodal origin of language. However, we stress that the origins of language are not only multimodal, but more broadly multicausal. We propose a multicausal theory of language origins which better explains current findings. It postulates that primates' communicative signalling is a complex trait continually shaped by a cost–benefit trade‐off of signal production and processing of interactants in relation to four closely interlinked categories of evolutionary and life cycle factors: species, individual and context‐related characteristics as well as behaviour and its characteristics. We conclude by suggesting directions for future research to improve our understanding of the evolutionary roots of gestures and language.     

When Age Matters: Differences in Facial Mimicry and Autonomic Responses to Peers' Emotions in Teenagers and Adults

Age-group membership effects on explicit emotional facial expressions recognition have been widely demonstrated. In this study we investigated whether Age-group membership could also affect implicit physiological responses, as facial mimicry and autonomic regulation, to observation of emotional facial expressions. To this aim, facial Electromyography (EMG) and Respiratory Sinus Arrhythmia (RSA) were recorded from teenager and adult participants during the observation of facial expressions performed by teenager and adult models. Results highlighted that teenagers exhibited greater facial EMG responses to peers'' facial expressions, whereas adults showed higher RSA-responses to adult facial expressions. The different physiological modalities through which young and adults respond to peers'' emotional expressions are likely to reflect two different ways to engage in social interactions with coetaneous. Findings confirmed that age is an important and powerful social feature that modulates interpersonal interactions by influencing low-level physiological responses.     

A computational model of the development of separate representations of facial identity and expression in the primate visual system

Experimental studies have provided evidence that the visual processing areas of the primate brain represent facial identity and facial expression within different subpopulations of neurons. For example, in non-human primates there is evidence that cells within the inferior temporal gyrus (TE) respond primarily to facial identity, while cells within the superior temporal sulcus (STS) respond to facial expression. More recently, it has been found that the orbitofrontal cortex (OFC) of non-human primates contains some cells that respond exclusively to changes in facial identity, while other cells respond exclusively to facial expression. How might the primate visual system develop physically separate representations of facial identity and expression given that the visual system is always exposed to simultaneous combinations of facial identity and expression during learning? In this paper, a biologically plausible neural network model, VisNet, of the ventral visual pathway is trained on a set of carefully-designed cartoon faces with different identities and expressions. The VisNet model architecture is composed of a hierarchical series of four Self-Organising Maps (SOMs), with associative learning in the feedforward synaptic connections between successive layers. During learning, the network develops separate clusters of cells that respond exclusively to either facial identity or facial expression. We interpret the performance of the network in terms of the learning properties of SOMs, which are able to exploit the statistical indendependence between facial identity and expression.     

Holistic gaze strategy to categorize facial expression of varying intensities

Using faces representing exaggerated emotional expressions, recent behaviour and eye-tracking studies have suggested a dominant role of individual facial features in transmitting diagnostic cues for decoding facial expressions. Considering that in everyday life we frequently view low-intensity expressive faces in which local facial cues are more ambiguous, we probably need to combine expressive cues from more than one facial feature to reliably decode naturalistic facial affects. In this study we applied a morphing technique to systematically vary intensities of six basic facial expressions of emotion, and employed a self-paced expression categorization task to measure participants' categorization performance and associated gaze patterns. The analysis of pooled data from all expressions showed that increasing expression intensity would improve categorization accuracy, shorten reaction time and reduce number of fixations directed at faces. The proportion of fixations and viewing time directed at internal facial features (eyes, nose and mouth region), however, was not affected by varying levels of intensity. Further comparison between individual facial expressions revealed that although proportional gaze allocation at individual facial features was quantitatively modulated by the viewed expressions, the overall gaze distribution in face viewing was qualitatively similar across different facial expressions and different intensities. It seems that we adopt a holistic viewing strategy to extract expressive cues from all internal facial features in processing of naturalistic facial expressions.     

The MPI facial expression database--a validated database of emotional and conversational facial expressions

The ability to communicate is one of the core aspects of human life. For this, we use not only verbal but also nonverbal signals of remarkable complexity. Among the latter, facial expressions belong to the most important information channels. Despite the large variety of facial expressions we use in daily life, research on facial expressions has so far mostly focused on the emotional aspect. Consequently, most databases of facial expressions available to the research community also include only emotional expressions, neglecting the largely unexplored aspect of conversational expressions. To fill this gap, we present the MPI facial expression database, which contains a large variety of natural emotional and conversational expressions. The database contains 55 different facial expressions performed by 19 German participants. Expressions were elicited with the help of a method-acting protocol, which guarantees both well-defined and natural facial expressions. The method-acting protocol was based on every-day scenarios, which are used to define the necessary context information for each expression. All facial expressions are available in three repetitions, in two intensities, as well as from three different camera angles. A detailed frame annotation is provided, from which a dynamic and a static version of the database have been created. In addition to describing the database in detail, we also present the results of an experiment with two conditions that serve to validate the context scenarios as well as the naturalness and recognizability of the video sequences. Our results provide clear evidence that conversational expressions can be recognized surprisingly well from visual information alone. The MPI facial expression database will enable researchers from different research fields (including the perceptual and cognitive sciences, but also affective computing, as well as computer vision) to investigate the processing of a wider range of natural facial expressions.     

Chimpanzees (Pan troglodytes) Produce the Same Types of ‘Laugh Faces’ when They Emit Laughter and when They Are Silent

The ability to flexibly produce facial expressions and vocalizations has a strong impact on the way humans communicate, as it promotes more explicit and versatile forms of communication. Whereas facial expressions and vocalizations are unarguably closely linked in primates, the extent to which these expressions can be produced independently in nonhuman primates is unknown. The present work, thus, examined if chimpanzees produce the same types of facial expressions with and without accompanying vocalizations, as do humans. Forty-six chimpanzees (Pan troglodytes) were video-recorded during spontaneous play with conspecifics at the Chimfunshi Wildlife Orphanage. ChimpFACS was applied, a standardized coding system to measure chimpanzee facial movements, based on FACS developed for humans. Data showed that the chimpanzees produced the same 14 configurations of open-mouth faces when laugh sounds were present and when they were absent. Chimpanzees, thus, produce these facial expressions flexibly without being morphologically constrained by the accompanying vocalizations. Furthermore, the data indicated that the facial expression plus vocalization and the facial expression alone were used differently in social play, i.e., when in physical contact with the playmates and when matching the playmates’ open-mouth faces. These findings provide empirical evidence that chimpanzees produce distinctive facial expressions independently from a vocalization, and that their multimodal use affects communicative meaning, important traits for a more explicit and versatile way of communication. As it is still uncertain how human laugh faces evolved, the ChimpFACS data were also used to empirically examine the evolutionary relation between open-mouth faces with laugh sounds of chimpanzees and laugh faces of humans. The ChimpFACS results revealed that laugh faces of humans must have gradually emerged from laughing open-mouth faces of ancestral apes. This work examines the main evolutionary changes of laugh faces since the last common ancestor of chimpanzees and humans.     

Time Perception and Dynamics of Facial Expressions of Emotions

Two experiments were run to examine the effects of dynamic displays of facial expressions of emotions on time judgments. The participants were given a temporal bisection task with emotional facial expressions presented in a dynamic or a static display. Two emotional facial expressions and a neutral expression were tested and compared. Each of the emotional expressions had the same affective valence (unpleasant), but one was high-arousing (expressing anger) and the other low-arousing (expressing sadness). Our results showed that time judgments are highly sensitive to movements in facial expressions and the emotions expressed. Indeed, longer perceived durations were found in response to the dynamic faces and the high-arousing emotional expressions compared to the static faces and low-arousing expressions. In addition, the facial movements amplified the effect of emotions on time perception. Dynamic facial expressions are thus interesting tools for examining variations in temporal judgments in different social contexts.     

Sex differences in neural activation to facial expressions denoting contempt and disgust

The facial expression of contempt has been regarded to communicate feelings of moral superiority. Contempt is an emotion that is closely related to disgust, but in contrast to disgust, contempt is inherently interpersonal and hierarchical. The aim of this study was twofold. First, to investigate the hypothesis of preferential amygdala responses to contempt expressions versus disgust. Second, to investigate whether, at a neural level, men would respond stronger to biological signals of interpersonal superiority (e.g., contempt) than women. We performed an experiment using functional magnetic resonance imaging (fMRI), in which participants watched facial expressions of contempt and disgust in addition to neutral expressions. The faces were presented as distractors in an oddball task in which participants had to react to one target face. Facial expressions of contempt and disgust activated a network of brain regions, including prefrontal areas (superior, middle and medial prefrontal gyrus), anterior cingulate, insula, amygdala, parietal cortex, fusiform gyrus, occipital cortex, putamen and thalamus. Contemptuous faces did not elicit stronger amygdala activation than did disgusted expressions. To limit the number of statistical comparisons, we confined our analyses of sex differences to the frontal and temporal lobes. Men displayed stronger brain activation than women to facial expressions of contempt in the medial frontal gyrus, inferior frontal gyrus, and superior temporal gyrus. Conversely, women showed stronger neural responses than men to facial expressions of disgust. In addition, the effect of stimulus sex differed for men versus women. Specifically, women showed stronger responses to male contemptuous faces (as compared to female expressions), in the insula and middle frontal gyrus. Contempt has been conceptualized as signaling perceived moral violations of social hierarchy, whereas disgust would signal violations of physical purity. Thus, our results suggest a neural basis for sex differences in moral sensitivity regarding hierarchy on the one hand and physical purity on the other.     

Seeing emotion with your ears: emotional prosody implicitly guides visual attention to faces

Interpersonal communication involves the processing of multimodal emotional cues, particularly facial expressions (visual modality) and emotional speech prosody (auditory modality) which can interact during information processing. Here, we investigated whether the implicit processing of emotional prosody systematically influences gaze behavior to facial expressions of emotion. We analyzed the eye movements of 31 participants as they scanned a visual array of four emotional faces portraying fear, anger, happiness, and neutrality, while listening to an emotionally-inflected pseudo-utterance (Someone migged the pazing) uttered in a congruent or incongruent tone. Participants heard the emotional utterance during the first 1250 milliseconds of a five-second visual array and then performed an immediate recall decision about the face they had just seen. The frequency and duration of first saccades and of total looks in three temporal windows ([0-1250 ms], [1250-2500 ms], [2500-5000 ms]) were analyzed according to the emotional content of faces and voices. Results showed that participants looked longer and more frequently at faces that matched the prosody in all three time windows (emotion congruency effect), although this effect was often emotion-specific (with greatest effects for fear). Effects of prosody on visual attention to faces persisted over time and could be detected long after the auditory information was no longer present. These data imply that emotional prosody is processed automatically during communication and that these cues play a critical role in how humans respond to related visual cues in the environment, such as facial expressions.     

Turning the other cheek: the viewpoint dependence of facial expression after-effects

How do we visually encode facial expressions? Is this done by viewpoint-dependent mechanisms representing facial expressions as two-dimensional templates or do we build more complex viewpoint independent three-dimensional representations? Recent facial adaptation techniques offer a powerful way to address these questions. Prolonged viewing of a stimulus (adaptation) changes the perception of subsequently viewed stimuli (an after-effect). Adaptation to a particular attribute is believed to target those neural mechanisms encoding that attribute. We gathered images of facial expressions taken simultaneously from five different viewpoints evenly spread from the three-quarter leftward to the three-quarter rightward facing view. We measured the strength of expression after-effects as a function of the difference between adaptation and test viewpoints. Our data show that, although there is a decrease in after-effect over test viewpoint, there remains a substantial after-effect when adapt and test are at differing three-quarter views. We take these results to indicate that neural systems encoding facial expressions contain a mixture of viewpoint-dependent and viewpoint-independent elements. This accords with evidence from single cell recording studies in macaque and is consonant with a view in which viewpoint-independent expression encoding arises from a combination of view-dependent expression-sensitive responses.     

Can Healthy Fetuses Show Facial Expressions of “Pain” or “Distress”?

Background

With advances of research on fetal behavioural development, the question of whether we can identify fetal facial expressions and determine their developmental progression, takes on greater importance. In this study we investigate longitudinally the increasing complexity of combinations of facial movements from 24 to 36 weeks gestation in a sample of healthy fetuses using frame-by-frame coding of 4-D ultrasound scans. The primary aim was to examine whether these complex facial movements coalesce into a recognisable facial expression of pain/distress.

Methodology/Findings

Fifteen fetuses (8 girls, 7 boys) were observed four times in the second and third trimester of pregnancy. Fetuses showed significant progress towards more complex facial expressions as gestational age increased. Statistical analysis of the facial movements making up a specific facial configuration namely “pain/distress” also demonstrates that this facial expression becomes significantly more complete as the fetus matures.

Conclusions/Significance

The study shows that one can determine the normal progression of fetal facial movements. Furthermore, our results suggest that healthy fetuses progress towards an increasingly complete pain/distress expression as they mature. We argue that this is an adaptive process which is beneficial to the fetus postnatally and has the potential to identify normal versus abnormal developmental pathways.     

Emotional Modulation of the Attentional Blink Is Awareness-Dependent

It is well known that emotion can modulate attentional processes. Previous studies have shown that even under restricted awareness, emotional facial expressions (especially threat-related) can guide the direction of spatial attention. However, it remains unclear whether emotional facial expressions under restricted awareness can affect temporal attention. To address this issue, we used a modified attentional blink (AB) paradigm in which masked (Experiment 1) or unmasked (Experiment 2) emotional faces (fearful or neutral) were presented before the AB sequence. We found that, in comparison with neutral faces, masked fearful faces significantly decreased the AB magnitude (Experiment 1), whereas unmasked fearful faces significantly increased the AB magnitude (Experiment 2). These results indicate that effects of emotional expression on the AB are modulated by the level of awareness.     

First Report of Generalized Face Processing Difficulties in M?bius Sequence

Reverse simulation models of facial expression recognition suggest that we recognize the emotions of others by running implicit motor programmes responsible for the production of that expression. Previous work has tested this theory by examining facial expression recognition in participants with Möbius sequence, a condition characterized by congenital bilateral facial paralysis. However, a mixed pattern of findings has emerged, and it has not yet been tested whether these individuals can imagine facial expressions, a process also hypothesized to be underpinned by proprioceptive feedback from the face. We investigated this issue by examining expression recognition and imagery in six participants with Möbius sequence, and also carried out tests assessing facial identity and object recognition, as well as basic visual processing. While five of the six participants presented with expression recognition impairments, only one was impaired at the imagery of facial expressions. Further, five participants presented with other difficulties in the recognition of facial identity or objects, or in lower-level visual processing. We discuss the implications of our findings for the reverse simulation model, and suggest that facial identity recognition impairments may be more severe in the condition than has previously been noted.     

Dogs Evaluate Threatening Facial Expressions by Their Biological Validity – Evidence from Gazing Patterns

Appropriate response to companions’ emotional signals is important for all social creatures. The emotional expressions of humans and non-human animals have analogies in their form and function, suggesting shared evolutionary roots, but very little is known about how animals other than primates view and process facial expressions. In primates, threat-related facial expressions evoke exceptional viewing patterns compared with neutral or positive stimuli. Here, we explore if domestic dogs (Canis familiaris) have such an attentional bias toward threatening social stimuli and whether observed emotional expressions affect dogs’ gaze fixation distribution among the facial features (eyes, midface and mouth). We recorded the voluntary eye gaze of 31 domestic dogs during viewing of facial photographs of humans and dogs with three emotional expressions (threatening, pleasant and neutral). We found that dogs’ gaze fixations spread systematically among facial features. The distribution of fixations was altered by the seen expression, but eyes were the most probable targets of the first fixations and gathered longer looking durations than mouth regardless of the viewed expression. The examination of the inner facial features as a whole revealed more pronounced scanning differences among expressions. This suggests that dogs do not base their perception of facial expressions on the viewing of single structures, but the interpretation of the composition formed by eyes, midface and mouth. Dogs evaluated social threat rapidly and this evaluation led to attentional bias, which was dependent on the depicted species: threatening conspecifics’ faces evoked heightened attention but threatening human faces instead an avoidance response. We propose that threatening signals carrying differential biological validity are processed via distinctive neurocognitive pathways. Both of these mechanisms may have an adaptive significance for domestic dogs. The findings provide a novel perspective on understanding the processing of emotional expressions and sensitivity to social threat in non-primates.     

Adaptive evolution of facial colour patterns in Neotropical primates

The rich diversity of primate faces has interested naturalists for over a century. Researchers have long proposed that social behaviours have shaped the evolution of primate facial diversity. However, the primate face constitutes a unique structure where the diverse and potentially competing functions of communication, ecology and physiology intersect, and the major determinants of facial diversity remain poorly understood. Here, we provide the first evidence for an adaptive role of facial colour patterns and pigmentation within Neotropical primates. Consistent with the hypothesis that facial patterns function in communication and species recognition, we find that species living in smaller groups and in sympatry with a higher number of congener species have evolved more complex patterns of facial colour. The evolution of facial pigmentation and hair length is linked to ecological factors, and ecogeographical rules related to UV radiation and thermoregulation are met by some facial regions. Our results demonstrate the interaction of behavioural and ecological factors in shaping one of the most outstanding facial diversities of any mammalian lineage.     

Brain systems for assessing the affective value of faces

Cognitive neuroscience research on facial expression recognition and face evaluation has proliferated over the past 15 years. Nevertheless, large questions remain unanswered. In this overview, we discuss the current understanding in the field, and describe what is known and what remains unknown. In §2, we describe three types of behavioural evidence that the perception of traits in neutral faces is related to the perception of facial expressions, and may rely on the same mechanisms. In §3, we discuss cortical systems for the perception of facial expressions, and argue for a partial segregation of function in the superior temporal sulcus and the fusiform gyrus. In §4, we describe the current understanding of how the brain responds to emotionally neutral faces. To resolve some of the inconsistencies in the literature, we perform a large group analysis across three different studies, and argue that one parsimonious explanation of prior findings is that faces are coded in terms of their typicality. In §5, we discuss how these two lines of research--perception of emotional expressions and face evaluation--could be integrated into a common, cognitive neuroscience framework.     

Geometrical features of lips using the properties of parabola for recognizing facial expression

Various real-time applications such as Human–Computer Interactions, Psychometric analysis, etc. use facial expressions as one of the important parameters. The researchers have used Action Units (AU) of the face as feature points and its deformation is compared with the reference points on the face to estimate the facial expressions. Among many parts of the face, features from the mouth contribute largely to all the well-known emotions. In this paper, the parabola theory is used to identify and mark various points on the lips. These points are considered as feature points to construct feature vectors. The Latus Rectum, Focal Point, Directrix, Vertex, etc. are also considered to identify the feature points of the lower lips and upper lips. The proposed approach is evaluated on benchmark datasets such as JAFFEE and Cohn–Kanade dataset and it is found that the performance is encouraging in understanding the facial expressions. The results are compared with contemporary methods and found that the proposed approach has given good classification accuracy in recognizing facial expressions.     

Revealing Variations in Perception of Mental States from Dynamic Facial Expressions: A Cautionary Note

Although a great deal of research has been conducted on the recognition of basic facial emotions (e.g., anger, happiness, sadness), much less research has been carried out on the more subtle facial expressions of an individual''s mental state (e.g., anxiety, disinterest, relief). Of particular concern is that these mental state expressions provide a crucial source of communication in everyday life but little is known about the accuracy with which natural dynamic facial expressions of mental states are identified and, in particular, the variability in mental state perception that is produced. Here we report the findings of two studies that investigated the accuracy and variability with which dynamic facial expressions of mental states were identified by participants. Both studies used stimuli carefully constructed using procedures adopted in previous research, and free-report (Study 1) and forced-choice (Study 2) measures of response accuracy and variability. The findings of both studies showed levels of response accuracy that were accompanied by substantial variation in the labels assigned by observers to each mental state. Thus, when mental states are identified from facial expressions in experiments, the identities attached to these expressions appear to vary considerably across individuals. This variability raises important issues for understanding the identification of mental states in everyday situations and for the use of responses in facial expression research.     

Optimal Geometrical Set for Automated Marker Placement to Virtualized Real-Time Facial Emotions

In recent years, real-time face recognition has been a major topic of interest in developing intelligent human-machine interaction systems. Over the past several decades, researchers have proposed different algorithms for facial expression recognition, but there has been little focus on detection in real-time scenarios. The present work proposes a new algorithmic method of automated marker placement used to classify six facial expressions: happiness, sadness, anger, fear, disgust, and surprise. Emotional facial expressions were captured using a webcam, while the proposed algorithm placed a set of eight virtual markers on each subject’s face. Facial feature extraction methods, including marker distance (distance between each marker to the center of the face) and change in marker distance (change in distance between the original and new marker positions), were used to extract three statistical features (mean, variance, and root mean square) from the real-time video sequence. The initial position of each marker was subjected to the optical flow algorithm for marker tracking with each emotional facial expression. Finally, the extracted statistical features were mapped into corresponding emotional facial expressions using two simple non-linear classifiers, K-nearest neighbor and probabilistic neural network. The results indicate that the proposed automated marker placement algorithm effectively placed eight virtual markers on each subject’s face and gave a maximum mean emotion classification rate of 96.94% using the probabilistic neural network.     

A Novel Method Testing the Ability to Imitate Composite Emotional Expressions Reveals an Association with Empathy

Social communication relies on intentional control of emotional expression. Its variability across cultures suggests important roles for imitation in developing control over enactment of subtly different facial expressions and therefore skills in emotional communication. Both empathy and the imitation of an emotionally communicative expression may rely on a capacity to share both the experience of an emotion and the intention or motor plan associated with its expression. Therefore, we predicted that facial imitation ability would correlate with empathic traits. We built arrays of visual stimuli by systematically blending three basic emotional expressions in controlled proportions. Raters then assessed accuracy of imitation by reconstructing the same arrays using photographs of participants’ attempts at imitations of the stimuli. Accuracy was measured as the mean proximity of the participant photographs to the target stimuli in the array. Levels of performance were high, and rating was highly reliable. More empathic participants, as measured by the empathy quotient (EQ), were better facial imitators and, in particular, performed better on the more complex, blended stimuli. This preliminary study offers a simple method for the measurement of facial imitation accuracy and supports the hypothesis that empathic functioning may utilise motor control mechanisms which are also used for emotional expression.