Multimodal communication and language origins: integrating gestures and vocalizations

The presence of divergent and independent research traditions in the gestural and vocal domains of primate communication has resulted in major discrepancies in the definition and operationalization of cognitive concepts. However, in recent years, accumulating evidence from behavioural and neurobiological research has shown that both human and non‐human primate communication is inherently multimodal. It is therefore timely to integrate the study of gestural and vocal communication. Herein, we review evidence demonstrating that there is no clear difference between primate gestures and vocalizations in the extent to which they show evidence for the presence of key language properties: intentionality, reference, iconicity and turn‐taking. We also find high overlap in the neurobiological mechanisms producing primate gestures and vocalizations, as well as in ontogenetic flexibility. These findings confirm that human language had multimodal origins. Nonetheless, we note that in great apes, gestures seem to fulfil a carrying (i.e. predominantly informative) role in close‐range communication, whereas the opposite holds for face‐to‐face interactions of humans. This suggests an evolutionary shift in the carrying role from the gestural to the vocal stream, and we explore this transition in the carrying modality. Finally, we suggest that future studies should focus on the links between complex communication, sociality and cooperative tendency to strengthen the study of language origins.     

Front Cover

This review highlights the scientific advances concerning the origins of human right‐handedness and language (speech and gestures). The comparative approach we adopted provides evidence that research on human and non‐human animals’ behavioural asymmetries helps understand the processes that lead to the strong human left‐hemisphere specialisation. We review four major non‐mutually exclusive environmental factors that are likely to have shaped the evolution of human and non‐human primates’ manual asymmetry: socioecological lifestyle, postural characteristics, task‐level complexity and tool use. We hypothesise the following scenario for the evolutionary origins of human right‐handedness: the right‐direction of modern humans’ manual laterality would have emerged from our ecological (terrestrial) and social (multilevel system) lifestyle; then, it would have been strengthened by the gradual adoption of the bipedal stance associated with bipedal locomotion, and the increasing level of complexity of our daily tasks including bimanual coordinated actions and tool use. Although hemispheric functional lateralisation has been shaped through evolution, reports indicate that many factors and their mutual intertwinement can modulate human and non‐human primates’ manual laterality throughout their life cycle: genetic and environmental factors, mainly individual sociodemographic characteristics (e.g., age, sex and rank), behavioural characteristics (e.g., gesture per se and gestural sensory modality) and context‐related characteristics (e.g., emotional context and position of target). These environmental (evolutionary and life cycle) factors could also have influenced primates’ manual asymmetry indirectly through epigenetic modifications. All these findings led us to propose the hypothesis of a multicausal origin of human right‐handedness.     

Chimpanzee vocal signaling points to a multimodal origin of human language

The evolutionary origin of human language and its neurobiological foundations has long been the object of intense scientific debate. Although a number of theories have been proposed, one particularly contentious model suggests that human language evolved from a manual gestural communication system in a common ape-human ancestor. Consistent with a gestural origins theory are data indicating that chimpanzees intentionally and referentially communicate via manual gestures, and the production of manual gestures, in conjunction with vocalizations, activates the chimpanzee Broca's area homologue--a region in the human brain that is critical for the planning and execution of language. However, it is not known if this activity observed in the chimpanzee Broca's area is the result of the chimpanzees producing manual communicative gestures, communicative sounds, or both. This information is critical for evaluating the theory that human language evolved from a strictly manual gestural system. To this end, we used positron emission tomography (PET) to examine the neural metabolic activity in the chimpanzee brain. We collected PET data in 4 subjects, all of whom produced manual communicative gestures. However, 2 of these subjects also produced so-called attention-getting vocalizations directed towards a human experimenter. Interestingly, only the two subjects that produced these attention-getting sounds showed greater mean metabolic activity in the Broca's area homologue as compared to a baseline scan. The two subjects that did not produce attention-getting sounds did not. These data contradict an exclusive "gestural origins" theory for they suggest that it is vocal signaling that selectively activates the Broca's area homologue in chimpanzees. In other words, the activity observed in the Broca's area homologue reflects the production of vocal signals by the chimpanzees, suggesting that this critical human language region was involved in vocal signaling in the common ancestor of both modern humans and chimpanzees.     

Chimpanzee Alarm Call Production Meets Key Criteria for Intentionality

Determining the intentionality of primate communication is critical to understanding the evolution of human language. Although intentional signalling has been claimed for some great ape gestural signals, comparable evidence is currently lacking for their vocal signals. We presented wild chimpanzees with a python model and found that two of three alarm call types exhibited characteristics previously used to argue for intentionality in gestural communication. These alarm calls were: (i) socially directed and given to the arrival of friends, (ii) associated with visual monitoring of the audience and gaze alternations, and (iii) goal directed, as calling only stopped when recipients were safe from the predator. Our results demonstrate that certain vocalisations of our closest living relatives qualify as intentional signals, in a directly comparable way to many great ape gestures. We conclude that our results undermine a central argument of gestural theories of language evolution and instead support a multimodal origin of human language.     

What Role Do Mothers Play in the Gestural Acquisition of Bonobos (Pan paniscus) and Chimpanzees (Pan troglodytes)?

Contemporary research hypothesizes that biological inheritance and ontogenetic factors shape the development of gestural communication in nonhuman great apes. However, little is known about the specific role that mothers play in the acquisition of their infants’ gestures. We observed 6 bonobo (Pan paniscus) and 4 chimpanzee (Pan troglodytes) mother–infant dyads and recorded their gesture types and frequency. We analyzed all behavioral contexts in which gestures occurred as well as the play context alone. Infants of both species were unlikely to share gestures with their mother or unrelated adult females. However, gestural sharing was prevalent within age groups. Within and across species, infant–infant and mother–mother groups were homogeneous regarding the types of gestures they shared, although there was individual variation in the frequency of gesture use. Our findings provide limited evidence that infants learned their gestures by imitating their mothers. Phylogenetic influences seem to be vital in gestural acquisition but, we suggest, repertoire development cannot be disentangled from individual social encounters during life.     

Human infants and baboons show the same pattern of handedness for a communicative gesture

To test the role of gestures in the origin of language, we studied hand preferences for grasping or pointing to objects at several spatial positions in human infants and adult baboons. If the roots of language are indeed in gestural communication, we expect that human infants and baboons will present a comparable difference in their pattern of laterality according to task: both should be more right-hand/left-hemisphere specialized when communicating by pointing than when simply grasping objects. Our study is the first to test both human infants and baboons on the same communicative task. Our results show remarkable convergence in the distribution of the two species' hand biases on the two kinds of tasks: In both human infants and baboons, right-hand preference was significantly stronger for the communicative task than for grasping objects. Our findings support the hypothesis that left-lateralized language may be derived from a gestural communication system that was present in the common ancestor of baboons and humans.     

Language, gesture, skill: the co-evolutionary foundations of language

This paper defends a gestural origins hypothesis about the evolution of enhanced communication and language in the hominin lineage. The paper shows that we can develop an incremental model of language evolution on that hypothesis, but not if we suppose that language originated in an expansion of great ape vocalization. On the basis of the gestural origins hypothesis, the paper then advances solutions to four classic problems about the evolution of language: (i) why did language evolve only in the hominin lineage? (ii) why is language use an evolutionarily stable form of informational cooperation, despite the fact that hominins have diverging evolutionary interests? (iii) how did stimulus independent symbols emerge? (iv) what were the origins of complex, syntactically organized symbols? The paper concludes by confronting two challenges: those of testability and of explaining the gesture-to-speech transition; crucial issues for any gestural origins hypothesis.     

A structure-based repertoire of manual gestures in wild chimpanzees: statistical analyses of a graded communication system

Great ape gestural communication is considered important in understanding the evolution of human language as these share important features, namely, flexible and intentional signal use. Although gestural repertoires have been compiled for captive and wild primates, reports are largely qualitative. We quantify the morphological structure and variation of gestural signals identified in the repertoire of a community of wild chimpanzees. Gestures were classified on the basis of 29 morphological features, such as trajectory and orientation during the preparatory and stroke phases of a gesture. Hierarchical cluster and discriminant function analyses identified 30 morphologically distinct manual gesture types; the majority was subsequently correctly classified using a cross-validation technique, with incorrect classifications for rare gesture types only. Comparisons of this statistically determined repertoire with previous repertoires did not identify systematic variation between captive and wild chimpanzees. Moreover, consensus was not greater within studies of the same populations, highlighting the importance of systematic and well-documented inventories. Our morphologically based analyses indicate that manual gestures are best considered as graded rather than discrete communication signals, similar to some vocalisation systems. We discuss these findings in light of current theories of human language evolution.     

Gestural communication in young gorillas (Gorilla gorilla): gestural repertoire,learning, and use

In the present study we investigated the gestural communication of gorillas (Gorilla gorilla). The subjects were 13 gorillas (1-6 years old) living in two different groups in captivity. Our goal was to compile the gestural repertoire of subadult gorillas, with a special focus on processes of social cognition, including attention to individual and developmental variability, group variability, and flexibility of use. Thirty-three different gestures (six auditory, 11 tactile, and 16 visual gestures) were recorded. We found idiosyncratic gestures, individual differences, and similar degrees of concordance between and within groups, as well as some group-specific gestures. These results provide evidence that ontogenetic ritualization is the main learning process involved, but some form of social learning may also be responsible for the acquisition of special gestures. The present study establishes that gorillas have a multifaceted gestural repertoire, characterized by a great deal of flexibility with accommodations to various communicative circumstances, including the attentional state of the recipient. The possibility of assigning Seyfarth and Cheney's [1997] model for nonhuman primate vocal development to the development of nonhuman primate gestural communication is discussed.     

A neuropsychological perspective on the link between language and praxis in modern humans

Hypotheses about the emergence of human cognitive abilities postulate strong evolutionary links between language and praxis, including the possibility that language was originally gestural. The present review considers functional and neuroanatomical links between language and praxis in brain-damaged patients with aphasia and/or apraxia. The neural systems supporting these functions are predominantly located in the left hemisphere. There are many parallels between action and language for recognition, imitation and gestural communication suggesting that they rely partially on large, common networks, differentially recruited depending on the nature of the task. However, this relationship is not unequivocal and the production and understanding of gestural communication are dependent on the context in apraxic patients and remains to be clarified in aphasic patients. The phonological, semantic and syntactic levels of language seem to share some common cognitive resources with the praxic system. In conclusion, neuropsychological observations do not allow support or rejection of the hypothesis that gestural communication may have constituted an evolutionary link between tool use and language. Rather they suggest that the complexity of human behaviour is based on large interconnected networks and on the evolution of specific properties within strategic areas of the left cerebral hemisphere.     

Playing Charades in the fMRI: Are Mirror and/or Mentalizing Areas Involved in Gestural Communication?

Communication is an important aspect of human life, allowing us to powerfully coordinate our behaviour with that of others. Boiled down to its mere essentials, communication entails transferring a mental content from one brain to another. Spoken language obviously plays an important role in communication between human individuals. Manual gestures however often aid the semantic interpretation of the spoken message, and gestures may have played a central role in the earlier evolution of communication. Here we used the social game of charades to investigate the neural basis of gestural communication by having participants produce and interpret meaningful gestures while their brain activity was measured using functional magnetic resonance imaging. While participants decoded observed gestures, the putative mirror neuron system (pMNS: premotor, parietal and posterior mid-temporal cortex), associated with motor simulation, and the temporo-parietal junction (TPJ), associated with mentalizing and agency attribution, were significantly recruited. Of these areas only the pMNS was recruited during the production of gestures. This suggests that gestural communication relies on a combination of simulation and, during decoding, mentalizing/agency attribution brain areas. Comparing the decoding of gestures with a condition in which participants viewed the same gestures with an instruction not to interpret the gestures showed that although parts of the pMNS responded more strongly during active decoding, most of the pMNS and the TPJ did not show such significant task effects. This suggests that the mere observation of gestures recruits most of the system involved in voluntary interpretation.     

The origins of non-human primates' manual gestures

The increasing body of research into human and non-human primates' gestural communication reflects the interest in a comparative approach to human communication, particularly possible scenarios of language evolution. One of the central challenges of this field of research is to identify appropriate criteria to differentiate a gesture from other non-communicative actions. After an introduction to the criteria currently used to define non-human primates' gestures and an overview of ongoing research, we discuss different pathways of how manual actions are transformed into manual gestures in both phylogeny and ontogeny. Currently, the relationship between actions and gestures is not only investigated on a behavioural, but also on a neural level. Here, we focus on recent evidence concerning the differential laterality of manual actions and gestures in apes in the framework of a functional asymmetry of the brain for both hand use and language.     

Metasignaling and Language Origins

Over the past two decades, comparative linguistic anthropological research has disclosed the significance of metadiscourse and metasignaling forthe establishment, maintenance, and transformation of social relations. One type of metasignal-signal relationship has, because of its role in producing sociability, come to seem especially interesting with regard to the evolutionary origins of human language—the type wherein a strategically (and presumably neocortically) induced metasignal is modeled after, but distinct from, an instinctively triggered signal, as in ritualized lamentation. Strategic vocal deceptions in nonhuman primates are possible precursors of true socially constructed, socially shared metasignals, which in turn may be ancestors of modern human language. This article charts an evolutionary path from strategic vocal manipulation, to cultural metasignals, to language, using data from primate studies, discourse-centered research, and early child language acquisition. [Key words: language origins, semiotics, metacommunication, chimpanzees, early language acquisition]     

Gestural forms of communication in primates. II. The relation of gestures to acoustic signals and tool activities in primate phylogeny

The connection between the frequency of acoustic signals application, contact and gestural communication and the manipulation activity of Cercopithecidea and Anthropoidea have been examined. A scheme reflecting the changes of correlational behavioural systems at different periods of primate phylogenesis is proposed. The behavioural changes in the connections of gestures, vocalisation and manipulation activity during the process of their transition into the language and labour on the way to hominization are shown. On the basis of the exact data a possible role of gestural an acoustic signals in the origin of language is given.     

Functionally referential communication in a chimpanzee

The evolutionary origins of the use of speech signals to refer to events or objects in the world have remained obscure. Although functionally referential calls have been described in some monkey species, studies with our closest living relatives, the great apes, have not generated comparable findings. These negative results have been taken to suggest that ape vocalizations are not the product of their otherwise sophisticated mentality and that ape gestural communication is more informative for theories of language evolution. We tested whether chimpanzee rough grunts, which are produced during feeding contexts, functioned as referential signals. Individuals produced acoustically distinct types of "rough grunts" when encountering different foods. In a naturalistic playback experiment, a focal subject was able to use the information conveyed by these calls produced by several group mates to guide his search for food, demonstrating that the different grunt types were meaningful to him. This study provides experimental evidence that our closest living relatives can produce and understand functionally referential calls as part of their natural communication. We suggest that these findings give support to the vocal rather than gestural theories of language evolution.     

Multimodal Signaling in Male and Female Foot-Flagging Frogs Staurois guttatus (Ranidae): An Alerting Function of Calling

In multimodal communication, individuals use several sensory modalities for information transfer. We report on novel observations of foot‐flagging in the Bornean ranid frog Staurois guttatus that is temporally linked to advertisement calling. In addition, we document the first case of foot‐flagging in a female anuran as well as additional visual displays in both males and females including arm‐waving, vocal‐sac pumping and open‐mouth display. In males, advertisement calls and foot‐flags were given throughout most of the day, suggesting that acoustic and visual signals form a multicomponent and multimodal display. We tested the efficacy‐based alerting signal hypothesis of multimodal communication using acoustic playback experiments with males. This hypothesis predicts that an initial signal draws the receiver's attention to the location of a subsequent more informative signal. Several lines of evidence supported the alerting hypothesis. First, the latency between foot‐flags and advertisement calls was significantly higher than that between advertisement calls and foot‐flags, suggesting a functional linkage with calls drawing attention to foot‐flags. Secondly, advertisement calling had a signaling function with males responding significantly more often with both calls and foot‐flags compared with pre‐ and post‐playback control periods. Finally, and most notably, all males tested turned towards the playback stimulus, suggesting that the advertisement call serves to focus their attention on subsequent signals. We discuss the potential of multimodal signaling for conspecific and heterospecific communication and the circumstances under which such a multimodal communication system could evolve.     

Language: Origins, definitions and chimpanzees

Language and its evolutionary origins are examined in the light of empirical data from studies on artificial language acquisition, human language acquisition in the chimpanzee and the definitional problems surrounding language. Rather than providing solutions, questions are raised in regard to the various approaches to the problems of language origin and definitions of language.     

Origins of cheating and loss of symbiosis in wild Bradyrhizobium

Rhizobial bacteria nodulate legume roots and fix nitrogen in exchange for photosynthates. These symbionts are infectiously acquired from the environment and in such cases selection models predict evolutionary spread of uncooperative mutants. Uncooperative rhizobia – including nonfixing and non‐nodulating strains – appear common in agriculture, yet their population biology and origins remain unknown in natural soils. Here, a phylogenetically broad sample of 62 wild‐collected rhizobial isolates was experimentally inoculated onto Lotus strigosus to assess their nodulation ability and effects on host growth. A cheater strain was discovered that proliferated in host tissue while offering no benefit; its fitness was superior to that of beneficial strains. Phylogenetic reconstruction of Bradyrhizobium rDNA and transmissible symbiosis‐island loci suggest that the cheater evolved via symbiotic gene transfer. Many strains were also identified that failed to nodulate L. strigosus, and it appears that nodulation ability on this host has been recurrently lost in the symbiont population. This is the first study to reveal the adaptive nature of rhizobial cheating and to trace the evolutionary origins of uncooperative rhizobial mutants.     

Functionally referential signals: A promising paradigm whose time has passed

Finding the evolutionary origins of human language in the communication systems of our closest living relatives has, for the last several decades, been a major goal of many in the field of animal communication generally and primate communication specifically. 1 - 4 The so‐called “functionally referential” signals have long been considered promising in this regard, with apparent parallels with the semantic communication that characterizes language. The once‐prominent idea that functionally referential signals are word‐like, in that they are arbitrary sounds that refer to phenomena external to the caller, has largely been abandoned. 5 However, the idea that these signals may offer the strongest link between primate communication and human language remains widespread, primarily due to the fact the behavior of receivers indicates that such signals enable them to make very specific inferences about their physical or social environment. Here we review the concept of functional reference and discuss modern perspectives that indicate that, although the sophistication of receivers provides some continuity between nonhuman primate and human cognition, this continuity is not unique to functionally referential signals. In fact, because functionally referential signals are, by definition, produced only in specific contexts, receivers are less dependent on the integration of contextual cues with signal features to determine an appropriate response. The processing of functionally referential signals is therefore likely to entail simpler cognitive operations than does that of less context‐specific signals. While studies of functional reference have been important in highlighting the relatively sophisticated processes that underlie receiver behavior, we believe that the continued focus on context‐specific calls detracts from the potentially more complex processes underlying responses to more unspecific calls. In this sense, we argue that the concept of functional reference, while historically important for the field, has outlived its usefulness and become a red herring in the pursuit of the links between primate communication and human language. © 2012 Wiley Periodicals, Inc.     

Language as a multimodal phenomenon: implications for language learning,processing and evolution

Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic gestures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages.