Widening the lens: what the manual modality reveals about language,learning and cognition

The goal of this paper is to widen the lens on language to include the manual modality. We look first at hearing children who are acquiring language from a spoken language model and find that even before they use speech to communicate, they use gesture. Moreover, those gestures precede, and predict, the acquisition of structures in speech. We look next at deaf children whose hearing losses prevent them from using the oral modality, and whose hearing parents have not presented them with a language model in the manual modality. These children fall back on the manual modality to communicate and use gestures, which take on many of the forms and functions of natural language. These homemade gesture systems constitute the first step in the emergence of manual sign systems that are shared within deaf communities and are full-fledged languages. We end by widening the lens on sign language to include gesture and find that signers not only gesture, but they also use gesture in learning contexts just as speakers do. These findings suggest that what is key in gesture''s ability to predict learning is its ability to add a second representational format to communication, rather than a second modality. Gesture can thus be language, assuming linguistic forms and functions, when other vehicles are not available; but when speech or sign is possible, gesture works along with language, providing an additional representational format that can promote learning.     

Chimpanzee vocal signaling points to a multimodal origin of human language

The evolutionary origin of human language and its neurobiological foundations has long been the object of intense scientific debate. Although a number of theories have been proposed, one particularly contentious model suggests that human language evolved from a manual gestural communication system in a common ape-human ancestor. Consistent with a gestural origins theory are data indicating that chimpanzees intentionally and referentially communicate via manual gestures, and the production of manual gestures, in conjunction with vocalizations, activates the chimpanzee Broca's area homologue--a region in the human brain that is critical for the planning and execution of language. However, it is not known if this activity observed in the chimpanzee Broca's area is the result of the chimpanzees producing manual communicative gestures, communicative sounds, or both. This information is critical for evaluating the theory that human language evolved from a strictly manual gestural system. To this end, we used positron emission tomography (PET) to examine the neural metabolic activity in the chimpanzee brain. We collected PET data in 4 subjects, all of whom produced manual communicative gestures. However, 2 of these subjects also produced so-called attention-getting vocalizations directed towards a human experimenter. Interestingly, only the two subjects that produced these attention-getting sounds showed greater mean metabolic activity in the Broca's area homologue as compared to a baseline scan. The two subjects that did not produce attention-getting sounds did not. These data contradict an exclusive "gestural origins" theory for they suggest that it is vocal signaling that selectively activates the Broca's area homologue in chimpanzees. In other words, the activity observed in the Broca's area homologue reflects the production of vocal signals by the chimpanzees, suggesting that this critical human language region was involved in vocal signaling in the common ancestor of both modern humans and chimpanzees.     

The origins of non-human primates' manual gestures

The increasing body of research into human and non-human primates' gestural communication reflects the interest in a comparative approach to human communication, particularly possible scenarios of language evolution. One of the central challenges of this field of research is to identify appropriate criteria to differentiate a gesture from other non-communicative actions. After an introduction to the criteria currently used to define non-human primates' gestures and an overview of ongoing research, we discuss different pathways of how manual actions are transformed into manual gestures in both phylogeny and ontogeny. Currently, the relationship between actions and gestures is not only investigated on a behavioural, but also on a neural level. Here, we focus on recent evidence concerning the differential laterality of manual actions and gestures in apes in the framework of a functional asymmetry of the brain for both hand use and language.     

Language as a multimodal phenomenon: implications for language learning,processing and evolution

Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic gestures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages.     

A word in the hand: action, gesture and mental representation in humans and non-human primates

The movements we make with our hands both reflect our mental processes and help to shape them. Our actions and gestures can affect our mental representations of actions and objects. In this paper, we explore the relationship between action, gesture and thought in both humans and non-human primates and discuss its role in the evolution of language. Human gesture (specifically representational gesture) may provide a unique link between action and mental representation. It is kinaesthetically close to action and is, at the same time, symbolic. Non-human primates use gesture frequently to communicate, and do so flexibly. However, their gestures mainly resemble incomplete actions and lack the representational elements that characterize much of human gesture. Differences in the mirror neuron system provide a potential explanation for non-human primates' lack of representational gestures; the monkey mirror system does not respond to representational gestures, while the human system does. In humans, gesture grounds mental representation in action, but there is no evidence for this link in other primates. We argue that gesture played an important role in the transition to symbolic thought and language in human evolution, following a cognitive leap that allowed gesture to incorporate representational elements.     

Imitation,objects, tools,and the rudiments of language in human ontogeny

Human beings are imitative generalists. We can immediately imitate a wide range of behaviors with great facility, whether they be vocal maneuvers, body postures, or actions on objects. The ontogeny of this skill has been an enduring question in developmental psychology. Classical theory holds that the ability to imitate facial gestures is a milestone that is passed at about one year. Before this time infants are thought to lack the perceptual-cognitive sophistication necessary to match a gesture they can see with one they cannot see themselves perform. A second developmental milestone is the capacity for deferred imitation, i.e. imitation of an absent model. This is said to emerge at about 18 months, in close synchrony with other higher-order activities such as object permanence and tool use, as part of a general cognitive shift from a purely sensory-motor level of functioning to one that allows language. Research suggests that the imitative capacity of young infants has been underestimated. Human infants are capable of imitating facial gestures at birth, with infants less than one day old manifesting this skill. Moreover recent experiments have established deferred imitation well before the predicted age of 18 months. Studies discussed here show that 9-month-olds can duplicate acts after a delay of 24 hours, and that 14-month-olds can retain and duplicate as many as five actions over a 1-week delay. These new findings re-raise questions about the relation between nonverbal cognitive development and language development: What aspects, if any, of these two domains are linked? A hypothesis is delineated that predicts certain very specific relations between particular cognitive and semantic achievements during the one-word stage, and data are reported supporting this hypothesis. Specifically, relations are reported between: (a) the development of object permanence and the use of words encoding disappearance, (b) means-ends understanding (as manifest in tool use) and words encoding success and failure, and (c) categorization behavior and the onset of the naming explosion. This research on human ontogeny suggests close and highly specific links between aspects of early language and thought.     

Multimodal communication and language origins: integrating gestures and vocalizations

The presence of divergent and independent research traditions in the gestural and vocal domains of primate communication has resulted in major discrepancies in the definition and operationalization of cognitive concepts. However, in recent years, accumulating evidence from behavioural and neurobiological research has shown that both human and non‐human primate communication is inherently multimodal. It is therefore timely to integrate the study of gestural and vocal communication. Herein, we review evidence demonstrating that there is no clear difference between primate gestures and vocalizations in the extent to which they show evidence for the presence of key language properties: intentionality, reference, iconicity and turn‐taking. We also find high overlap in the neurobiological mechanisms producing primate gestures and vocalizations, as well as in ontogenetic flexibility. These findings confirm that human language had multimodal origins. Nonetheless, we note that in great apes, gestures seem to fulfil a carrying (i.e. predominantly informative) role in close‐range communication, whereas the opposite holds for face‐to‐face interactions of humans. This suggests an evolutionary shift in the carrying role from the gestural to the vocal stream, and we explore this transition in the carrying modality. Finally, we suggest that future studies should focus on the links between complex communication, sociality and cooperative tendency to strengthen the study of language origins.     

Front Cover

This review highlights the scientific advances concerning the origins of human right‐handedness and language (speech and gestures). The comparative approach we adopted provides evidence that research on human and non‐human animals’ behavioural asymmetries helps understand the processes that lead to the strong human left‐hemisphere specialisation. We review four major non‐mutually exclusive environmental factors that are likely to have shaped the evolution of human and non‐human primates’ manual asymmetry: socioecological lifestyle, postural characteristics, task‐level complexity and tool use. We hypothesise the following scenario for the evolutionary origins of human right‐handedness: the right‐direction of modern humans’ manual laterality would have emerged from our ecological (terrestrial) and social (multilevel system) lifestyle; then, it would have been strengthened by the gradual adoption of the bipedal stance associated with bipedal locomotion, and the increasing level of complexity of our daily tasks including bimanual coordinated actions and tool use. Although hemispheric functional lateralisation has been shaped through evolution, reports indicate that many factors and their mutual intertwinement can modulate human and non‐human primates’ manual laterality throughout their life cycle: genetic and environmental factors, mainly individual sociodemographic characteristics (e.g., age, sex and rank), behavioural characteristics (e.g., gesture per se and gestural sensory modality) and context‐related characteristics (e.g., emotional context and position of target). These environmental (evolutionary and life cycle) factors could also have influenced primates’ manual asymmetry indirectly through epigenetic modifications. All these findings led us to propose the hypothesis of a multicausal origin of human right‐handedness.     

Stone tools, language and the brain in human evolution

Long-standing speculations and more recent hypotheses propose a variety of possible evolutionary connections between language, gesture and tool use. These arguments have received important new support from neuroscientific research on praxis, observational action understanding and vocal language demonstrating substantial functional/anatomical overlap between these behaviours. However, valid reasons for scepticism remain as well as substantial differences in detail between alternative evolutionary hypotheses. Here, we review the current status of alternative 'gestural' and 'technological' hypotheses of language origins, drawing on current evidence of the neural bases of speech and tool use generally, and on recent studies of the neural correlates of Palaeolithic technology specifically.     

The origins of gestures and language: history,current advances and proposed theories

Investigating in depth the mechanisms underlying human and non‐human primate intentional communication systems (involving gestures, vocalisations, facial expressions and eye behaviours) can shed light on the evolutionary roots of language. Reports on non‐human primates, particularly great apes, suggest that gestural communication would have been a crucial prerequisite for the emergence of language, mainly based on the evidence of large communication repertoires and their associated multifaceted nature of intentionality that are key properties of language. Such research fuels important debates on the origins of gestures and language. We review here three non‐mutually exclusive processes that can explain mainly great apes' gestural acquisition and development: phylogenetic ritualisation, ontogenetic ritualisation, and learning via social negotiation. We hypothesise the following scenario for the evolutionary origins of gestures: gestures would have appeared gradually through evolution via signal ritualisation following the principle of derived activities, with the key involvement of emotional expression and processing. The increasing level of complexity of socioecological lifestyles and associated daily manipulative activities might then have enabled the acquisition and development of different interactional strategies throughout the life cycle. Many studies support a multimodal origin of language. However, we stress that the origins of language are not only multimodal, but more broadly multicausal. We propose a multicausal theory of language origins which better explains current findings. It postulates that primates' communicative signalling is a complex trait continually shaped by a cost–benefit trade‐off of signal production and processing of interactants in relation to four closely interlinked categories of evolutionary and life cycle factors: species, individual and context‐related characteristics as well as behaviour and its characteristics. We conclude by suggesting directions for future research to improve our understanding of the evolutionary roots of gestures and language.     

"Meaning-making" in language and biology

The linguistic metaphor in biology adheres to a representational theory that seeks similarities between pre-given domains. The point of departure of this paper is the generative and nonrepresentational conception of metaphor. This paper argues that by adopting the nonrepresentational conception of metaphor, meaning-making may be the appropriate perspective for understanding biological systems. In both cases (the linguistic and the biological), boundary conditions between different levels of organization use micro-level disorganization to create macro-level organization.     

Aging in language dynamics

Human languages evolve continuously, and a puzzling problem is how to reconcile the apparent robustness of most of the deep linguistic structures we use with the evidence that they undergo possibly slow, yet ceaseless, changes. Is the state in which we observe languages today closer to what would be a dynamical attractor with statistically stationary properties or rather closer to a non-steady state slowly evolving in time? Here we address this question in the framework of the emergence of shared linguistic categories in a population of individuals interacting through language games. The observed emerging asymptotic categorization, which has been previously tested - with success - against experimental data from human languages, corresponds to a metastable state where global shifts are always possible but progressively more unlikely and the response properties depend on the age of the system. This aging mechanism exhibits striking quantitative analogies to what is observed in the statistical mechanics of glassy systems. We argue that this can be a general scenario in language dynamics where shared linguistic conventions would not emerge as attractors, but rather as metastable states.     

A neuropsychological perspective on the link between language and praxis in modern humans

Hypotheses about the emergence of human cognitive abilities postulate strong evolutionary links between language and praxis, including the possibility that language was originally gestural. The present review considers functional and neuroanatomical links between language and praxis in brain-damaged patients with aphasia and/or apraxia. The neural systems supporting these functions are predominantly located in the left hemisphere. There are many parallels between action and language for recognition, imitation and gestural communication suggesting that they rely partially on large, common networks, differentially recruited depending on the nature of the task. However, this relationship is not unequivocal and the production and understanding of gestural communication are dependent on the context in apraxic patients and remains to be clarified in aphasic patients. The phonological, semantic and syntactic levels of language seem to share some common cognitive resources with the praxic system. In conclusion, neuropsychological observations do not allow support or rejection of the hypothesis that gestural communication may have constituted an evolutionary link between tool use and language. Rather they suggest that the complexity of human behaviour is based on large interconnected networks and on the evolution of specific properties within strategic areas of the left cerebral hemisphere.     

Prediction of psychosis across protocols and risk cohorts using automated language analysis

Language and speech are the primary source of data for psychiatrists to diagnose and treat mental disorders. In psychosis, the very structure of language can be disturbed, including semantic coherence (e.g., derailment and tangentiality) and syntactic complexity (e.g., concreteness). Subtle disturbances in language are evident in schizophrenia even prior to first psychosis onset, during prodromal stages. Using computer‐based natural language processing analyses, we previously showed that, among English‐speaking clinical (e.g., ultra) high‐risk youths, baseline reduction in semantic coherence (the flow of meaning in speech) and in syntactic complexity could predict subsequent psychosis onset with high accuracy. Herein, we aimed to cross‐validate these automated linguistic analytic methods in a second larger risk cohort, also English‐speaking, and to discriminate speech in psychosis from normal speech. We identified an automated machine‐learning speech classifier – comprising decreased semantic coherence, greater variance in that coherence, and reduced usage of possessive pronouns – that had an 83% accuracy in predicting psychosis onset (intra‐protocol), a cross‐validated accuracy of 79% of psychosis onset prediction in the original risk cohort (cross‐protocol), and a 72% accuracy in discriminating the speech of recent‐onset psychosis patients from that of healthy individuals. The classifier was highly correlated with previously identified manual linguistic predictors. Our findings support the utility and validity of automated natural language processing methods to characterize disturbances in semantics and syntax across stages of psychotic disorder. The next steps will be to apply these methods in larger risk cohorts to further test reproducibility, also in languages other than English, and identify sources of variability. This technology has the potential to improve prediction of psychosis outcome among at‐risk youths and identify linguistic targets for remediation and preventive intervention. More broadly, automated linguistic analysis can be a powerful tool for diagnosis and treatment across neuropsychiatry.     

Worldwide analysis of multiple microsatellites: language diversity has a detectable influence on DNA diversity

Previous studies of the correlations between the languages spoken by human populations and the genes carried by the members of those populations have been limited by the small amount of genetic markers available and by approximations in the treatment of linguistic data. In this study we analyzed a large collection of polymorphic microsatellite loci (377), distributed on all autosomes, and used Ruhlen's linguistic classification, to investigate the relative roles of geography and language in shaping the distribution of human DNA diversity at a worldwide scale. For this purpose, we performed three different kinds of analysis: (i) we partitioned genetic variances at three hierarchical levels of population subdivision according to language group by means of a molecular analysis of variance (AMOVA); (ii) we quantified by a series of Mantel's tests the correlation between measures of genetic and linguistic differentiation; and (iii) we tested whether linguistic differences are increased across known zones of increased genetic change between populations. Genetic differences appear to more closely reflect geographic than linguistic differentiation. However, our analyses show that language differences also have a detectable effect on DNA diversity at the genomic level, above and beyond the effects of geographic distance.     

The bridge of iconicity: from a world of experience to the experience of language

Iconicity, a resemblance between properties of linguistic form (both in spoken and signed languages) and meaning, has traditionally been considered to be a marginal, irrelevant phenomenon for our understanding of language processing, development and evolution. Rather, the arbitrary and symbolic nature of language has long been taken as a design feature of the human linguistic system. In this paper, we propose an alternative framework in which iconicity in face-to-face communication (spoken and signed) is a powerful vehicle for bridging between language and human sensori-motor experience, and, as such, iconicity provides a key to understanding language evolution, development and processing. In language evolution, iconicity might have played a key role in establishing displacement (the ability of language to refer beyond what is immediately present), which is core to what language does; in ontogenesis, iconicity might play a critical role in supporting referentiality (learning to map linguistic labels to objects, events, etc., in the world), which is core to vocabulary development. Finally, in language processing, iconicity could provide a mechanism to account for how language comes to be embodied (grounded in our sensory and motor systems), which is core to meaningful communication.     

How hierarchical is language use?

It is generally assumed that hierarchical phrase structure plays a central role in human language. However, considerations of simplicity and evolutionary continuity suggest that hierarchical structure should not be invoked too hastily. Indeed, recent neurophysiological, behavioural and computational studies show that sequential sentence structure has considerable explanatory power and that hierarchical processing is often not involved. In this paper, we review evidence from the recent literature supporting the hypothesis that sequential structure may be fundamental to the comprehension, production and acquisition of human language. Moreover, we provide a preliminary sketch outlining a non-hierarchical model of language use and discuss its implications and testable predictions. If linguistic phenomena can be explained by sequential rather than hierarchical structure, this will have considerable impact in a wide range of fields, such as linguistics, ethology, cognitive neuroscience, psychology and computer science.     

A structure-based repertoire of manual gestures in wild chimpanzees: statistical analyses of a graded communication system

Great ape gestural communication is considered important in understanding the evolution of human language as these share important features, namely, flexible and intentional signal use. Although gestural repertoires have been compiled for captive and wild primates, reports are largely qualitative. We quantify the morphological structure and variation of gestural signals identified in the repertoire of a community of wild chimpanzees. Gestures were classified on the basis of 29 morphological features, such as trajectory and orientation during the preparatory and stroke phases of a gesture. Hierarchical cluster and discriminant function analyses identified 30 morphologically distinct manual gesture types; the majority was subsequently correctly classified using a cross-validation technique, with incorrect classifications for rare gesture types only. Comparisons of this statistically determined repertoire with previous repertoires did not identify systematic variation between captive and wild chimpanzees. Moreover, consensus was not greater within studies of the same populations, highlighting the importance of systematic and well-documented inventories. Our morphologically based analyses indicate that manual gestures are best considered as graded rather than discrete communication signals, similar to some vocalisation systems. We discuss these findings in light of current theories of human language evolution.     

Cost and complexity: selection for speech and language

The handicap principle has been applied to a number of different traits in the last three decades, but it is difficult to characterize its record, or even its perceived relevance, when it comes to an important human attribute—spoken language. In some cases, assumptions regarding the energetic cost of speech, and the veracity of linguistically encoded messages, have failed to recognize critical aspects of human development, cognition, and social ecology. In other cases, the fact that speech contains honest (physiological) information, and tends to be used honestly with family and friends, has been overlooked. Speech and language are functionally related but they involve different resources. Individuals can increase the attractiveness of their speech, and of more stylized vocal and verbal performances, without enhancing linguistic structure or content; and they can modify their use of language without significant changes in the physical form of speech. That its production costs are normally low enables speech to be produced extravagantly in bids for status and mating relationships, and in evolution, may have allowed its content—linguistic knowledge and structure—to become complex.     

Neural correlates of Early Stone Age toolmaking: technology, language and cognition in human evolution

Archaeological and palaeontological evidence from the Early Stone Age (ESA) documents parallel trends of brain expansion and technological elaboration in human evolution over a period of more than 2Myr. However, the relationship between these defining trends remains controversial and poorly understood. Here, we present results from a positron emission tomography study of functional brain activation during experimental ESA (Oldowan and Acheulean) toolmaking by expert subjects. Together with a previous study of Oldowan toolmaking by novices, these results document increased demands for effective visuomotor coordination and hierarchical action organization in more advanced toolmaking. This includes an increased activation of ventral premotor and inferior parietal elements of the parietofrontal praxis circuits in both the hemispheres and of the right hemisphere homologue of Broca's area. The observed patterns of activation and of overlap with language circuits suggest that toolmaking and language share a basis in more general human capacities for complex, goal-directed action. The results are consistent with coevolutionary hypotheses linking the emergence of language, toolmaking, population-level functional lateralization and association cortex expansion in human evolution.