Language as a multimodal phenomenon: implications for language learning,processing and evolution

Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic gestures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages.     

The sound symbolism bootstrapping hypothesis for language acquisition and language evolution

Sound symbolism is a non-arbitrary relationship between speech sounds and meaning. We review evidence that, contrary to the traditional view in linguistics, sound symbolism is an important design feature of language, which affects online processing of language, and most importantly, language acquisition. We propose the sound symbolism bootstrapping hypothesis, claiming that (i) pre-verbal infants are sensitive to sound symbolism, due to a biologically endowed ability to map and integrate multi-modal input, (ii) sound symbolism helps infants gain referential insight for speech sounds, (iii) sound symbolism helps infants and toddlers associate speech sounds with their referents to establish a lexical representation and (iv) sound symbolism helps toddlers learn words by allowing them to focus on referents embedded in a complex scene, alleviating Quine''s problem. We further explore the possibility that sound symbolism is deeply related to language evolution, drawing the parallel between historical development of language across generations and ontogenetic development within individuals. Finally, we suggest that sound symbolism bootstrapping is a part of a more general phenomenon of bootstrapping by means of iconic representations, drawing on similarities and close behavioural links between sound symbolism and speech-accompanying iconic gesture.     

Hearing and seeing meaning in speech and gesture: insights from brain and behaviour

As we speak, we use not only the arbitrary form–meaning mappings of the speech channel but also motivated form–meaning correspondences, i.e. iconic gestures that accompany speech (e.g. inverted V-shaped hand wiggling across gesture space to demonstrate walking). This article reviews what we know about processing of semantic information from speech and iconic gestures in spoken languages during comprehension of such composite utterances. Several studies have shown that comprehension of iconic gestures involves brain activations known to be involved in semantic processing of speech: i.e. modulation of the electrophysiological recording component N400, which is sensitive to the ease of semantic integration of a word to previous context, and recruitment of the left-lateralized frontal–posterior temporal network (left inferior frontal gyrus (IFG), medial temporal gyrus (MTG) and superior temporal gyrus/sulcus (STG/S)). Furthermore, we integrate the information coming from both channels recruiting brain areas such as left IFG, posterior superior temporal sulcus (STS)/MTG and even motor cortex. Finally, this integration is flexible: the temporal synchrony between the iconic gesture and the speech segment, as well as the perceived communicative intent of the speaker, modulate the integration process. Whether these findings are special to gestures or are shared with actions or other visual accompaniments to speech (e.g. lips) or other visual symbols such as pictures are discussed, as well as the implications for a multimodal view of language.     

The origin of human multi-modal communication

One reason for the apparent gulf between animal and human communication systems is that the focus has been on the presence or the absence of language as a complex expressive system built on speech. But language normally occurs embedded within an interactional exchange of multi-modal signals. If this larger perspective takes central focus, then it becomes apparent that human communication has a layered structure, where the layers may be plausibly assigned different phylogenetic and evolutionary origins—especially in the light of recent thoughts on the emergence of voluntary breathing and spoken language. This perspective helps us to appreciate the different roles that the different modalities play in human communication, as well as how they function as one integrated system despite their different roles and origins. It also offers possibilities for reconciling the ‘gesture-first hypothesis’ with that of gesture and speech having evolved together, hand in hand—or hand in mouth, rather—as one system.     

Widening the lens: what the manual modality reveals about language,learning and cognition

The goal of this paper is to widen the lens on language to include the manual modality. We look first at hearing children who are acquiring language from a spoken language model and find that even before they use speech to communicate, they use gesture. Moreover, those gestures precede, and predict, the acquisition of structures in speech. We look next at deaf children whose hearing losses prevent them from using the oral modality, and whose hearing parents have not presented them with a language model in the manual modality. These children fall back on the manual modality to communicate and use gestures, which take on many of the forms and functions of natural language. These homemade gesture systems constitute the first step in the emergence of manual sign systems that are shared within deaf communities and are full-fledged languages. We end by widening the lens on sign language to include gesture and find that signers not only gesture, but they also use gesture in learning contexts just as speakers do. These findings suggest that what is key in gesture''s ability to predict learning is its ability to add a second representational format to communication, rather than a second modality. Gesture can thus be language, assuming linguistic forms and functions, when other vehicles are not available; but when speech or sign is possible, gesture works along with language, providing an additional representational format that can promote learning.     

Stone tools, language and the brain in human evolution

Long-standing speculations and more recent hypotheses propose a variety of possible evolutionary connections between language, gesture and tool use. These arguments have received important new support from neuroscientific research on praxis, observational action understanding and vocal language demonstrating substantial functional/anatomical overlap between these behaviours. However, valid reasons for scepticism remain as well as substantial differences in detail between alternative evolutionary hypotheses. Here, we review the current status of alternative 'gestural' and 'technological' hypotheses of language origins, drawing on current evidence of the neural bases of speech and tool use generally, and on recent studies of the neural correlates of Palaeolithic technology specifically.     

The shape and tempo of language evolution

There are approximately 7000 languages spoken in the world today. This diversity reflects the legacy of thousands of years of cultural evolution. How far back we can trace this history depends largely on the rate at which the different components of language evolve. Rates of lexical evolution are widely thought to impose an upper limit of 6000–10 000 years on reliably identifying language relationships. In contrast, it has been argued that certain structural elements of language are much more stable. Just as biologists use highly conserved genes to uncover the deepest branches in the tree of life, highly stable linguistic features hold the promise of identifying deep relationships between the world''s languages. Here, we present the first global network of languages based on this typological information. We evaluate the relative evolutionary rates of both typological and lexical features in the Austronesian and Indo-European language families. The first indications are that typological features evolve at similar rates to basic vocabulary but their evolution is substantially less tree-like. Our results suggest that, while rates of vocabulary change are correlated between the two language families, the rates of evolution of typological features and structural subtypes show no consistent relationship across families.     

Exorcising Grice's ghost: an empirical approach to studying intentional communication in animals

Language's intentional nature has been highlighted as a crucial feature distinguishing it from other communication systems. Specifically, language is often thought to depend on highly structured intentional action and mutual mindreading by a communicator and recipient. Whilst similar abilities in animals can shed light on the evolution of intentionality, they remain challenging to detect unambiguously. We revisit animal intentional communication and suggest that progress in identifying analogous capacities has been complicated by (i) the assumption that intentional (that is, voluntary) production of communicative acts requires mental‐state attribution, and (ii) variation in approaches investigating communication across sensory modalities. To move forward, we argue that a framework fusing research across modalities and species is required. We structure intentional communication into a series of requirements, each of which can be operationalised, investigated empirically, and must be met for purposive, intentionally communicative acts to be demonstrated. Our unified approach helps elucidate the distribution of animal intentional communication and subsequently serves to clarify what is meant by attributions of intentional communication in animals and humans.     

Evidence of an evolutionary precursor to human language affixation in a non-human primate

Human language, and grammatical competence in particular, relies on a set of computational operations that, in its entirety, is not observed in other animals. Such uniqueness leaves open the possibility that components of our linguistic competence are shared with other animals, having evolved for non-linguistic functions. Here, we explore this problem from a comparative perspective, asking whether cotton-top tamarin monkeys (Saguinus oedipus) can spontaneously (no training) acquire an affixation rule that shares important properties with our inflectional morphology (e.g. the rule that adds –ed to create the past tense, as in the transformation of walk into walk-ed). Using playback experiments, we show that tamarins discriminate between bisyllabic items that start with a specific ‘prefix’ syllable and those that end with the same syllable as a ‘suffix’. These results suggest that some of the computational mechanisms subserving affixation in a diversity of languages are shared with other animals, relying on basic perceptual or memory primitives that evolved for non-linguistic functions.     

Cultural evolution: implications for understanding the human language faculty and its evolution

Human language is unique among the communication systems of the natural world: it is socially learned and, as a consequence of its recursively compositional structure, offers open-ended communicative potential. The structure of this communication system can be explained as a consequence of the evolution of the human biological capacity for language or the cultural evolution of language itself. We argue, supported by a formal model, that an explanatory account that involves some role for cultural evolution has profound implications for our understanding of the biological evolution of the language faculty: under a number of reasonable scenarios, cultural evolution can shield the language faculty from selection, such that strongly constraining language-specific learning biases are unlikely to evolve. We therefore argue that language is best seen as a consequence of cultural evolution in populations with a weak and/or domain-general language faculty.     

Hominid cultural transmission and the evolution of language

This paper presents the hypothesis that linguistic capacity evolved through the action of natural selection as an instrument which increased the efficiency of the cultural transmission system of early hominids. We suggest that during the early stages of hominization, hominid social learning, based on indirect social learning mechanisms and true imitation, came to constitute cumulative cultural transmission based on true imitation and the approval or disapproval of the learned behaviour of offspring. A key factor for this transformation was the development of a conceptual capacity for categorizing learned behaviour in value terms - positive or negative, good or bad. We believe that some hominids developed this capacity for categorizing behaviour, and such an ability allowed them to approve or disapprove of their offsprings- learned behaviour. With such an ability, hominids were favoured, as they could transmit to their offspring all their behavioural experience about what can and cannot be done. This capacity triggered a cultural transmission system similar to the human one, though pre-linguistic. We suggest that the adaptive advantage provided by this new system of social learning generated a selection pressure in favour of the development of a linguistic capacity allowing children to better understand the new kind of evaluative information received from parents.     

The roads to and from the RNA world

The historical existence of the RNA world, in which early life used RNA for both genetic information and catalytic ability, is widely accepted. However, there has been little discussion of whether protein synthesis arose before DNA or what preceded the RNA world (i.e. the pre-RNA world). We outline arguments of what route life may have taken out of the RNA world: whether DNA or protein followed. Metabolic arguments favor the possibility that RNA genomes preceded the use of DNA as the informational macromolecule. However, the opposite can also be argued based on the enhanced stability, reactivity, and solubility of 2-deoxyribose as compared to ribose. The possibility that DNA may have come before RNA is discussed, although it is a less parsimonious explanation than DNA following RNA.     

Contributions of infant word learning to language development

Infants learn the forms of words by listening to the speech they hear. Though little is known about the degree to which these forms are meaningful for young infants, the words still play a role in early language development. Words guide the infant to his or her first syntactic intuitions, aid in the development of the lexicon, and, it is proposed, may help infants learn phonetic categories.     

From temporal processing to developmental language disorders: mind the gap

The ‘rapid temporal processing’ and the ‘temporal sampling framework’ hypotheses have been proposed to account for the deficits in language and literacy development seen in specific language impairment and dyslexia. This paper reviews these hypotheses and concludes that the proposed causal chains between the presumed auditory processing deficits and the observed behavioural manifestation of the disorders are vague and not well established empirically. Several problems and limitations are identified. Most data concern correlations between distantly related tasks, and there is considerable heterogeneity and variability in performance as well as concerns about reliability and validity. Little attention is paid to the distinction between ostensibly perceptual and metalinguistic tasks or between implicit and explicit modes of performance, yet measures are assumed to be pure indicators of underlying processes or representations. The possibility that diagnostic categories do not refer to causally and behaviourally homogeneous groups needs to be taken seriously, taking into account genetic and neurodevelopmental studies to construct multiple-risk models. To make progress in the field, cognitive models of each task must be specified, including performance domains that are predicted to be deficient versus intact, testing multiple indicators of latent constructs and demonstrating construct reliability and validity.     

Words as alleles: connecting language evolution with Bayesian learners to models of genetic drift

Scientists studying how languages change over time often make an analogy between biological and cultural evolution, with words or grammars behaving like traits subject to natural selection. Recent work has exploited this analogy by using models of biological evolution to explain the properties of languages and other cultural artefacts. However, the mechanisms of biological and cultural evolution are very different: biological traits are passed between generations by genes, while languages and concepts are transmitted through learning. Here we show that these different mechanisms can have the same results, demonstrating that the transmission of frequency distributions over variants of linguistic forms by Bayesian learners is equivalent to the Wright–Fisher model of genetic drift. This simple learning mechanism thus provides a justification for the use of models of genetic drift in studying language evolution. In addition to providing an explicit connection between biological and cultural evolution, this allows us to define a ‘neutral’ model that indicates how languages can change in the absence of selection at the level of linguistic variants. We demonstrate that this neutral model can account for three phenomena: the s-shaped curve of language change, the distribution of word frequencies, and the relationship between word frequencies and extinction rates.     

A systematic review of MRI studies of language development from birth to 2 years of age

Neurocognitive functions supporting language development start to develop well before first words are spoken during the first years of life. This process coincides with the initial growth spurt of the brain. While the core components of the language network are well characterized in adults and children, the initial neural correlates of language skills are still relatively unknown. We reviewed 10 studies identified via a systematic search that combined magnetic resonance imaging and language‐related measures in healthy infants from birth to 2 years of age. We aimed to describe the current knowledge as well as point out viable future directions for similar studies. Expectedly, the implicated cerebral areas included many established components of the language networks, including frontal and temporal regions. A volumetric leftward asymmetry of the brain was suggested as a determinant of language skills, yet with marked interindividual variation. Overall, temporal and frontal brain volumes associated positively with language skills. Positive associations were described between the maturation of language related white matter tracts and language skills. The language networks showed adult‐like structural similarities already in neonates, with weaker asymmetry compared to adults. In summary, we found some evidence that the language circuit described in older age groups is also associated to language skills during the first 2 years of life. However, across the reviewed studies there were no systematic neural correlates of language skills, which is partly explained by a modest number of studies, scattered representation of ages in measurements and the variance in the used methods.     

Phonetic learning as a pathway to language: new data and native language magnet theory expanded (NLM-e)

Infants' speech perception skills show a dual change towards the end of the first year of life. Not only does non-native speech perception decline, as often shown, but native language speech perception skills show improvement, reflecting a facilitative effect of experience with native language. The mechanism underlying change at this point in development, and the relationship between the change in native and non-native speech perception, is of theoretical interest. As shown in new data presented here, at the cusp of this developmental change, infants' native and non-native phonetic perception skills predict later language ability, but in opposite directions. Better native language skill at 7.5 months of age predicts faster language advancement, whereas better non-native language skill predicts slower advancement. We suggest that native language phonetic performance is indicative of neural commitment to the native language, while non-native phonetic performance reveals uncommitted neural circuitry. This paper has three goals: (i) to review existing models of phonetic perception development, (ii) to present new event-related potential data showing that native and non-native phonetic perception at 7.5 months of age predicts language growth over the next 2 years, and (iii) to describe a revised version of our previous model, the native language magnet model, expanded (NLM-e). NLM-e incorporates five new principles. Specific testable predictions for future research programmes are described.