Language as a multimodal phenomenon: implications for language learning,processing and evolution

Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic gestures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages.     

The invasion of language: emergence, change and death

Research into the emergence and evolution of human language has received unprecedented attention during the past 15 years. Efforts to better understand the processes of language emergence and evolution have proceeded in two main directions: from the top-down (linguists) and from the bottom-up (cognitive scientists). Language can be viewed as an invading process that has had profound impact on the human phenotype at all levels, from the structure of the brain to modes of cultural interaction. In our view, the most effective way to form a connection between the two efforts (essential if theories for language evolution are to reflect the constraints imposed on language by the brain) lies in computational modelling, an approach that enables numerous hypotheses to be explored and tested against objective criteria and which suggest productive paths for empirical researchers to then follow. Here, with the aim of promoting the cross-fertilization of ideas across disciplines, we review some of the recent research that has made use of computational methods in three principal areas of research into language evolution: language emergence, language change, and language death.     

Chaos and language

Human language is a complex communication system with unlimited expressibility. Children spontaneously develop a native language by exposure to linguistic data from their speech community. Over historical time, languages change dramatically and unpredictably by accumulation of small changes and by interaction with other languages. We have previously developed a mathematical model for the acquisition and evolution of language in heterogeneous populations of speakers. This model is based on game dynamical equations with learning. Here, we show that simple examples of such equations can display complex limit cycles and chaos. Hence, language dynamical equations mimic complicated and unpredictable changes of languages over time. In terms of evolutionary game theory, we note that imperfect learning can induce chaotic switching among strict Nash equilibria.     

How hierarchical is language use?

It is generally assumed that hierarchical phrase structure plays a central role in human language. However, considerations of simplicity and evolutionary continuity suggest that hierarchical structure should not be invoked too hastily. Indeed, recent neurophysiological, behavioural and computational studies show that sequential sentence structure has considerable explanatory power and that hierarchical processing is often not involved. In this paper, we review evidence from the recent literature supporting the hypothesis that sequential structure may be fundamental to the comprehension, production and acquisition of human language. Moreover, we provide a preliminary sketch outlining a non-hierarchical model of language use and discuss its implications and testable predictions. If linguistic phenomena can be explained by sequential rather than hierarchical structure, this will have considerable impact in a wide range of fields, such as linguistics, ethology, cognitive neuroscience, psychology and computer science.     

Action at a distance: dependency sensitivity in a New World primate

Sensitivity to dependencies (correspondences between distant items) in sensory stimuli plays a crucial role in human music and language. Here, we show that squirrel monkeys (Saimiri sciureus) can detect abstract, non-adjacent dependencies in auditory stimuli. Monkeys discriminated between tone sequences containing a dependency and those lacking it, and generalized to previously unheard pitch classes and novel dependency distances. This constitutes the first pattern learning study where artificial stimuli were designed with the species'' communication system in mind. These results suggest that the ability to recognize dependencies represents a capability that had already evolved in humans’ last common ancestor with squirrel monkeys, and perhaps before.     

Evolution of protolinguistic abilities as a by-product of learning to forage in structured environments

The skills required for the learning and use of language are the focus of extensive research, and their evolutionary origins are widely debated. Using agent-based simulations in a range of virtual environments, we demonstrate that challenges of foraging for food can select for cognitive mechanisms supporting complex, hierarchical, sequential learning, the need for which arises in language acquisition. Building on previous work, where we explored the conditions under which reinforcement learning is out-competed by seldom-reinforced continuous learning that constructs a network model of the environment, we now show that realistic features of the foraging environment can select for two critical advances: (i) chunking of meaningful sequences found in the data, leading to representations composed of units that better fit the prevalent statistical patterns in the environment; and (ii) generalization across units based on their contextual similarity. Importantly, these learning processes, which in our framework evolved for making better foraging decisions, had been earlier shown to reproduce a range of findings in language learning in humans. Thus, our results suggest a possible evolutionary trajectory that may have led from basic learning mechanisms to complex hierarchical sequential learning that can support advanced cognitive abilities of the kind needed for language acquisition.     

Biological adaptations for functional features of language in the face of cultural evolution

Although there may be no true language universals, it is nonetheless possible to discern several family resemblance patterns across the languages of the world. Recent work on the cultural evolution of language indicates the source of these patterns is unlikely to be an innate universal grammar evolved through biological adaptations for arbitrary linguistic features. Instead, it has been suggested that the patterns of resemblance emerge because language has been shaped by the brain, with individual languages representing different but partially overlapping solutions to the same set of nonlinguistic constraints. Here, we use computational simulations to investigate whether biological adaptation for functional features of language, deriving from cognitive and communicative constraints, may nonetheless be possible alongside rapid cultural evolution. Specifically, we focus on the Baldwin effect as an evolutionary mechanism by which previously learned linguistic features might become innate through natural selection across many generations of language users. The results indicate that cultural evolution of language does not necessarily prevent functional features of language from becoming genetically fixed, thus potentially providing a particularly informative source of constraints on cross-linguistic resemblance patterns.     

All Things in Mind: Panpsychist Elements in Spinoza, Deleuze, and Peirce

Benedict de Spinoza, C.S. Peirce, and Gilles Deleuze delineate a trajectory through the history of ideas in the dialogue about the potentials and limitations of panpsychism, the view that world is fundamentally made up of mind. As a parallel trajectory to the panpsychism debate in contemporary philosophy of mind and cognitive psychology, this approach can inform and enrich the discussion of the role and scope of mind in the natural world. The philosophies of mind developed by Deleuze and Peirce are Spinozistic in their natural monism but move beyond Spinoza to explain mind as a part of the natural world in semiotic terms.     

Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system

The prefrontal cortex (PFC) has long been thought to serve as an 'executive' that controls the selection of actions and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified often amounting to a homunculus. This paper reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a 'Go' versus 'NoGo' modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.     

Orangutan information broadcast via consonant-like and vowel-like calls breaches mathematical models of linguistic evolution

The origin of language is one of the most significant evolutionary milestones of life on Earth, but one of the most persevering scientific unknowns. Two decades ago, game theorists and mathematicians predicted that the first words and grammar emerged as a response to transmission errors and information loss in language''s precursor system, however, empirical proof is lacking. Here, we assessed information loss in proto-consonants and proto-vowels in human pre-linguistic ancestors as proxied by orangutan consonant-like and vowel-like calls that compose syllable-like combinations. We played back and re-recorded calls at increasing distances across a structurally complex habitat (i.e. adverse to sound transmission). Consonant-like and vowel-like calls degraded acoustically over distance, but no information loss was detected regarding three distinct classes of information (viz. individual ID, context and population ID). Our results refute prevailing mathematical predictions and herald a turning point in language evolution theory and heuristics. Namely, explaining how the vocal–verbal continuum was crossed in the hominid family will benefit from future mathematical and computational models that, in order to enjoy empirical validity and superior explanatory power, will be informed by great ape behaviour and repertoire.     

A Supervisory Control Approach for Execution Control of an FMC

This paper presents a generic methodology based on formal language theory for the modeling and control of flexible manufacturing cell (FMC) systems. The motivating idea behind the overall approach stems from the supervisory control theory under the framework of Ramadge and Wonham. Essentially, we characterize the asynchronous and dynamic behavior of an FMC as a regular language and formulate the control logic generation problem as a sublanguage calculation problem, which requires the resulting language to satisfy at least two properties: maximal permissiveness and controllability. Then an algorithm for resolving the problem is presented. Based on the solution of the problem called supervisor, we propose a controller architecture that guarantees coordinated operation of an FMC through the regulation of occurrences of events. An adaptive control policy that regenerates supervisors on changes in task configurations is presented and a dynamic equation that describes the evolution of the control logic along time is derived. Then, we show that the proposed maximally permissive adaptive control policy has a number of preferred properties, including computational efficiency and consistency between the successive supervisors. Finally, a controller for an example FMC is implemented, using the object-oriented software modules. Our procedure has the merit of mathematical soundness, modular design, and systematic implementation.     

Visual immersion for cultural understanding and multimodal literacy

ABSTRACT

When considering inclusive art curriculum that accommodates all learners, including English language learners, two distinct yet inseparable issues come to mind. The first is that English language learner students can use visual language and visual literacy skills inherent in visual arts curriculum to scaffold learning in and through the arts. Second, in facilitating a sense of belonging for students whose home language and cultural aesthetic may be different from those of the dominant school culture, an authentically developed multicultural art curriculum can guide self-efficacy and inclusiveness. Both aspects of teaching art for English language learners can have the added benefits of facilitating collaborative learning opportunities and increasing worldviews for all students.     

When words fail us: insights into language processing from developmental and acquired disorders

Acquired disorders of language represent loss of previously acquired skills, usually with relatively specific impairments. In children with developmental disorders of language, we may also see selective impairment in some skills; but in this case, the acquisition of language or literacy is affected from the outset. Because systems for processing spoken and written language change as they develop, we should beware of drawing too close a parallel between developmental and acquired disorders. Nevertheless, comparisons between the two may yield new insights. A key feature of connectionist models simulating acquired disorders is the interaction of components of language processing with each other and with other cognitive domains. This kind of model might help make sense of patterns of comorbidity in developmental disorders. Meanwhile, the study of developmental disorders emphasizes learning and change in underlying representations, allowing us to study how heterogeneity in cognitive profile may relate not just to neurobiology but also to experience. Children with persistent language difficulties pose challenges both to our efforts at intervention and to theories of learning of written and spoken language. Future attention to learning in individuals with developmental and acquired disorders could be of both theoretical and applied value.     

The P-chain: relating sentence production and its disorders to comprehension and acquisition

This article introduces the P-chain, an emerging framework for theory in psycholinguistics that unifies research on comprehension, production and acquisition. The framework proposes that language processing involves incremental prediction, which is carried out by the production system. Prediction necessarily leads to prediction error, which drives learning, including both adaptive adjustment to the mature language processing system as well as language acquisition. To illustrate the P-chain, we review the Dual-path model of sentence production, a connectionist model that explains structural priming in production and a number of facts about language acquisition. The potential of this and related models for explaining acquired and developmental disorders of sentence production is discussed.     

Grist and mills: on the cultural origins of cultural learning

Cumulative cultural evolution is what 'makes us odd'; our capacity to learn facts and techniques from others, and to refine them over generations, plays a major role in making human minds and lives radically different from those of other animals. In this article, I discuss cognitive processes that are known collectively as 'cultural learning' because they enable cumulative cultural evolution. These cognitive processes include reading, social learning, imitation, teaching, social motivation and theory of mind. Taking the first of these three types of cultural learning as examples, I ask whether and to what extent these cognitive processes have been adapted genetically or culturally to enable cumulative cultural evolution. I find that recent empirical work in comparative psychology, developmental psychology and cognitive neuroscience provides surprisingly little evidence of genetic adaptation, and ample evidence of cultural adaptation. This raises the possibility that it is not only 'grist' but also 'mills' that are culturally inherited; through social interaction in the course of development, we not only acquire facts about the world and how to deal with it (grist), we also build the cognitive processes that make 'fact inheritance' possible (mills).     

Incidental Acquisition of Foreign Language Vocabulary through Brief Multi-Modal Exposure

First language acquisition requires relatively little effort compared to foreign language acquisition and happens more naturally through informal learning. Informal exposure can also benefit foreign language learning, although evidence for this has been limited to speech perception and production. An important question is whether informal exposure to spoken foreign language also leads to vocabulary learning through the creation of form-meaning links. Here we tested the impact of exposure to foreign language words presented with pictures in an incidental learning phase on subsequent explicit foreign language learning. In the explicit learning phase, we asked adults to learn translation equivalents of foreign language words, some of which had appeared in the incidental learning phase. Results revealed rapid learning of the foreign language words in the incidental learning phase showing that informal exposure to multi-modal foreign language leads to foreign language vocabulary acquisition. The creation of form-meaning links during the incidental learning phase is discussed.     

Dreaming and the Impossible Art of Translation

This essay identifies translation both as an historically core metaphor in Euro-American dream theory and as a contemporary ally for dream teaching and research. It also traces the mutuality of translators' reliance on Freud, Jung, and other dream theorists in their metaphorical expression of the art and craft of translation. The sustained interaction of oneiric and linguistic metaphor-making stems from the fact that virtually all of our knowledge about dreams has been mediated through language and that all dream reporting is itself an act of intersemiotic translation. In an attempt to stimulate further comparative inquiry, the overlapping concerns and often uncanny affinities between language/translation studies and dream/interpretation studies are presented.     

Neurophysiological mechanisms involved in language learning in adults

Little is known about the brain mechanisms involved in word learning during infancy and in second language acquisition and about the way these new words become stable representations that sustain language processing. In several studies we have adopted the human simulation perspective, studying the effects of brain-lesions and combining different neuroimaging techniques such as event-related potentials and functional magnetic resonance imaging in order to examine the language learning (LL) process. In the present article, we review this evidence focusing on how different brain signatures relate to (i) the extraction of words from speech, (ii) the discovery of their embedded grammatical structure, and (iii) how meaning derived from verbal contexts can inform us about the cognitive mechanisms underlying the learning process. We compile these findings and frame them into an integrative neurophysiological model that tries to delineate the major neural networks that might be involved in the initial stages of LL. Finally, we propose that LL simulations can help us to understand natural language processing and how the recovery from language disorders in infants and adults can be accomplished.     

The bridge of iconicity: from a world of experience to the experience of language

Iconicity, a resemblance between properties of linguistic form (both in spoken and signed languages) and meaning, has traditionally been considered to be a marginal, irrelevant phenomenon for our understanding of language processing, development and evolution. Rather, the arbitrary and symbolic nature of language has long been taken as a design feature of the human linguistic system. In this paper, we propose an alternative framework in which iconicity in face-to-face communication (spoken and signed) is a powerful vehicle for bridging between language and human sensori-motor experience, and, as such, iconicity provides a key to understanding language evolution, development and processing. In language evolution, iconicity might have played a key role in establishing displacement (the ability of language to refer beyond what is immediately present), which is core to what language does; in ontogenesis, iconicity might play a critical role in supporting referentiality (learning to map linguistic labels to objects, events, etc., in the world), which is core to vocabulary development. Finally, in language processing, iconicity could provide a mechanism to account for how language comes to be embodied (grounded in our sensory and motor systems), which is core to meaningful communication.     

Understanding Language Evolution: Beyond Pan-Centrism

Language does not fossilize but this does not mean that the language's evolutionary timeline is lost forever. Great apes provide a window back in time on our last prelinguistic ancestor's communication and cognition. Phylogeny and cladistics implicitly conjure Pan (chimpanzees, bonobos) as a superior (often the only) model for language evolution compared with earlier diverging lineages, Gorilla and Pongo (orangutans). Here, in reviewing the literature, it is shown that Pan do not surpass other great apes along genetic, cognitive, ecologic, or vocal traits that are putatively paramount for language onset and evolution. Instead, revived herein is the idea that only by abandoning single-species models and learning about the variation among great apes, there might be a chance to retrieve lost fragments of the evolutionary timeline of language.