The fractionation of spoken language understanding by measuring electrical and magnetic brain signals

This paper focuses on what electrical and magnetic recordings of human brain activity reveal about spoken language understanding. Based on the high temporal resolution of these recordings, a fine-grained temporal profile of different aspects of spoken language comprehension can be obtained. Crucial aspects of speech comprehension are lexical access, selection and semantic integration. Results show that for words spoken in context, there is no 'magic moment' when lexical selection ends and semantic integration begins. Irrespective of whether words have early or late recognition points, semantic integration processing is initiated before words can be identified on the basis of the acoustic information alone. Moreover, for one particular event-related brain potential (ERP) component (the N400), equivalent impact of sentence- and discourse-semantic contexts is observed. This indicates that in comprehension, a spoken word is immediately evaluated relative to the widest interpretive domain available. In addition, this happens very quickly. Findings are discussed that show that often an unfolding word can be mapped onto discourse-level representations well before the end of the word. Overall, the time course of the ERP effects is compatible with the view that the different information types (lexical, syntactic, phonological, pragmatic) are processed in parallel and influence the interpretation process incrementally, that is as soon as the relevant pieces of information are available. This is referred to as the immediacy principle.     

Different neurophysiological mechanisms underlying word and rule extraction from speech

The initial process of identifying words from spoken language and the detection of more subtle regularities underlying their structure are mandatory processes for language acquisition. Little is known about the cognitive mechanisms that allow us to extract these two types of information and their specific time-course of acquisition following initial contact with a new language. We report time-related electrophysiological changes that occurred while participants learned an artificial language. These changes strongly correlated with the discovery of the structural rules embedded in the words. These changes were clearly different from those related to word learning and occurred during the first minutes of exposition. There is a functional distinction in the nature of the electrophysiological signals during acquisition: an increase in negativity (N400) in the central electrodes is related to word-learning and development of a frontal positivity (P2) is related to rule-learning. In addition, the results of an online implicit and a post-learning test indicate that, once the rules of the language have been acquired, new words following the rule are processed as words of the language. By contrast, new words violating the rule induce syntax-related electrophysiological responses when inserted online in the stream (an early frontal negativity followed by a late posterior positivity) and clear lexical effects when presented in isolation (N400 modulation). The present study provides direct evidence suggesting that the mechanisms to extract words and structural dependencies from continuous speech are functionally segregated. When these mechanisms are engaged, the electrophysiological marker associated with rule-learning appears very quickly, during the earliest phases of exposition to a new language.     

Lexical learning and lexical processing in children with developmental language impairments

Lexical skills are a crucial component of language comprehension and production. This paper reviews evidence for lexical-level deficits in children and young people with developmental language impairment (LI). Across a range of tasks, LI is associated with reduced vocabulary knowledge in terms of both breadth and depth and difficulty with learning and retaining new words; evidence is emerging from on-line tasks to suggest that low levels of language skill are associated with differences in lexical competition in spoken word recognition. The role of lexical deficits in understanding the nature of LI is also discussed.     

Reading “Sun” and Looking Up: The Influence of Language on Saccadic Eye Movements in the Vertical Dimension

Traditionally, language processing has been attributed to a separate system in the brain, which supposedly works in an abstract propositional manner. However, there is increasing evidence suggesting that language processing is strongly interrelated with sensorimotor processing. Evidence for such an interrelation is typically drawn from interactions between language and perception or action. In the current study, the effect of words that refer to entities in the world with a typical location (e.g., sun, worm) on the planning of saccadic eye movements was investigated. Participants had to perform a lexical decision task on visually presented words and non-words. They responded by moving their eyes to a target in an upper (lower) screen position for a word (non-word) or vice versa. Eye movements were faster to locations compatible with the word''s referent in the real world. These results provide evidence for the importance of linguistic stimuli in directing eye movements, even if the words do not directly transfer directional information.     

A Complex Network Approach to Distributional Semantic Models

A number of studies on network analysis have focused on language networks based on free word association, which reflects human lexical knowledge, and have demonstrated the small-world and scale-free properties in the word association network. Nevertheless, there have been very few attempts at applying network analysis to distributional semantic models, despite the fact that these models have been studied extensively as computational or cognitive models of human lexical knowledge. In this paper, we analyze three network properties, namely, small-world, scale-free, and hierarchical properties, of semantic networks created by distributional semantic models. We demonstrate that the created networks generally exhibit the same properties as word association networks. In particular, we show that the distribution of the number of connections in these networks follows the truncated power law, which is also observed in an association network. This indicates that distributional semantic models can provide a plausible model of lexical knowledge. Additionally, the observed differences in the network properties of various implementations of distributional semantic models are consistently explained or predicted by considering the intrinsic semantic features of a word-context matrix and the functions of matrix weighting and smoothing. Furthermore, to simulate a semantic network with the observed network properties, we propose a new growing network model based on the model of Steyvers and Tenenbaum. The idea underlying the proposed model is that both preferential and random attachments are required to reflect different types of semantic relations in network growth process. We demonstrate that this model provides a better explanation of network behaviors generated by distributional semantic models.     

Structural Similarities between Brain and Linguistic Data Provide Evidence of Semantic Relations in the Brain

This paper presents a new method of analysis by which structural similarities between brain data and linguistic data can be assessed at the semantic level. It shows how to measure the strength of these structural similarities and so determine the relatively better fit of the brain data with one semantic model over another. The first model is derived from WordNet, a lexical database of English compiled by language experts. The second is given by the corpus-based statistical technique of latent semantic analysis (LSA), which detects relations between words that are latent or hidden in text. The brain data are drawn from experiments in which statements about the geography of Europe were presented auditorily to participants who were asked to determine their truth or falsity while electroencephalographic (EEG) recordings were made. The theoretical framework for the analysis of the brain and semantic data derives from axiomatizations of theories such as the theory of differences in utility preference. Using brain-data samples from individual trials time-locked to the presentation of each word, ordinal relations of similarity differences are computed for the brain data and for the linguistic data. In each case those relations that are invariant with respect to the brain and linguistic data, and are correlated with sufficient statistical strength, amount to structural similarities between the brain and linguistic data. Results show that many more statistically significant structural similarities can be found between the brain data and the WordNet-derived data than the LSA-derived data. The work reported here is placed within the context of other recent studies of semantics and the brain. The main contribution of this paper is the new method it presents for the study of semantics and the brain and the focus it permits on networks of relations detected in brain data and represented by a semantic model.     

How do we use language? Shared patterns in the frequency of word use across 17 world languages

We present data from 17 languages on the frequency with which a common set of words is used in everyday language. The languages are drawn from six language families representing 65 per cent of the world's 7000 languages. Our data were collected from linguistic corpora that record frequencies of use for the 200 meanings in the widely used Swadesh fundamental vocabulary. Our interest is to assess evidence for shared patterns of language use around the world, and for the relationship of language use to rates of lexical replacement, defined as the replacement of a word by a new unrelated or non-cognate word. Frequencies of use for words in the Swadesh list range from just a few per million words of speech to 191 000 or more. The average inter-correlation among languages in the frequency of use across the 200 words is 0.73 (p < 0.0001). The first principal component of these data accounts for 70 per cent of the variance in frequency of use. Elsewhere, we have shown that frequently used words in the Indo-European languages tend to be more conserved, and that this relationship holds separately for different parts of speech. A regression model combining the principal factor loadings derived from the worldwide sample along with their part of speech predicts 46 per cent of the variance in the rates of lexical replacement in the Indo-European languages. This suggests that Indo-European lexical replacement rates might be broadly representative of worldwide rates of change. Evidence for this speculation comes from using the same factor loadings and part-of-speech categories to predict a word's position in a list of 110 words ranked from slowest to most rapidly evolving among 14 of the world's language families. This regression model accounts for 30 per cent of the variance. Our results point to a remarkable regularity in the way that human speakers use language, and hint that the words for a shared set of meanings have been slowly evolving and others more rapidly evolving throughout human history.     

Children with reading disability show brain differences in effective connectivity for visual, but not auditory word comprehension

Background

Previous literature suggests that those with reading disability (RD) have more pronounced deficits during semantic processing in reading as compared to listening comprehension. This discrepancy has been supported by recent neuroimaging studies showing abnormal activity in RD during semantic processing in the visual but not in the auditory modality. Whether effective connectivity between brain regions in RD could also show this pattern of discrepancy has not been investigated.

Methodology/Principal Findings

Children (8- to 14-year-olds) were given a semantic task in the visual and auditory modality that required an association judgment as to whether two sequentially presented words were associated. Effective connectivity was investigated using Dynamic Causal Modeling (DCM) on functional magnetic resonance imaging (fMRI) data. Bayesian Model Selection (BMS) was used separately for each modality to find a winning family of DCM models separately for typically developing (TD) and RD children. BMS yielded the same winning family with modulatory effects on bottom-up connections from the input regions to middle temporal gyrus (MTG) and inferior frontal gyrus(IFG) with inconclusive evidence regarding top-down modulations. Bayesian Model Averaging (BMA) was thus conducted across models in this winning family and compared across groups. The bottom-up effect from the fusiform gyrus (FG) to MTG rather than the top-down effect from IFG to MTG was stronger in TD compared to RD for the visual modality. The stronger bottom-up influence in TD was only evident for related word pairs but not for unrelated pairs. No group differences were noted in the auditory modality.

Conclusions/Significance

This study revealed a modality-specific deficit for children with RD in bottom-up effective connectivity from orthographic to semantic processing regions. There were no group differences in connectivity from frontal regions, suggesting that the core deficit in RD is not in top-down modulation.     

Neural bases of syntax–semantics interface processing

The binding problem—question of how information between the modules of the linguistic system is integrated during language processing—is as yet unresolved. The remarkable speed of language processing and comprehension (Pulvermüller et al. 2009) suggests that at least coarse semantic information (e.g. noun animacy) and syntactically-relevant information (e.g. verbal template) are integrated rapidly to allow for coarse comprehension. This EEG study investigated syntax–semantics interface processing during word-by-word sentence reading. As alpha-band neural activity serves as an inhibition mechanism for local networks, we used topographical distribution of alpha power to help identify the timecourse of the binding process. We manipulated the syntactic parameter of verbal event structure, and semantic parameter of noun animacy in reduced relative clauses (RRCs, e.g. “The witness/mansion seized/protected by the agent was in danger”), to investigate the neural bases of interaction between syntactic and semantic networks during sentence processing. The word-by-word stimulus presentation method in the present experiment required manipulation of both syntactic structure and semantic features in the working memory. The results demonstrated a gradient distribution of early components (biphasic posterior P1–N2 and anterior N1–P2) over function words “by” and “the”, and the verb, corresponding to facilitation or conflict resulting from the syntactic (telicity) and semantic (animacy) cues in the preceding portion of the sentence. This was followed by assimilation of power distribution in the α band at the second noun. The flattened distribution of α power during the mental manipulation with high demand on working memory—thematic role re-assignment—demonstrates a state of α equilibrium with strong functional coupling between posterior and anterior regions. These results demonstrate that the processing of semantic and syntactic features during sentence comprehension proceeds in highly integrated fashion using gating of attentional resources to facilitate rapid comprehension, with attentional suppression of global alpha power to facilitate interaction of local networks.     

The Effects of Gender and Self-Insight on Early Semantic Processing

This event-related potential (ERP) study explored individual differences associated with gender and level of self-insight in early semantic processing. Forty-eight Chinese native speakers completed a semantic judgment task with three different categories of words: abstract neutral words (e.g., logic, effect), concrete neutral words (e.g., teapot, table), and emotion words (e.g., despair, guilt). They then assessed their levels of self-insight. Results showed that women engaged in greater processing than did men. Gender differences also manifested in the relationship between level of self-insight and word processing. For women, level of self-insight was associated with level of semantic activation for emotion words and abstract neutral words, but not for concrete neutral words. For men, level of self-insight was related to processing speed, particularly in response to abstract and concrete neutral words. These findings provide electrophysiological evidence for the effects of gender and self-insight on semantic processing and highlight the need to take into consideration subject variables in related research.     

From Mind to Mouth: Event Related Potentials of Sentence Production in Classic Galactosemia

Patients with classic galactosemia, an inborn error of metabolism, have speech and language production impairments. Past research primarily focused on speech (motor) problems, but these cannot solely explain the language impairments. Which specific deficits contribute to the impairments in language production is not yet known. Deficits in semantic and syntactic planning are plausible and require further investigation. In the present study, we examined syntactic encoding while patients and matched controls overtly described scenes of moving objects using either separate words (minimal syntactic planning) or sentences (sentence-level syntactic planning). The design of the paradigm also allowed tapping into local noun phrase- and more global sentence-level syntactic planning. Simultaneously, we recorded event-related potentials (ERPs). The patients needed more time to prepare and finish the utterances and made more errors. The patient ERPs had a very similar morphology to that of healthy controls, indicating overall comparable neural processing. Most importantly, the ERPs diverged from those of controls in several functionally informative time windows, ranging from very early (90–150 ms post scene onset) to relatively late (1820–2020 ms post scene onset). These time windows can be associated with different linguistic encoding stages. The ERP results form the first neuroscientific evidence for language production impairments in patients with galactosemia in lexical and syntactic planning stages, i.e., prior to the linguistic output phase. These findings hence shed new light on the language impairments in this disease.     

Mechanisms of Reading in Persons with Different Levels of Written Text Comprehension

The aim of the present study was to investigate the reading mechanisms in adults (27 subjects; mean age, 19.5 ± 0.8 [SD] years) with different levels of written text comprehension using fMRI. The main objective was to analyze the basic brain mechanisms of verbal stimuli perception with and without semantic component during reading discrimination tasks. The BOLD signal changes during WORD and PSEUDOWORD reading comparing to GAZE FIXATION state were estimated using both analysis of whole brain activation and ROIs (structures connected with the brain system providing reading) in two groups of subjects, “good” and “poor” readers. It was revealed that activations were higher in “poor” readers in lingual gyrus, SMG, STG compared to “good” readers during PSEUDOWORD reading. It was supposed, that the strategies of words and pseudowords recognition differed in two groups of readers: “good” readers identified words or pseudowords already at the stage of visual analysis of “word” structure and demonstrated attempts to decode pseudowords (i.e., language lexical zones were not activated); “poor” readers, apparently, tried to read pseudowords using the same strategy as for the words reading referring to the lexicon, and after failure identified pseudowords as meaningless concepts. In that case, activations of both lexical “language” zones and visual word form area (VWFA) were observed.     

Lexical–semantic priming effects during infancy

When and how do infants develop a semantic system of words that are related to each other? We investigated word–word associations in early lexical development using an adaptation of the inter-modal preferential looking task where word pairs (as opposed to single target words) were used to direct infants’ attention towards a target picture. Two words (prime and target) were presented in quick succession after which infants were presented with a picture pair (target and distracter). Prime–target word pairs were either semantically and associatively related or unrelated; the targets were either named or unnamed. Experiment 1 demonstrated a lexical–semantic priming effect for 21-month olds but not for 18-month olds: unrelated prime words interfered with linguistic target identification for 21-month olds. Follow-up experiments confirmed the interfering effects of unrelated prime words and identified the existence of repetition priming effects as young as 18 months of age. The results of these experiments indicate that infants have begun to develop semantic–associative links between lexical items as early as 21 months of age.     

Syntactic Priming in American Sign Language

Psycholinguistic studies of sign language processing provide valuable opportunities to assess whether language phenomena, which are primarily studied in spoken language, are fundamentally shaped by peripheral biology. For example, we know that when given a choice between two syntactically permissible ways to express the same proposition, speakers tend to choose structures that were recently used, a phenomenon known as syntactic priming. Here, we report two experiments testing syntactic priming of a noun phrase construction in American Sign Language (ASL). Experiment 1 shows that second language (L2) signers with normal hearing exhibit syntactic priming in ASL and that priming is stronger when the head noun is repeated between prime and target (the lexical boost effect). Experiment 2 shows that syntactic priming is equally strong among deaf native L1 signers, deaf late L1 learners, and hearing L2 signers. Experiment 2 also tested for, but did not find evidence of, phonological or semantic boosts to syntactic priming in ASL. These results show that despite the profound differences between spoken and signed languages in terms of how they are produced and perceived, the psychological representation of sentence structure (as assessed by syntactic priming) operates similarly in sign and speech.     

A neural network model of semantic memory linking feature-based object representation and words

Recent theories in cognitive neuroscience suggest that semantic memory is a distributed process, which involves many cortical areas and is based on a multimodal representation of objects. The aim of this work is to extend a previous model of object representation to realize a semantic memory, in which sensory-motor representations of objects are linked with words. The model assumes that each object is described as a collection of features, coded in different cortical areas via a topological organization. Features in different objects are segmented via γ-band synchronization of neural oscillators. The feature areas are further connected with a lexical area, devoted to the representation of words. Synapses among the feature areas, and among the lexical area and the feature areas are trained via a time-dependent Hebbian rule, during a period in which individual objects are presented together with the corresponding words. Simulation results demonstrate that, during the retrieval phase, the network can deal with the simultaneous presence of objects (from sensory-motor inputs) and words (from acoustic inputs), can correctly associate objects with words and segment objects even in the presence of incomplete information. Moreover, the network can realize some semantic links among words representing objects with shared features. These results support the idea that semantic memory can be described as an integrated process, whose content is retrieved by the co-activation of different multimodal regions. In perspective, extended versions of this model may be used to test conceptual theories, and to provide a quantitative assessment of existing data (for instance concerning patients with neural deficits).     

Building epidemiological models from R0: an implicit treatment of transmission in networks

Simple deterministic models are still at the core of theoretical epidemiology despite the increasing evidence for the importance of contact networks underlying transmission at the individual level. These mean-field or 'compartmental' models based on homogeneous mixing have made, and continue to make, important contributions to the epidemiology and the ecology of infectious diseases but fail to reproduce many of the features observed for disease spread in contact networks. In this work, we show that it is possible to incorporate the important effects of network structure on disease spread with a mean-field model derived from individual level considerations. We propose that the fundamental number known as the basic reproductive number of the disease, R0, which is typically derived as a threshold quantity, be used instead as a central parameter to construct the model from. We show that reliable estimates of individual level parameters can replace a detailed knowledge of network structure, which in general may be difficult to obtain. We illustrate the proposed model with small world networks and the classical example of susceptible-infected-recovered (SIR) epidemics.     

Cooperation prevails when individuals adjust their social ties

Conventional evolutionary game theory predicts that natural selection favours the selfish and strong even though cooperative interactions thrive at all levels of organization in living systems. Recent investigations demonstrated that a limiting factor for the evolution of cooperative interactions is the way in which they are organized, cooperators becoming evolutionarily competitive whenever individuals are constrained to interact with few others along the edges of networks with low average connectivity. Despite this insight, the conundrum of cooperation remains since recent empirical data shows that real networks exhibit typically high average connectivity and associated single-to-broad–scale heterogeneity. Here, a computational model is constructed in which individuals are able to self-organize both their strategy and their social ties throughout evolution, based exclusively on their self-interest. We show that the entangled evolution of individual strategy and network structure constitutes a key mechanism for the sustainability of cooperation in social networks. For a given average connectivity of the population, there is a critical value for the ratio W between the time scales associated with the evolution of strategy and of structure above which cooperators wipe out defectors. Moreover, the emerging social networks exhibit an overall heterogeneity that accounts very well for the diversity of patterns recently found in acquired data on social networks. Finally, heterogeneity is found to become maximal when W reaches its critical value. These results show that simple topological dynamics reflecting the individual capacity for self-organization of social ties can produce realistic networks of high average connectivity with associated single-to-broad–scale heterogeneity. On the other hand, they show that cooperation cannot evolve as a result of “social viscosity” alone in heterogeneous networks with high average connectivity, requiring the additional mechanism of topological co-evolution to ensure the survival of cooperative behaviour.     

The Yin and the Yang of Prediction: An fMRI Study of Semantic Predictive Processing

Probabilistic prediction plays a crucial role in language comprehension. When predictions are fulfilled, the resulting facilitation allows for fast, efficient processing of ambiguous, rapidly-unfolding input; when predictions are not fulfilled, the resulting error signal allows us to adapt to broader statistical changes in this input. We used functional Magnetic Resonance Imaging to examine the neuroanatomical networks engaged in semantic predictive processing and adaptation. We used a relatedness proportion semantic priming paradigm, in which we manipulated the probability of predictions while holding local semantic context constant. Under conditions of higher (versus lower) predictive validity, we replicate previous observations of reduced activity to semantically predictable words in the left anterior superior/middle temporal cortex, reflecting facilitated processing of targets that are consistent with prior semantic predictions. In addition, under conditions of higher (versus lower) predictive validity we observed significant differences in the effects of semantic relatedness within the left inferior frontal gyrus and the posterior portion of the left superior/middle temporal gyrus. We suggest that together these two regions mediated the suppression of unfulfilled semantic predictions and lexico-semantic processing of unrelated targets that were inconsistent with these predictions. Moreover, under conditions of higher (versus lower) predictive validity, a functional connectivity analysis showed that the left inferior frontal and left posterior superior/middle temporal gyrus were more tightly interconnected with one another, as well as with the left anterior cingulate cortex. The left anterior cingulate cortex was, in turn, more tightly connected to superior lateral frontal cortices and subcortical regions—a network that mediates rapid learning and adaptation and that may have played a role in switching to a more predictive mode of processing in response to the statistical structure of the wider environmental context. Together, these findings highlight close links between the networks mediating semantic prediction, executive function and learning, giving new insights into how our brains are able to flexibly adapt to our environment.