How Is Sentence Processing Affected by External Semantic and Syntactic Information? Evidence from Event-Related Potentials

Background

A crucial question for understanding sentence comprehension is the openness of syntactic and semantic processes for other sources of information. Using event-related potentials in a dual task paradigm, we had previously found that sentence processing takes into consideration task relevant sentence-external semantic but not syntactic information. In that study, internal and external information both varied within the same linguistic domain—either semantic or syntactic. Here we investigated whether across-domain sentence-external information would impact within-sentence processing.

Methodology

In one condition, adjectives within visually presented sentences of the structure [Det]-[Noun]-[Adjective]-[Verb] were semantically correct or incorrect. Simultaneously with the noun, auditory adjectives were presented that morphosyntactically matched or mismatched the visual adjectives with respect to gender.

Findings

As expected, semantic violations within the sentence elicited N400 and P600 components in the ERP. However, these components were not modulated by syntactic matching of the sentence-external auditory adjective. In a second condition, syntactic within-sentence correctness-variations were combined with semantic matching variations between the auditory and the visual adjective. Here, syntactic within-sentence violations elicited a LAN and a P600 that did not interact with semantic matching of the auditory adjective. However, semantic mismatching of the latter elicited a frontocentral positivity, presumably related to an increase in discourse level complexity.

Conclusion

The current findings underscore the open versus algorithmic nature of semantic and syntactic processing, respectively, during sentence comprehension.     

Distinguishing Neurocognitive Processes Reflected by P600 Effects: Evidence from ERPs and Neural Oscillations

Research on language comprehension using event-related potentials (ERPs) reported distinct ERP components reliably related to the processing of semantic (N400) and syntactic information (P600). Recent ERP studies have challenged this well-defined distinction by showing P600 effects for semantic and pragmatic anomalies. So far, it is still unresolved whether the P600 reflects specific or rather common processes. The present study addresses this question by investigating ERPs in response to a syntactic and pragmatic (irony) manipulation, as well as a combined syntactic and pragmatic manipulation. For the syntactic condition, a morphosyntactic violation was applied, whereas for the pragmatic condition, such as “That is rich”, either an ironic or literal interpretation was achieved, depending on the prior context. The ERPs at the critical word showed a LAN-P600 pattern for syntactically incorrect sentences relative to correct ones. For ironic compared to literal sentences, ERPs showed a P200 effect followed by a P600 component. In comparison of the syntax-related P600 to the irony-related P600, distributional differences were found. Moreover, for the P600 time window (i.e., 500–900 ms), different changes in theta power between the syntax and pragmatics effects were found, suggesting that different patterns of neural activity contributed to each respective effect. Thus, both late positivities seem to be differently sensitive to these two types of linguistic information, and might reflect distinct neurocognitive processes, such as reanalysis of the sentence structure versus pragmatic reanalysis.     

CNTNAP2 and Language Processing in Healthy Individuals as Measured with ERPs

The genetic FOXP2-CNTNAP2 pathway has been shown to be involved in the language capacity. We investigated whether a common variant of CNTNAP2 (rs7794745) is relevant for syntactic and semantic processing in the general population by using a visual sentence processing paradigm while recording ERPs in 49 healthy adults. While both AA homozygotes and T-carriers showed a standard N400 effect to semantic anomalies, the response to subject-verb agreement violations differed across genotype groups. T-carriers displayed an anterior negativity preceding the P600 effect, whereas for the AA group only a P600 effect was observed. These results provide another piece of evidence that the neuronal architecture of the human faculty of language is shaped differently by effects that are genetically determined.     

How the emotional content of discourse affects language comprehension

Emotion effects on cognition have often been reported. However, only few studies investigated emotional effects on subsequent language processing, and in most cases these effects were induced by non-linguistic stimuli such as films, faces, or pictures. Here, we investigated how a paragraph of positive, negative, or neutral emotional valence affects the processing of a subsequent emotionally neutral sentence, which contained either semantic, syntactic, or no violation, respectively, by means of event-related brain potentials (ERPs). Behavioral data revealed strong effects of emotion; error rates and reaction times increased significantly in sentences preceded by a positive paragraph relative to negative and neutral ones. In ERPs, the N400 to semantic violations was not affected by emotion. In the syntactic experiment, however, clear emotion effects were observed on ERPs. The left anterior negativity (LAN) to syntactic violations, which was not visible in the neutral condition, was present in the negative and positive conditions. This is interpreted as reflecting modulatory effects of prior emotions on syntactic processing, which is discussed in the light of three alternative or complementary explanations based on emotion-induced cognitive styles, working memory, and arousal models. The present effects of emotion on the LAN are especially remarkable considering that syntactic processing has often been regarded as encapsulated and autonomous.     

Age Effects in L2 Grammar Processing as Revealed by ERPs and How (Not) to Study Them

In this study we investigate the effect of age of acquisition (AoA) on grammatical processing in second language learners as measured by event-related brain potentials (ERPs). We compare a traditional analysis involving the calculation of averages across a certain time window of the ERP waveform, analyzed with categorical groups (early vs. late), with a generalized additive modeling analysis, which allows us to take into account the full range of variability in both AoA and time. Sixty-six Slavic advanced learners of German listened to German sentences with correct and incorrect use of non-finite verbs and grammatical gender agreement. We show that the ERP signal depends on the AoA of the learner, as well as on the regularity of the structure under investigation. For gender agreement, a gradual change in processing strategies can be shown that varies by AoA, with younger learners showing a P600 and older learners showing a posterior negativity. For verb agreement, all learners show a P600 effect, irrespective of AoA. Based on their behavioral responses in an offline grammaticality judgment task, we argue that the late learners resort to computationally less efficient processing strategies when confronted with (lexically determined) syntactic constructions different from the L1. In addition, this study highlights the insights the explicit focus on the time course of the ERP signal in our analysis framework can offer compared to the traditional analysis.     

Re-Examination of Chinese Semantic Processing and Syntactic Processing: Evidence from Conventional ERPs and Reconstructed ERPs by Residue Iteration Decomposition (RIDE)

A number of studies have explored the time course of Chinese semantic and syntactic processing. However, whether syntactic processing occurs earlier than semantics during Chinese sentence reading is still under debate. To further explore this issue, an event-related potentials (ERPs) experiment was conducted on 21 native Chinese speakers who read individually-presented Chinese simple sentences (NP1+VP+NP2) word-by-word for comprehension and made semantic plausibility judgments. The transitivity of the verbs was manipulated to form three types of stimuli: congruent sentences (CON), sentences with a semantically violated NP2 following a transitive verb (semantic violation, SEM), and sentences with a semantically violated NP2 following an intransitive verb (combined semantic and syntactic violation, SEM+SYN). The ERPs evoked from the target NP2 were analyzed by using the Residue Iteration Decomposition (RIDE) method to reconstruct the ERP waveform blurred by trial-to-trial variability, as well as by using the conventional ERP method based on stimulus-locked averaging. The conventional ERP analysis showed that, compared with the critical words in CON, those in SEM and SEM+SYN elicited an N400–P600 biphasic pattern. The N400 effects in both violation conditions were of similar size and distribution, but the P600 in SEM+SYN was bigger than that in SEM. Compared with the conventional ERP analysis, RIDE analysis revealed a larger N400 effect and an earlier P600 effect (in the time window of 500–800 ms instead of 570–810ms). Overall, the combination of conventional ERP analysis and the RIDE method for compensating for trial-to-trial variability confirmed the non-significant difference between SEM and SEM+SYN in the earlier N400 time window. Converging with previous findings on other Chinese structures, the current study provides further precise evidence that syntactic processing in Chinese does not occur earlier than semantic processing.     

Focus, newness and their combination: processing of information structure in discourse

The relationship between focus and new information has been unclear despite being the subject of several information structure studies. Here, we report an eye-tracking experiment that explored the relationship between them in on-line discourse processing in Chinese reading. Focus was marked by the Chinese focus-particle "shi", which is equivalent to the cleft structure "it was… who…" in English. New information was defined as the target word that was not present in previous contexts. Our results show that, in the target region, focused information was processed more quickly than non-focused information, while new information was processed more slowly than given information. These results reveal differences in processing patterns between focus and newness, and suggest that they are different concepts that relate to different aspects of cognitive processing. In addition, the effect of new/given information occurred in the post-target region for the focus condition, but not for the non-focus condition, suggesting a complex relationship between focus and newness in the discourse integration stage.     

How the speed of working memory updating influences the on-line thematic processing of simple sentences in Mandarin Chinese

This ERP study used electrophysiological technique to examine how individual differences in the speed of working memory updating influence the use of syntactic and semantic information during on-line sentence argument interpretation, and the time course of that working memory updating effect. The basic structure of the experimental sentences was “Noun + Verb + adverb + ‘le’ + a two-character word”, with the Noun being the sentence initial argument. This initial argument is animate or inanimate and the following verb disambiguates it as an agent or patient. The results at the initial argument revealed that, the quick-updating group elicited a larger positivity over the frontal cortex (within 500–800 ms post-noun onset) as compared with the slow-updating group. At the following disambiguating verb, the slow-updating group only showed a word order effect, indicating that the patient-first condition elicited a larger P600 (within 500–1,000 ms post-verb onset) than the agent-first one; for the quick-updating group, at the early stage of processing, the patient-first sentences elicited a larger N400 (within 300–500 ms post-verb onset) than the agent-first ones only when the initial argument was inanimate; however, at the late stage, the patient-first sentences elicited an enhanced P600 (within 800–1,000 ms post-verb onset) only when the initial argument was animate. These results suggested that the speed of working memory updating not only influences the maintenance of sentence argument when the contents of working memory change but also influences the efficiency of integrating that argument with the verb at a late time point. When integrating the argument with the disambiguating verb, individuals with quick-updating ability can combine multiple sources of information (both noun animacy and word order), and conduct rapid and fine-grained two-stage processing; individuals with slow-updating ability, however, only rely on one dominant source of information types (word order), and conducted slow and course-grained processing.     

Role of the corpus callosum in speech comprehension: interfacing syntax and prosody

The role of the corpus callosum (CC) in the interhemispheric interaction of prosodic and syntactic information during speech comprehension was investigated in patients with lesions in the CC, and in healthy controls. The event-related brain potential experiment examined the effect of prosodic phrase structure on the processing of a verb whose argument structure matched or did not match the prior prosody-induced syntactic structure. While controls showed an N400-like effect for prosodically mismatching verb argument structures, thus indicating a stable interplay between prosody and syntax, patients with lesions in the posterior third of the CC did not show this effect. Because these patients displayed a prosody-independent semantic N400 effect, the present data indicate that the posterior third of the CC is the crucial neuroanatomical structure for the interhemispheric interplay of suprasegmental prosodic information and syntactic information.     

Left inferior frontal activations depending on the canonicity determined by the argument structures of ditransitive sentences: an MEG study

To elucidate the relationships between syntactic and semantic processes, one interesting question is how syntactic structures are constructed by the argument structure of a verb, where each argument corresponds to a semantic role of each noun phrase (NP). Here we examined the effects of possessivity [sentences with or without a possessor] and canonicity [canonical or noncanonical word orders] using Japanese ditransitive sentences. During a syntactic decision task, the syntactic structure of each sentence would be constructed in an incremental manner based on the predicted argument structure of the ditransitive verb in a verb-final construction. Using magnetoencephalography, we found a significant canonicity effect on the current density in the left inferior frontal gyrus (IFG) at 530-550 ms after the verb onset. This effect was selective to canonical sentences, and significant even when the precedent NP was physically identical. We suggest that the predictive effects associated with syntactic processing became larger for canonical sentences, where the NPs and verb were merged with a minimum structural distance, leading to the left IFG activations. For monotransitive and intransitive verbs, in which structural computation of the sentences was simpler than that of ditransitive sentences, we observed a significant effect selective to noncanonical sentences in the temporoparietal regions during 480-670 ms. This effect probably reflects difficulty in semantic processing of noncanonical sentences. These results demonstrate that the left IFG plays a predictive role in syntactic processing, which depends on the canonicity determined by argument structures, whereas other temporoparietal regions would subserve more semantic aspects of sentence processing.     

Negativity Bias in Dangerous Drivers

The behavioral and cognitive characteristics of dangerous drivers differ significantly from those of safe drivers. However, differences in emotional information processing have seldom been investigated. Previous studies have revealed that drivers with higher anger/anxiety trait scores are more likely to be involved in crashes and that individuals with higher anger traits exhibit stronger negativity biases when processing emotions compared with control groups. However, researchers have not explored the relationship between emotional information processing and driving behavior. In this study, we examined the emotional information processing differences between dangerous drivers and safe drivers. Thirty-eight non-professional drivers were divided into two groups according to the penalty points that they had accrued for traffic violations: 15 drivers with 6 or more points were included in the dangerous driver group, and 23 drivers with 3 or fewer points were included in the safe driver group. The emotional Stroop task was used to measure negativity biases, and both behavioral and electroencephalograph data were recorded. The behavioral results revealed stronger negativity biases in the dangerous drivers than in the safe drivers. The bias score was correlated with self-reported dangerous driving behavior. Drivers with strong negativity biases reported having been involved in mores crashes compared with the less-biased drivers. The event-related potentials (ERPs) revealed that the dangerous drivers exhibited reduced P3 components when responding to negative stimuli, suggesting decreased inhibitory control of information that is task-irrelevant but emotionally salient. The influence of negativity bias provides one possible explanation of the effects of individual differences on dangerous driving behavior and traffic crashes.     

The syntactic and semantic processing of mass and count nouns: an ERP study

The present study addressed the question of whether count and mass nouns are differentially processed in the brain. In two different ERP (Event-Related Potentials) tasks we explored the semantic and syntactic levels of such distinction. Mass and count nouns typically differ in concreteness, hence the effect of this important variable was factorially examined in each task. Thus the stimuli presented were: count concrete, count abstract, mass concrete or mass abstract. The first experiment (concrete/abstract semantic judgment task) involved the interaction between the N400 concreteness effect and the Mass/Count condition, revealing a substantial effect between mass and count nouns at the semantic level. The second experiment (sentence syntactic violation task) showed a Mass/Count distinction on left anterior negativity (LAN) and on P600 components, confirming the difference at the syntactic level. This study suggests that the brain differentiates between count and mass nouns not only at the syntactic level but also at the semantic level. Implications for our understanding of the brain mechanisms underlying the Mass/Count distinction are discussed.     

Brain Responses before and after Intensive Second Language Learning: Proficiency Based Changes and First Language Background Effects in Adult Learners

This longitudinal study tracked the neuro-cognitive changes associated with second language (L2) grammar learning in adults in order to investigate how L2 processing is shaped by a learner’s first language (L1) background and L2 proficiency. Previous studies using event-related potentials (ERPs) have argued that late L2 learners cannot elicit a P600 in response to L2 grammatical structures that do not exist in the L1 or that are different in the L1 and L2. We tested whether the neuro-cognitive processes underlying this component become available after intensive L2 instruction. Korean- and Chinese late-L2-learners of English were tested at the beginning and end of a 9-week intensive English-L2 course. ERPs were recorded while participants read English sentences containing violations of regular past tense (a grammatical structure that operates differently in Korean and does not exist in Chinese). Whereas no P600 effects were present at the start of instruction, by the end of instruction, significant P600s were observed for both L1 groups. Latency differences in the P600 exhibited by Chinese and Korean speakers may be attributed to differences in L1–L2 reading strategies. Across all participants, larger P600 effects at session 2 were associated with: 1) higher levels of behavioural performance on an online grammaticality judgment task; and 2) with correct, rather than incorrect, behavioural responses. These findings suggest that the neuro-cognitive processes underlying the P600 (e.g., “grammaticalization”) are modulated by individual levels of L2 behavioural performance and learning.     

Questions Left Unanswered: How the Brain Responds to Missing Information

It sometimes happens that when someone asks a question, the addressee does not give an adequate answer, for instance by leaving out part of the required information. The person who posed the question may wonder why the information was omitted, and engage in extensive processing to find out what the partial answer actually means. The present study looks at the neural correlates of the pragmatic processes invoked by partial answers to questions. Two experiments are presented in which participants read mini-dialogues while their Event-Related brain Potentials (ERPs) are being measured. In both experiments, violating the dependency between questions and answers was found to lead to an increase in the amplitude of the P600 component. We interpret these P600-effects as reflecting the increased effort in creating a coherent representation of what is communicated. This effortful processing might include the computation of what the dialogue participant meant to communicate by withholding information. Our study is one of few investigating language processing in conversation, be it that our participants were ‘eavesdroppers’ instead of real interactants. Our results contribute to the as of yet small range of pragmatic phenomena that modulate the processes underlying the P600 component, and suggest that people immediately attempt to regain cohesion if a question-answer dependency is violated in an ongoing conversation.     

Chinese Tone and Vowel Processing Exhibits Distinctive Temporal Characteristics: An Electrophysiological Perspective from Classical Chinese Poem Processing

Classical Chinese poems have strict regulations on the acoustic pattern of each syllable and are semantically meaningless. Using such poems, this study characterized the temporal order of tone and vowel processing using event-related potentials (ERPs). The target syllable of the poem was either correct or deviated from the correct syllable at tone, vowel or both levels. Vowel violation elicited a negative effect between 300 and 500 ms regardless of the tone correctness, while tone violation elicited a positive effect between 600 and 1000 ms. The results suggest that the vowel information was available earlier than the tone information. Moreover, there was an interaction between the effect of vowel and tone violations between 600 and 1000 ms, showing that the vowel violation produced a positive effect only when the tone was correct. This indicates that vowel and tone processing interacts in the later processing stage, which involves both error detection and reanalysis of the spoken input. Implications of the present results for models of speech perception are discussed.     

Speed of Human Biological Form and Motion Processing

Recent work suggests that biological motion processing can begin within ~110 ms of stimulus onset, as indexed by the P1 component of the event-related potential (ERP). Here, we investigated whether modulation of the P1 component reflects configural processing alone, rather than the processing of both configuration and motion cues. A three-stimulus oddball task was employed to evaluate bottom-up processing of biological motion. Intact point-light walkers (PLWs) or scrambled PLWs served as distractor stimuli, whereas point-light displays of tool motion served as standard and target stimuli. In a second experiment, the same design was used, but the dynamic stimuli were replaced with static point-light displays. The first experiment revealed that dynamic PLWs elicited a larger P1 as compared to scrambled PLWs. A similar P1 increase was also observed for static PLWs in the second experiment, indicating that these stimuli were more salient than static, scrambled PLWs. These findings suggest that the visual system can rapidly extract global form information from static PLWs and that the observed P1 effect for dynamic PLWs is not dependent on the presence of motion cues. Finally, we found that the N1 component was sensitive to dynamic, but not static, PLWs, suggesting that this component reflects the processing of both form and motion information. The sensitivity of P1 to static PLWs has implications for dynamic form models of biological motion processing that posit temporal integration of configural cues present in individual frames of PLW animations.     

Syntactic Priming in American Sign Language

Psycholinguistic studies of sign language processing provide valuable opportunities to assess whether language phenomena, which are primarily studied in spoken language, are fundamentally shaped by peripheral biology. For example, we know that when given a choice between two syntactically permissible ways to express the same proposition, speakers tend to choose structures that were recently used, a phenomenon known as syntactic priming. Here, we report two experiments testing syntactic priming of a noun phrase construction in American Sign Language (ASL). Experiment 1 shows that second language (L2) signers with normal hearing exhibit syntactic priming in ASL and that priming is stronger when the head noun is repeated between prime and target (the lexical boost effect). Experiment 2 shows that syntactic priming is equally strong among deaf native L1 signers, deaf late L1 learners, and hearing L2 signers. Experiment 2 also tested for, but did not find evidence of, phonological or semantic boosts to syntactic priming in ASL. These results show that despite the profound differences between spoken and signed languages in terms of how they are produced and perceived, the psychological representation of sentence structure (as assessed by syntactic priming) operates similarly in sign and speech.     

Depth Structure from Asymmetric Shading Supports Face Discrimination

To examine the effect of illumination direction on the ability of observers to discriminate between faces, we manipulated the direction of illumination on scanned 3D face models. In order to dissociate the surface reflectance and illumination components of front-view face images, we introduce a symmetry algorithm that can separate the symmetric and asymmetric components of the face in both low and high spatial frequency bands. Based on this approach, hybrid faces stimuli were constructed with different combinations of symmetric and asymmetric spatial content. Discrimination results with these images showed that asymmetric illumination information biased face perception toward the structure of the shading component, while the symmetric illumination information had little, if any, effect. Measures of perceived depth showed that this property increased systematically with the asymmetric but not the symmetric low spatial frequency component. Together, these results suggest that (1) the asymmetric 3D shading information dramatically affects both the perceived facial information and the perceived depth of the facial structure; and (2) these effects both increase as the illumination direction is shifted to the side. Thus, our results support the hypothesis that face processing has a strong 3D component.     

Semantics,Syntax or Neither? A Case for Resolution in the Interpretation of N500 and P600 Responses to Harmonic Incongruities

The processing of notes and chords which are harmonically incongruous with their context has been shown to elicit two distinct late ERP effects. These effects strongly resemble two effects associated with the processing of linguistic incongruities: a P600, resembling a typical response to syntactic incongruities in language, and an N500, evocative of the N400, which is typically elicited in response to semantic incongruities in language. Despite the robustness of these two patterns in the musical incongruity literature, no consensus has yet been reached as to the reasons for the existence of two distinct responses to harmonic incongruities. This study was the first to use behavioural and ERP data to test two possible explanations for the existence of these two patterns: the musicianship of listeners, and the resolved or unresolved nature of the harmonic incongruities. Results showed that harmonically incongruous notes and chords elicited a late positivity similar to the P600 when they were embedded within sequences which started and ended in the same key (harmonically resolved). The notes and chords which indicated that there would be no return to the original key (leaving the piece harmonically unresolved) were associated with a further P600 in musicians, but with a negativity resembling the N500 in non-musicians. We suggest that the late positivity reflects the conscious perception of a specific element as being incongruous with its context and the efforts of musicians to integrate the harmonic incongruity into its local context as a result of their analytic listening style, while the late negativity reflects the detection of the absence of resolution in non-musicians as a result of their holistic listening style.     

An ERP Study of the Processing of Common and Decimal Fractions: How Different They Are

This study explored event-related potential (ERP) correlates of common fractions (1/5) and decimal fractions (0.2). Thirteen subjects performed a numerical magnitude matching task under two conditions. In the common fraction condition, a nonsymbolic fraction was asked to be judged whether its magnitude matched the magnitude of a common fraction; in the decimal fraction condition, a nonsymbolic fraction was asked to be matched with a decimal fraction. Behavioral results showed significant main effects of condition and numerical distance, but no significant interaction of condition and numerical distance. Electrophysiological data showed that when nonsymbolic fractions were compared to common fractions, they displayed larger N1 and P3 amplitudes than when they were compared to decimal fractions. This finding suggested that the visual identification for nonsymbolic fractions was different under the two conditions, which was not due to perceptual differences but to task demands. For symbolic fractions, the condition effect was observed in the N1 and P3 components, revealing stimulus-specific visual identification processing. The effect of numerical distance as an index of numerical magnitude representation was observed in the P2, N3 and P3 components under the two conditions. However, the topography of the distance effect was different under the two conditions, suggesting stimulus specific semantic processing of common fractions and decimal fractions.