Components of Attention in Grapheme-Color Synesthesia: A Modeling Approach

Grapheme-color synesthesia is a condition where the perception of graphemes consistently and automatically evokes an experience of non-physical color. Many have studied how synesthesia affects the processing of achromatic graphemes, but less is known about the synesthetic processing of physically colored graphemes. Here, we investigated how the visual processing of colored letters is affected by the congruence or incongruence of synesthetic grapheme-color associations. We briefly presented graphemes (10–150 ms) to 9 grapheme-color synesthetes and to 9 control observers. Their task was to report as many letters (targets) as possible, while ignoring digit (distractors). Graphemes were either congruently or incongruently colored with the synesthetes’ reported grapheme-color association. A mathematical model, based on Bundesen’s (1990) Theory of Visual Attention (TVA), was fitted to each observer’s data, allowing us to estimate discrete components of visual attention. The models suggested that the synesthetes processed congruent letters faster than incongruent ones, and that they were able to retain more congruent letters in visual short-term memory, while the control group’s model parameters were not significantly affected by congruence. The increase in processing speed, when synesthetes process congruent letters, suggests that synesthesia affects the processing of letters at a perceptual level. To account for the benefit in processing speed, we propose that synesthetic associations become integrated into the categories of graphemes, and that letter colors are considered as evidence for making certain perceptual categorizations in the visual system. We also propose that enhanced visual short-term memory capacity for congruently colored graphemes can be explained by the synesthetes’ expertise regarding their specific grapheme-color associations.     

Training Synesthetic Letter-color Associations by Reading in Color

Synesthesia is a rare condition in which a stimulus from one modality automatically and consistently triggers unusual sensations in the same and/or other modalities. A relatively common and well-studied type is grapheme-color synesthesia, defined as the consistent experience of color when viewing, hearing and thinking about letters, words and numbers. We describe our method for investigating to what extent synesthetic associations between letters and colors can be learned by reading in color in nonsynesthetes. Reading in color is a special method for training associations in the sense that the associations are learned implicitly while the reader reads text as he or she normally would and it does not require explicit computer-directed training methods. In this protocol, participants are given specially prepared books to read in which four high-frequency letters are paired with four high-frequency colors. Participants receive unique sets of letter-color pairs based on their pre-existing preferences for colored letters. A modified Stroop task is administered before and after reading in order to test for learned letter-color associations and changes in brain activation. In addition to objective testing, a reading experience questionnaire is administered that is designed to probe for differences in subjective experience. A subset of questions may predict how well an individual learned the associations from reading in color. Importantly, we are not claiming that this method will cause each individual to develop grapheme-color synesthesia, only that it is possible for certain individuals to form letter-color associations by reading in color and these associations are similar in some aspects to those seen in developmental grapheme-color synesthetes. The method is quite flexible and can be used to investigate different aspects and outcomes of training synesthetic associations, including learning-induced changes in brain function and structure.     

Neurocognitive mechanisms of synesthesia

Synesthesia is a condition in which stimulation of one sensory modality causes unusual experiences in a second, unstimulated modality. Although long treated as a curiosity, recent research with a combination of phenomenological, behavioral, and neuroimaging methods has begun to identify the cognitive and neural basis of synesthesia. Here, we review this literature with an emphasis on grapheme-color synesthesia, in which viewing letters and numbers induces the perception of colors. We discuss both the substantial progress that has been made in the past fifteen years and some open questions. In particular, we focus on debates in the field relating to the neural basis of synesthesia, including the relationship between synesthesia and attention and the role of meaning in synesthetic colors. We propose that some, but probably not all, of these differences can be accounted for by differences in the synesthetes studied and discuss some methodological implications of these individual differences.     

Individual differences among grapheme-color synesthetes: brain-behavior correlations

Grapheme-color synesthetes experience specific colors associated with specific number or letter characters. To determine the neural locus of this condition, we compared behavioral and fMRI responses in six grapheme-color synesthetes to control subjects. In our behavioral experiments, we found that a subject's synesthetic experience can aid in texture segregation (experiment 1) and reduce the effects of crowding (experiment 2). For synesthetes, graphemes produced larger fMRI responses in color-selective area human V4 than for control subjects (experiment 3). Importantly, we found a correlation within subjects between the behavioral and fMRI results; subjects with better performance on the behavioral experiments showed larger fMRI responses in early retinotopic visual areas (V1, V2, V3, and hV4). These results suggest that grapheme-color synesthesia is the result of cross-activation between grapheme-selective and color-selective brain areas. The correlation between the behavioral and fMRI results suggests that grapheme-color synesthetes may constitute a heterogeneous group.     

Enhanced cortical excitability in grapheme-color synesthesia and its modulation

Synesthesia is an unusual condition characterized by the over-binding of two or more features and the concomitant automatic and conscious experience of atypical, ancillary images or perceptions. Previous research suggests that synesthetes display enhanced modality-specific perceptual processing, but it remains unclear whether enhanced processing contributes to conscious awareness of color photisms. In three experiments, we investigated whether grapheme-color synesthesia is characterized by enhanced cortical excitability in primary visual cortex and the role played by this hyperexcitability in the expression of synesthesia. Using transcranial magnetic stimulation, we show that synesthetes display 3-fold lower phosphene thresholds than controls during stimulation of the primary visual cortex. We next used transcranial direct current stimulation to discriminate between two competing hypotheses of the role of hyperexcitability in the expression of synesthesia. We demonstrate that synesthesia can be selectively augmented with cathodal stimulation and attenuated with anodal stimulation of primary visual cortex. A control task revealed that the effect of the brain stimulation was specific to the experience of synesthesia. These results indicate that hyperexcitability acts as a source of noise in visual cortex that influences the availability of the neuronal signals underlying conscious awareness of synesthetic photisms.     

Audiovisual integration of letters in the human brain

Letters of the alphabet have auditory (phonemic) and visual (graphemic) qualities. To investigate the neural representations of such audiovisual objects, we recorded neuromagnetic cortical responses to auditorily, visually, and audiovisually presented single letters. The auditory and visual brain activations first converged around 225 ms after stimulus onset and then interacted predominantly in the right temporo-occipito-parietal junction (280345 ms) and the left (380-540 ms) and right (450-535 ms) superior temporal sulci. These multisensory brain areas, playing a role in audiovisual integration of phonemes and graphemes, participate in the neural network supporting the supramodal concept of a "letter." The dynamics of these functions bring new insight into the interplay between sensory and association cortices during object recognition.     

Hemispheric differences in visual recognition of two Kanji characters

The purpose of the present study was to examine hemispheric differences in visual recognition of two kanji letters. Kanji phrases and nonsense kanji letters were presented unilaterally to the left or right visual hemifield in thirty normal right-handed men by a tachistoscope. The experiments were separated into three conditions; the mixing tasks (ten phrases and ten sets of nonsense letters are presented by mixture), the phase task (twenty phrases are presented), and the nonsense letters task (twenty sets of nonsense letters are presented). Under all conditions, significant right visual hemifield superiorities for the accuracy of recognition of phrase and nonsense letters were observed.     

Movement in the normal visual hemifield induces a percept in the 'blind' hemifield of a human hemianope.

We have investigated visual responses to moving stimuli presented to the normal hemifield of a hemianope, GY, who exhibits residual visual function in his right, ''blind'' hemifield. Preliminary experiments established that his perception of moving stimuli localized in his ''blind'' hemifield is retained when a similar stimulus is presented simultaneously in the normal hemifield. In response to a grating stimulus moving horizontally towards fixation in the non-foveal region of the normal, left hemifield, he perceives in addition to a normal motion percept in the left hemifield, a sensation of movement localized in the right hemifield. Qualitatively, this latter is indistinguishable from responses elicited by direct stimulation localized within his ''blind'' hemifield by moving stimuli. We have investigated the characteristics of the mechanisms which induce the ''blind'' field component of GY''s responses to stimulation of the normal hemifield. We show that GY''s sensitivity for detection of movement localized within his ''blind'' hemifield is dependent on the direction of movement, the contrast and the velocity of a grating presented to the normal hemifield. No induced effects were recorded in response to colour or to non-moving, flickering stimuli. We examine the possible contribution of scattered light to our observations, and eliminate this factor by consideration of our experimental results. We discuss the neural mechanisms which may be involved in this response.     

Prevalence of Learned Grapheme-Color Pairings in a Large Online Sample of Synesthetes

In this paper we estimate the minimum prevalence of grapheme-color synesthetes with letter-color matches learned from an external stimulus, by analyzing a large sample of English-speaking grapheme-color synesthetes. We find that at least 6% (400/6588 participants) of the total sample learned many of their matches from a widely available colored letter toy. Among those born in the decade after the toy began to be manufactured, the proportion of synesthetes with learned letter-color pairings approaches 15% for some 5-year periods. Among those born 5 years or more before it was manufactured, none have colors learned from the toy. Analysis of the letter-color matching data suggests the only difference between synesthetes with matches to the toy and those without is exposure to the stimulus. These data indicate learning of letter-color pairings from external contingencies can occur in a substantial fraction of synesthetes, and are consistent with the hypothesis that grapheme-color synesthesia is a kind of conditioned mental imagery.     

Differences in latent inhibition between men and women under conditions of nonverbal visually-spaatial masking]

In order to study gender differences in development of the latent inhibition (LI) and the influence of laterality (hemisphere) factor, two groups of subjects were examined. In the preexposure phase, subjects of the control group were presented with a pair of geometric figures, which appeared either in the right or in the left visual hemifield (at random) with the exposure time of 80 ms. A subject had to compare the figures in a pair and to press a button in case of their identity. In the preexposure (experimental) group, the subject solved the same task, but half of them was presented with the same set of stimuli which appeared in the right hemifield, another half of subjects received these stimuli in the left hemifield. Simultaneously with the target stimuli, the opposite hemifield was stimulated with a nonattended symbol "[symbol: see text]". In the test phase, all the subjects (both of the control and experimental groups) had to respond to the symbol "[symbol: see text]" (previously nonattended in the experimental group), which appeared in part of the trials, by pressing the button. Pairs of stimuli which had been target ones in the preexposure phase, became masking stimuli in the test phase. Correct responses were reinforced by presentation of a special sound. The number of trials required for acquisition of the instrumental contingency served as an index of the LI. Analysis of variance showed that significant strong effect of the LI was observed only in men (comparison between the control and preexposure groups), while there was no significant difference between women of the control and preexposure groups. There was no laterality effect. Gender differences and hemispheric asymmetry in the LI organization are discussed.     

Mechanisms and asymmetries in visual perception of simultaneity and temporal order

Temporally overlapping, spatially separated visual stimuli were used for studying perception of simultaneity and temporal order. Pairs of flashes each of 100 ms duration were presented with stimulus onset asynchronies of 0, 30, 50, and 70 ms. Three spatial arrangements of flash presentation were tested: 1) both flashes were presented foveally; 2) one flash was presented foveally and the other at 9 deg in the left visual hemifield; 3) one flash was presented foveally and the other at 8 deg in the right visual hemifield. Onset asynchronies of 30 and 50 ms were not sufficient for correct identification of temporal order although the flashes were not perceived as simultaneous. Analysis of the response distributions suggests the existence of two-independent mechanisms for evaluating temporal interrelations: one for detecting simultaneity and the other for identifying temporal order. A better detection of simultaneity was found when synchroneous flashes were presented together with pairs of flashes separated by larger onset asynchronies. Reading habits may explain only part of the left-right asymmetries of the response distributions. The possible lateralization of the two suggested mechanisms within the cerebral hemispheres is discussed.     

The importance of individual differences in grapheme-color synesthesia

In this issue of Neuron, Hubbard et al. show individual differences in how grapheme-color synesthetes perform on cognitive tasks. Importantly, these behavioral differences were correlated with fMRI measures. Such individual differences have important ramifications for synesthesia research. If individual differences are ignored, then synesthesia research will be characterized by erroneous conclusions and failures to replicate.     

The encoding of temporally irregular and regular visual patterns in the human brain

In the work reported here, we set out to study the neural systems that detect predictable temporal patterns and departures from them. We used functional magnetic resonance imaging (fMRI) to locate activity in the brains of subjects when they viewed temporally regular and irregular patterns produced by letters, numbers, colors and luminance. Activity induced by irregular sequences was located within dorsolateral prefrontal cortex, including an area that was responsive to irregular patterns regardless of the type of visual stimuli producing them. Conversely, temporally regular arrangements resulted in activity in the right frontal lobe (medial frontal gyrus), in the left orbito-frontal cortex and in the left pallidum. The results show that there is an abstractive system in the brain for detecting temporal irregularity, regardless of the source producing it.     

Relativistic compression and expansion of experiential time in the left and right space

Time, space and numbers are closely linked in the physical world. However, the relativistic-like effects on time perception of spatial and magnitude factors remain poorly investigated. Here we wanted to investigate whether duration judgments of digit visual stimuli are biased depending on the side of space where the stimuli are presented and on the magnitude of the stimulus itself. Different groups of healthy subjects performed duration judgment tasks on various types of visual stimuli. In the first two experiments visual stimuli were constituted by digit pairs (1 and 9), presented in the centre of the screen or in the right and left space. In a third experiment visual stimuli were constituted by black circles. The duration of the reference stimulus was fixed at 300 ms. Subjects had to indicate the relative duration of the test stimulus compared with the reference one. The main results showed that, regardless of digit magnitude, duration of stimuli presented in the left hemispace is underestimated and that of stimuli presented in the right hemispace is overestimated. On the other hand, in midline position, duration judgments are affected by the numerical magnitude of the presented stimulus, with time underestimation of stimuli of low magnitude and time overestimation of stimuli of high magnitude. These results argue for the presence of strict interactions between space, time and magnitude representation on the human brain.     

‘When Birds of a Feather Flock Together’: Synesthetic Correspondences Modulate Audiovisual Integration in Non-Synesthetes

Background

Synesthesia is a condition in which the stimulation of one sense elicits an additional experience, often in a different (i.e., unstimulated) sense. Although only a small proportion of the population is synesthetic, there is growing evidence to suggest that neurocognitively-normal individuals also experience some form of synesthetic association between the stimuli presented to different sensory modalities (i.e., between auditory pitch and visual size, where lower frequency tones are associated with large objects and higher frequency tones with small objects). While previous research has highlighted crossmodal interactions between synesthetically corresponding dimensions, the possible role of synesthetic associations in multisensory integration has not been considered previously.

Methodology

Here we investigate the effects of synesthetic associations by presenting pairs of asynchronous or spatially discrepant visual and auditory stimuli that were either synesthetically matched or mismatched. In a series of three psychophysical experiments, participants reported the relative temporal order of presentation or the relative spatial locations of the two stimuli.

Principal Findings

The reliability of non-synesthetic participants'' estimates of both audiovisual temporal asynchrony and spatial discrepancy were lower for pairs of synesthetically matched as compared to synesthetically mismatched audiovisual stimuli.

Conclusions

Recent studies of multisensory integration have shown that the reduced reliability of perceptual estimates regarding intersensory conflicts constitutes the marker of a stronger coupling between the unisensory signals. Our results therefore indicate a stronger coupling of synesthetically matched vs. mismatched stimuli and provide the first psychophysical evidence that synesthetic congruency can promote multisensory integration. Synesthetic crossmodal correspondences therefore appear to play a crucial (if unacknowledged) role in the multisensory integration of auditory and visual information.     

Recognition of visual images by separate hemispheres in conditions of masked blocking of interhemispheric transmission

The subjects learned to recognize three figures presented in the left visual hemifield and three figures presented in the right visual hemifield. During presentation of a stimulus, the contralateral hemifield was overlapped by a mask. After the training, recognition of all six figures presented in the right and left visual hemifields, was compared. Each hemisphere recognizes figures which were learned in the corresponding visual hemifield, but the recognition of figures learned in the opposite visual hemifield was poor. Thus, the ability of the hemispheres to act separately in recognizing different sets of visual images, was established.     

Visual hemifield differences in recognition of kanji and hiragana and its relation to hemispheric cerebral asymmetries

Visual hemifield differences in recognition of kanji and hiragana were studied on forty male right handers. A letter of kanji or hiragana was presented unilaterally to the right or left visual hemifield on a CRT display for 123 msec. A hundred and twenty recognition trials were performed for each subject using 20 well-acquainted kanji, 20 unfamiliar kanji and 20 hiragana. Kanji was more accurately recognized in the left visual hemifield than in the right hemifield. This tendency was more prominent in unfamiliar kanji compared with well-acquainted kanji. There were no visual hemifield differences in recognition of hiragana. Learning effects were observed for the right hemifield on kanji and both hemifields on hiragana. The results were discussed in relation to cerebral asymmetries of function. Kanji might be processed in the right cerebral hemisphere as geometric forms. The results on hiragana may be explained by mental set. It is suggested that modes of processing may be different between kanji and hiragana.     

Electromagnetic field of mobile phones affects visual event related potential in patients with narcolepsy

The effects of the mobile phone (MP) electromagnetic fields on electroencephalography (EEG) and event-related potentials (ERP) were examined. With regard to the reported effects of MP on sleep, 22 patients with narcolepsy-cataplexy were exposed or sham exposed for 45 min to the MP (900 MHz, specific absorption rate 0.06 W/kg) placed close to the right ear in a double blind study. There were no changes of the EEG recorded after the MP exposure. A subgroup of 17 patients was studied on visual ERP recorded during the MP exposure. Using an adapted "odd-ball" paradigm, each patient was instructed to strike a key whenever rare target stimuli were presented. There were three variants of target stimuli (horizontal stripes in (i) left, (ii) right hemifields or (iii) whole field of the screen). The exposure enhanced the positivity of the ERP endogenous complex solely in response to target stimuli in the right hemifield of the screen (P < 0.01). The reaction time was shortened by 20 ms in response to all target stimuli (P < 0.05). In conclusion, the electromagnetic field of MP may suppress the excessive sleepiness and improve performance while solving a monotonous cognitive task requiring sustained attention and vigilance.     

Enriching Tortoises: Assessing Color Preference

Environmental enrichment is a principle that is used to enhance the quality of care for nonhuman animals in captivity. To achieve this, it is necessary to understand the animal's needs. This study focused on color preference to provide food stimuli as a source of environmental enrichment for the tortoise, Chelonoidis denticulata. During this study, the stimuli green-, blue-, yellow-, and red-colored bananas and plaster blocks were randomly offered to the tortoises. Analysis of the data showed that the tortoises had a preference for the stimuli dyed with colors red and yellow over the other presented colors. It was possible to conclude that presenting food in different colors stimulated the animals to evaluate their environment and make choices in relation to their color preference. Thus, this experiment introduced an element of choice into their lives, beyond identifying color food preferences for the tortoises. The element of choice is known to be important to animal welfare.     

Sensory competition in the face processing areas of the human brain

The concurrent presentation of multiple stimuli in the visual field may trigger mutually suppressive interactions throughout the ventral visual stream. While several studies have been performed on sensory competition effects among non-face stimuli relatively little is known about the interactions in the human brain for multiple face stimuli. In the present study we analyzed the neuronal basis of sensory competition in an event-related functional magnetic resonance imaging (fMRI) study using multiple face stimuli. We varied the ratio of faces and phase-noise images within a composite display with a constant number of peripheral stimuli, thereby manipulating the competitive interactions between faces. For contralaterally presented stimuli we observed strong competition effects in the fusiform face area (FFA) bilaterally and in the right lateral occipital area (LOC), but not in the occipital face area (OFA), suggesting their different roles in sensory competition. When we increased the spatial distance among pairs of faces the magnitude of suppressive interactions was reduced in the FFA. Surprisingly, the magnitude of competition depended on the visual hemifield of the stimuli: ipsilateral stimulation reduced the competition effects somewhat in the right LOC while it increased them in the left LOC. This suggests a left hemifield dominance of sensory competition. Our results support the sensory competition theory in the processing of multiple faces and suggests that sensory competition occurs in several cortical areas in both cerebral hemispheres.