Marker correspondence, not processing latency, determines temporal binding of visual attributes

BACKGROUND: When simultaneous visual events appear to occur at different times, the discrepancy has generally been ascribed to time differences in neural transmission or cortical processing that lead to asynchronous awareness of the events. RESULTS: We found, however, that an apparent delay of changes in motion direction relative to synchronous color changes occurs only for rapid alternations, and this delay is not accompanied by a difference in reaction time. We also found that perceptual asynchrony depends on the temporal structure of the stimuli (transitions [first-order temporal change] versus turning points [second-order temporal change]) rather than the attribute type (color versus motion). CONCLUSIONS: We propose that the perception of the relative time of events is based on the relationship of representations of temporal pattern that we term time markers. We conclude that the perceptual asynchrony effects studied here do not reflect differential neural delays for different attributes; rather, they arise from a faulty correspondence match between color transitions and position transitions (motion), which in turn results from a difficulty in detecting turning points (direction reversals) and a preference for matching markers of the same type.     

Determinants of asynchronous processing in vision

When a stimulus oscillates in both colour and direction of motion, changes in colour must lag behind those in direction if they are to be seen as concurrent. It has been argued that this lag is the consequence of asynchronous visual processing, with colour being processed more rapidly than motion. This proposal is contentious: it has been criticized on the basis that the time-course of cortical activity may not correlate directly with that of perceptual experience. Here, we demonstrate that the extent of the apparent asynchrony can vary according to the prevailing stimulus conditions. The apparent asynchrony is greatest if the stimulus is composed of opponent directions of motion and is reduced if the angular difference between the directions is reduced. This pattern of results suggests that asynchronous neural activity arises, in part, as a consequence of differential levels of inhibition within relatively independent cortical structures.     

The hierarchy of directional interactions in visual motion processing

It is well known that context influences our perception of visual motion direction. For example, spatial and temporal context manipulations can be used to induce two well-known motion illusions: direction repulsion and the direction after-effect (DAE). Both result in inaccurate perception of direction when a moving pattern is either superimposed on (direction repulsion), or presented following adaptation to (DAE), another pattern moving in a different direction. Remarkable similarities in tuning characteristics suggest that common processes underlie the two illusions. What is not clear, however, is whether the processes driving the two illusions are expressions of the same or different neural substrates. Here we report two experiments demonstrating that direction repulsion and the DAE are, in fact, expressions of different neural substrates. Our strategy was to use each of the illusions to create a distorted perceptual representation upon which the mechanisms generating the other illusion could potentially operate. We found that the processes mediating direction repulsion did indeed access the distorted perceptual representation induced by the DAE. Conversely, the DAE was unaffected by direction repulsion. Thus parallels in perceptual phenomenology do not necessarily imply common neural substrates. Our results also demonstrate that the neural processes driving the DAE occur at an earlier stage of motion processing than those underlying direction repulsion.     

Delayed perceptual awareness in rapid perceptual decisions

The flourishing of studies on the neural correlates of decision-making calls for an appraisal of the relation between perceptual decisions and conscious perception. By exploiting the long integration time of noisy motion stimuli, and by forcing human observers to make difficult speeded decisions--sometimes a blind guess--about stimulus direction, we traced the temporal buildup of motion discrimination capability and perceptual awareness, as assessed trial by trial through direct rating. We found that both increased gradually with motion coherence and viewing time, but discrimination was systematically leading awareness, reaching a plateau much earlier. Sensitivity and criterion changes contributed jointly to the slow buildup of perceptual awareness. It made no difference whether motion discrimination was accomplished by saccades or verbal responses. These findings suggest that perceptual awareness emerges on the top of a developing or even mature perceptual decision. We argue that the middle temporal (MT) cortical region does not confer us the full phenomenic depth of motion perception, although it may represent a precursor stage in building our subjective sense of visual motion.     

Dissociable Perceptual Effects of Visual Adaptation

Neurons in the visual cortex are responsive to the presentation of oriented and curved line segments, which are thought to act as primitives for the visual processing of shapes and objects. Prolonged adaptation to such stimuli gives rise to two related perceptual effects: a slow change in the appearance of the adapting stimulus (perceptual drift), and the distortion of subsequently presented test stimuli (adaptational aftereffects). Here we used a psychophysical nulling technique to dissociate and quantify these two classical observations in order to examine their underlying mechanisms and their relationship to one another. In agreement with previous work, we found that during adaptation horizontal and vertical straight lines serve as attractors for perceived orientation and curvature. However, the rate of perceptual drift for different stimuli was not predictive of the corresponding aftereffect magnitudes, indicating that the two perceptual effects are governed by distinct neural processes. Finally, the rate of perceptual drift for curved line segments did not depend on the spatial scale of the stimulus, suggesting that its mechanisms lie outside strictly retinotopic processing stages. These findings provide new evidence that the visual system relies on statistically salient intrinsic reference stimuli for the processing of visual patterns, and point to perceptual drift as an experimental window for studying the mechanisms of visual perception.     

The dynamics of visual adaptation to faces

Several recent demonstrations using visual adaptation have revealed high-level aftereffects for complex patterns including faces. While traditional aftereffects involve perceptual distortion of simple attributes such as orientation or colour that are processed early in the visual cortical hierarchy, face adaptation affects perceived identity and expression, which are thought to be products of higher-order processing. And, unlike most simple aftereffects, those involving faces are robust to changes in scale, position and orientation between the adapting and test stimuli. These differences raise the question of how closely related face aftereffects are to traditional ones. Little is known about the build-up and decay of the face aftereffect, and the similarity of these dynamic processes to traditional aftereffects might provide insight into this relationship. We examined the effect of varying the duration of both the adapting and test stimuli on the magnitude of perceived distortions in face identity. We found that, just as with traditional aftereffects, the identity aftereffect grew logarithmically stronger as a function of adaptation time and exponentially weaker as a function of test duration. Even the subtle aspects of these dynamics, such as the power-law relationship between the adapting and test durations, closely resembled that of other aftereffects. These results were obtained with two different sets of face stimuli that differed greatly in their low-level properties. We postulate that the mechanisms governing these shared dynamics may be dissociable from the responses of feature-selective neurons in the early visual cortex.     

Emotional Picture and Word Processing: An fMRI Study on Effects of Stimulus Complexity

Neuroscientific investigations regarding aspects of emotional experiences usually focus on one stimulus modality (e.g., pictorial or verbal). Similarities and differences in the processing between the different modalities have rarely been studied directly. The comparison of verbal and pictorial emotional stimuli often reveals a processing advantage of emotional pictures in terms of larger or more pronounced emotion effects evoked by pictorial stimuli. In this study, we examined whether this picture advantage refers to general processing differences or whether it might partly be attributed to differences in visual complexity between pictures and words. We first developed a new stimulus database comprising valence and arousal ratings for more than 200 concrete objects representable in different modalities including different levels of complexity: words, phrases, pictograms, and photographs. Using fMRI we then studied the neural correlates of the processing of these emotional stimuli in a valence judgment task, in which the stimulus material was controlled for differences in emotional arousal. No superiority for the pictorial stimuli was found in terms of emotional information processing with differences between modalities being revealed mainly in perceptual processing regions. While visual complexity might partly account for previously found differences in emotional stimulus processing, the main existing processing differences are probably due to enhanced processing in modality specific perceptual regions. We would suggest that both pictures and words elicit emotional responses with no general superiority for either stimulus modality, while emotional responses to pictures are modulated by perceptual stimulus features, such as picture complexity.     

Expansion of direction space around the cardinal axes revealed by smooth pursuit eye movements

It is well established that perceptual direction discrimination shows an oblique effect; thresholds are higher for motion along diagonal directions than for motion along cardinal directions. Here, we compare simultaneous direction judgments and pursuit responses for the same motion stimuli and find that both pursuit and perceptual thresholds show similar anisotropies. The pursuit oblique effect is robust under a wide range of experimental manipulations, being largely resistant to changes in trajectory (radial versus tangential motion), speed (10 versus 25 deg/s), directional uncertainty (blocked versus randomly interleaved), and cognitive state (tracking alone versus concurrent tracking and perceptual tasks). Our data show that the pursuit oblique effect is caused by an effective expansion of direction space surrounding the cardinal directions and the requisite compression of space for other directions. This expansion suggests that the directions around the cardinal directions are in some way overrepresented in the visual cortical pathways that drive both smooth pursuit and perception.     

Stronger Neural Modulation by Visual Motion Intensity in Autism Spectrum Disorders

Theories of autism spectrum disorders (ASD) have focused on altered perceptual integration of sensory features as a possible core deficit. Yet, there is little understanding of the neuronal processing of elementary sensory features in ASD. For typically developed individuals, we previously established a direct link between frequency-specific neural activity and the intensity of a specific sensory feature: Gamma-band activity in the visual cortex increased approximately linearly with the strength of visual motion. Using magnetoencephalography (MEG), we investigated whether in individuals with ASD neural activity reflect the coherence, and thus intensity, of visual motion in a similar fashion. Thirteen adult participants with ASD and 14 control participants performed a motion direction discrimination task with increasing levels of motion coherence. A polynomial regression analysis revealed that gamma-band power increased significantly stronger with motion coherence in ASD compared to controls, suggesting excessive visual activation with increasing stimulus intensity originating from motion-responsive visual areas V3, V6 and hMT/V5. Enhanced neural responses with increasing stimulus intensity suggest an enhanced response gain in ASD. Response gain is controlled by excitatory-inhibitory interactions, which also drive high-frequency oscillations in the gamma-band. Thus, our data suggest that a disturbed excitatory-inhibitory balance underlies enhanced neural responses to coherent motion in ASD.     

Neural correlates of contrast detection at threshold

Human psychophysical studies have demonstrated that, for stimuli near the threshold of visibility, detection of motion in one direction is unaffected by the superimposition of motion in the opposite direction. To investigate the neural basis for this perceptual phenomenon, we recorded from directionally selective neurons in macaque visual area MT (middle temporal visual area). Contrast thresholds obtained for single gratings moving in a neuron's preferred direction were compared with those obtained for motion presented simultaneously in the neuron's preferred and antipreferred directions. A simple model based on probability summation between neurons tuned to opposite directions could sufficiently account for contrast thresholds revealed psychophysically, suggesting that area MT is likely to provide the neural basis for contrast detection of stimuli modulated in time.     

Unconscious orientation processing

Recent findings have shown that certain attributes of visual stimuli, like orientation, are registered in cortical areas when the stimulus is unresolvable or perceptually invisible; however, there is no evidence to show that complex forms of orientation processing (e.g., modulatory effects of orientation on the processing of other features) could occur in the absence of awareness. To address these questions, different psychophysical paradigms were designed in six experiments to probe unconscious orientation processing. First we demonstrated orientation-selective adaptation and color-contingent orientation adaptation for peripheral unresolvable Gabor patches. The next experiments showed the modulatory effects of perceptually indiscriminable orientations on apparent motion processing and attentional mechanisms. Finally we investigated disappearance patterns of unresolvable Gabor stimuli during motion-induced blindness (MIB). Abrupt changes in local unresolvable orientations truncated MIB; however, orientation-based grouping failed to affect the MIB pattern when the orientations were unresolvable. Overall results revealed that unresolvable orientations substantially influence perception at multiple levels.     

Satiation in cutaneous saltation.

The extent of cutaneous saltation (the illusory displacement of a tap presented to one skin locus by another tap occurring close in time at another locus) was modified by a "preconditioning" stimulus presented prior to and at a site distant from the saltatory test pattern. The 10-sec vibratory preconditioning (PC) stimulus appears to be analogous to inspection figures that "satiate" the perceptual field in experiments on figural aftereffects, producing changes in the perceived size, position, or shape of subsequent stimuli. The direction of displacement of the saltatory phantom was always away from the locus of the prior PC stimulus, consistent with results observed in studies of visual and kinesthetic aftereffects. Th- amount of repulsion and the rate at which the saltatory phantom returned to its initial position depend on the intensity, locus, and number of PC stimuli. As with figural aftereffects, these results resist explanation by peripheral mechanisms such as adaptation.     

Smooth Pursuit Eye Movements Improve Temporal Resolution for Color Perception

Human observers see a single mixed color (yellow) when different colors (red and green) rapidly alternate. Accumulating evidence suggests that the critical temporal frequency beyond which chromatic fusion occurs does not simply reflect the temporal limit of peripheral encoding. However, it remains poorly understood how the central processing controls the fusion frequency. Here we show that the fusion frequency can be elevated by extra-retinal signals during smooth pursuit. This eye movement can keep the image of a moving target in the fovea, but it also introduces a backward retinal sweep of the stationary background pattern. We found that the fusion frequency was higher when retinal color changes were generated by pursuit-induced background motions than when the same retinal color changes were generated by object motions during eye fixation. This temporal improvement cannot be ascribed to a general increase in contrast gain of specific neural mechanisms during pursuit, since the improvement was not observed with a pattern flickering without changing position on the retina or with a pattern moving in the direction opposite to the background motion during pursuit. Our findings indicate that chromatic fusion is controlled by a cortical mechanism that suppresses motion blur. A plausible mechanism is that eye-movement signals change spatiotemporal trajectories along which color signals are integrated so as to reduce chromatic integration at the same locations (i.e., along stationary trajectories) on the retina that normally causes retinal blur during fixation.     

A Laterally Interconnected Neural Architecture in MST Accounts for Psychophysical Discrimination of Complex Motion Patterns

The complex patterns of visual motion formed across the retina during self-motion, often referred to as optic flow, provide a rich source of information describing our dynamic relationship within the environment. Psychophysical studies indicate the existence of specialized detectors for component motion patterns (radial, circular, planar) that are consistent with the visual motion properties of cells in the medial superior temporal area (MST) of nonhuman primates. Here we use computational modeling and psychophysics to investigate the structural and functional role of these specialized detectors in performing a graded motion pattern (GMP) discrimination task. In the psychophysical task perceptual discrimination varied significantly with the type of motion pattern presented, suggesting perceptual correlates to the preferred motion bias reported in MST. Simulated perceptual discrimination in a population of independent MST-like neural responses showed inconsistent psychophysical performance that varied as a function of the visual motion properties within the population code. Robust psychophysical performance was achieved by fully interconnecting neural populations such that they inhibited nonpreferred units. Taken together, these results suggest that robust processing of the complex motion patterns associated with self-motion and optic flow may be mediated by an inhibitory structure of neural interactions in MST.     

Adaptation Aftereffects in Vocal Emotion Perception Elicited by Expressive Faces and Voices

The perception of emotions is often suggested to be multimodal in nature, and bimodal as compared to unimodal (auditory or visual) presentation of emotional stimuli can lead to superior emotion recognition. In previous studies, contrastive aftereffects in emotion perception caused by perceptual adaptation have been shown for faces and for auditory affective vocalization, when adaptors were of the same modality. By contrast, crossmodal aftereffects in the perception of emotional vocalizations have not been demonstrated yet. In three experiments we investigated the influence of emotional voice as well as dynamic facial video adaptors on the perception of emotion-ambiguous voices morphed on an angry-to-happy continuum. Contrastive aftereffects were found for unimodal (voice) adaptation conditions, in that test voices were perceived as happier after adaptation to angry voices, and vice versa. Bimodal (voice + dynamic face) adaptors tended to elicit larger contrastive aftereffects. Importantly, crossmodal (dynamic face) adaptors also elicited substantial aftereffects in male, but not in female participants. Our results (1) support the idea of contrastive processing of emotions (2), show for the first time crossmodal adaptation effects under certain conditions, consistent with the idea that emotion processing is multimodal in nature, and (3) suggest gender differences in the sensory integration of facial and vocal emotional stimuli.     

Neurocognitive processing efficiency for discriminating human non-alarm rather than alarm scream calls

Across many species, scream calls signal the affective significance of events to other agents. Scream calls were often thought to be of generic alarming and fearful nature, to signal potential threats, with instantaneous, involuntary, and accurate recognition by perceivers. However, scream calls are more diverse in their affective signaling nature than being limited to fearfully alarming a threat, and thus the broader sociobiological relevance of various scream types is unclear. Here we used 4 different psychoacoustic, perceptual decision-making, and neuroimaging experiments in humans to demonstrate the existence of at least 6 psychoacoustically distinctive types of scream calls of both alarming and non-alarming nature, rather than there being only screams caused by fear or aggression. Second, based on perceptual and processing sensitivity measures for decision-making during scream recognition, we found that alarm screams (with some exceptions) were overall discriminated the worst, were responded to the slowest, and were associated with a lower perceptual sensitivity for their recognition compared with non-alarm screams. Third, the neural processing of alarm compared with non-alarm screams during an implicit processing task elicited only minimal neural signal and connectivity in perceivers, contrary to the frequent assumption of a threat processing bias of the primate neural system. These findings show that scream calls are more diverse in their signaling and communicative nature in humans than previously assumed, and, in contrast to a commonly observed threat processing bias in perceptual discriminations and neural processes, we found that especially non-alarm screams, and positive screams in particular, seem to have higher efficiency in speeded discriminations and the implicit neural processing of various scream types in humans.

Human screams are more diverse in their communicative nature than those of other species, and are not limited to alarm signals of threat. This study shows that surprisingly, non-alarming screams, and positive screams in particular, have higher efficiency of their cognitive and neural processing than alarm screams.     

The motion aftereffect as a universal phenomenon in sensory systems involved in spatial orientation. III. Aftereffect of motion adaptation in the somatosensory and vestibular systems

The motion aftereffect may be considered as a consequence of visual illusions of self-motion (vection) and the persistence of sensory information processing. There is ample experimental evidence indicating a uniformity of mechanisms that underlie motion aftereffects in different modalities based on the principle of motion detectors. Currently, there is firm ground to believe that the motion aftereffect is intrinsic to all sensory systems involved in spatial orientation, that motion adaptation in one sensory system elicits changes in another one, and that such adaptation is of great adaptive importance for spatial orientation and motion of an organism. This review seeks to substantiate these ideas.     

Stimulus timing-dependent plasticity in high-level vision

Humans are able to efficiently learn and remember complex visual patterns after only a few seconds of exposure [1]. At a cellular level, such learning is thought to involve changes in synaptic efficacy, which have been linked to the precise timing of action potentials relative to synaptic inputs [2-4]. Previous experiments have tapped into the timing of neural spiking events by using repeated asynchronous presentation of visual stimuli to induce changes in both the tuning properties of visual neurons and the perception of simple stimulus attributes [5, 6]. Here we used a similar approach to investigate potential mechanisms underlying the perceptual learning of face identity, a high-level stimulus property based on the spatial configuration of local features. Periods of stimulus pairing induced a systematic bias in face-identity perception in a manner consistent with the predictions of spike timing-dependent plasticity. The perceptual shifts induced for face identity were tolerant to a 2-fold change in stimulus size, suggesting that they reflected neuronal changes in nonretinotopic areas, and were more than twice as strong as the perceptual shifts induced for low-level visual features. These results support the idea that spike timing-dependent plasticity can rapidly adjust the neural encoding of high-level stimulus attributes [7-11].     

Seeing invisible motion: a human FMRI study

A common view about visual consciousness is that it could arise when and where activity reaches some higher level of processing along the cortical hierarchy. Reports showing that activity in striate cortex can be dissociated from awareness , whereas the latter modulates activity in higher areas , point in this direction. In the specific case of visual motion, a central, "perceptual" role has been assigned to area V5: several human and monkey studies have shown V5 activity to correlate with the motion percept. Here we show that activity in this and other higher cortical areas can be also dissociated from perception and follow the physical stimulus instead. The motion information in a peripheral grating modulated fMRI responses, despite being invisible to human volunteers: under crowding conditions , areas V3A, V5, and parietal cortex still showed increased activity when the grating was moving compared to when it was flickering. We conclude that stimulus-specific activation of higher cortical areas does not necessarily result in awareness of the underlying stimulus.     

Auditory adaptation in voice perception

Perceptual aftereffects following adaptation to simple stimulus attributes (e.g., motion, color) have been studied for hundreds of years. A striking recent discovery was that adaptation also elicits contrastive aftereffects in visual perception of complex stimuli and faces [1-6]. Here, we show for the first time that adaptation to nonlinguistic information in voices elicits systematic auditory aftereffects. Prior adaptation to male voices causes a voice to be perceived as more female (and vice versa), and these auditory aftereffects were measurable even minutes after adaptation. By contrast, crossmodal adaptation effects were absent, both when male or female first names and when silently articulating male or female faces were used as adaptors. When sinusoidal tones (with frequencies matched to male and female voice fundamental frequencies) were used as adaptors, no aftereffects on voice perception were observed. This excludes explanations for the voice aftereffect in terms of both pitch adaptation and postperceptual adaptation to gender concepts and suggests that contrastive voice-coding mechanisms may routinely influence voice perception. The role of adaptation in calibrating properties of high-level voice representations indicates that adaptation is not confined to vision but is a ubiquitous mechanism in the perception of nonlinguistic social information from both faces and voices.