Asynchrony adaptation reveals neural population code for audio-visual timing

The relative timing of auditory and visual stimuli is a critical cue for determining whether sensory signals relate to a common source and for making inferences about causality. However, the way in which the brain represents temporal relationships remains poorly understood. Recent studies indicate that our perception of multisensory timing is flexible--adaptation to a regular inter-modal delay alters the point at which subsequent stimuli are judged to be simultaneous. Here, we measure the effect of audio-visual asynchrony adaptation on the perception of a wide range of sub-second temporal relationships. We find distinctive patterns of induced biases that are inconsistent with the previous explanations based on changes in perceptual latency. Instead, our results can be well accounted for by a neural population coding model in which: (i) relative audio-visual timing is represented by the distributed activity across a relatively small number of neurons tuned to different delays; (ii) the algorithm for reading out this population code is efficient, but subject to biases owing to under-sampling; and (iii) the effect of adaptation is to modify neuronal response gain. These results suggest that multisensory timing information is represented by a dedicated population code and that shifts in perceived simultaneity following asynchrony adaptation arise from analogous neural processes to well-known perceptual after-effects.     

Asynchronous processing in vision: color leads motion

It has been demonstrated that subjects do not report changes in color and direction of motion as being co-incidental when they occur synchronously. Instead, for the changes to be reported as being synchronous, changes in direction of motion must precede changes in color. To explain this observation, some researchers have suggested that the neural processing of color and motion is asynchronous. This interpretation has been criticized on the basis that processing time may not correlate directly and invariantly with perceived time of occurrence. Here we examine this possibility by making use of the color-contingent motion aftereffect. By correlating color states disproportionately with two directions of motion, we produced and measured color-contingent motion aftereffects as a function of the range of physical correlations. The aftereffects observed are consistent with the perceptual correlation between color and motion being different from the physical correlation. These findings demonstrate asynchronous processing for different stimulus attributes, with color being processed more quickly than motion. This suggests that the time course of perceptual experience correlates directly with that of neural activity.     

Delayed perceptual awareness in rapid perceptual decisions

The flourishing of studies on the neural correlates of decision-making calls for an appraisal of the relation between perceptual decisions and conscious perception. By exploiting the long integration time of noisy motion stimuli, and by forcing human observers to make difficult speeded decisions--sometimes a blind guess--about stimulus direction, we traced the temporal buildup of motion discrimination capability and perceptual awareness, as assessed trial by trial through direct rating. We found that both increased gradually with motion coherence and viewing time, but discrimination was systematically leading awareness, reaching a plateau much earlier. Sensitivity and criterion changes contributed jointly to the slow buildup of perceptual awareness. It made no difference whether motion discrimination was accomplished by saccades or verbal responses. These findings suggest that perceptual awareness emerges on the top of a developing or even mature perceptual decision. We argue that the middle temporal (MT) cortical region does not confer us the full phenomenic depth of motion perception, although it may represent a precursor stage in building our subjective sense of visual motion.     

Determinants of asynchronous processing in vision

When a stimulus oscillates in both colour and direction of motion, changes in colour must lag behind those in direction if they are to be seen as concurrent. It has been argued that this lag is the consequence of asynchronous visual processing, with colour being processed more rapidly than motion. This proposal is contentious: it has been criticized on the basis that the time-course of cortical activity may not correlate directly with that of perceptual experience. Here, we demonstrate that the extent of the apparent asynchrony can vary according to the prevailing stimulus conditions. The apparent asynchrony is greatest if the stimulus is composed of opponent directions of motion and is reduced if the angular difference between the directions is reduced. This pattern of results suggests that asynchronous neural activity arises, in part, as a consequence of differential levels of inhibition within relatively independent cortical structures.     

The hierarchy of directional interactions in visual motion processing

It is well known that context influences our perception of visual motion direction. For example, spatial and temporal context manipulations can be used to induce two well-known motion illusions: direction repulsion and the direction after-effect (DAE). Both result in inaccurate perception of direction when a moving pattern is either superimposed on (direction repulsion), or presented following adaptation to (DAE), another pattern moving in a different direction. Remarkable similarities in tuning characteristics suggest that common processes underlie the two illusions. What is not clear, however, is whether the processes driving the two illusions are expressions of the same or different neural substrates. Here we report two experiments demonstrating that direction repulsion and the DAE are, in fact, expressions of different neural substrates. Our strategy was to use each of the illusions to create a distorted perceptual representation upon which the mechanisms generating the other illusion could potentially operate. We found that the processes mediating direction repulsion did indeed access the distorted perceptual representation induced by the DAE. Conversely, the DAE was unaffected by direction repulsion. Thus parallels in perceptual phenomenology do not necessarily imply common neural substrates. Our results also demonstrate that the neural processes driving the DAE occur at an earlier stage of motion processing than those underlying direction repulsion.     

The neural structures expressing perceptual hysteresis in visual letter recognition

Perception can change nonlinearly with stimulus contrast, and perceptual threshold may depend on the direction of contrast change. Such hysteresis effects in neurometric functions provide a signature of perceptual awareness. We recorded brain activity with functional neuroimaging in observers exposed to gradual contrast changes of initially hidden visual stimuli. Lateral occipital, frontal, and parietal regions all displayed both transient activations and hysteresis that correlated with change and maintenance of a percept, respectively. Medial temporal activity did not follow perception but increased during hysteresis and showed transient deactivations during perceptual transitions. These findings identify a set of brain regions sensitive to visual awareness and suggest that medial temporal structures may provide backward signals that account for neural and, thereby, perceptual hysteresis.     

Neural correlates of contrast detection at threshold

Human psychophysical studies have demonstrated that, for stimuli near the threshold of visibility, detection of motion in one direction is unaffected by the superimposition of motion in the opposite direction. To investigate the neural basis for this perceptual phenomenon, we recorded from directionally selective neurons in macaque visual area MT (middle temporal visual area). Contrast thresholds obtained for single gratings moving in a neuron's preferred direction were compared with those obtained for motion presented simultaneously in the neuron's preferred and antipreferred directions. A simple model based on probability summation between neurons tuned to opposite directions could sufficiently account for contrast thresholds revealed psychophysically, suggesting that area MT is likely to provide the neural basis for contrast detection of stimuli modulated in time.     

Coordinated Behaviour in Pigeon Flocks

We analysed pigeon flock flights using GPS trajectory data to reveal the most important kinematic aspects of flocking behaviour. We quantitatively investigated the internal motion of the flock based on pairwise statistics and found the following general relationships in all datasets: i) the temporal order of decisions characterised by the delay between directional changes is strictly related to the spatial order characterised by the longitudinal relative position within the flock; ii) during circling motion, pigeons use a mixture of two idealised and fundamentally different turning strategies, namely, parallel-path and equal-radius type turning. While pigeons tend to maintain their relative position within the flock on average, as in the parallel-path approximation, those who turn later also get behind as in the equal-radius case. Equal-radius type turning also tends to be expressed more during smaller radius turns.     

Covert tracking: a combined ERP and fixational eye movement study

Attention can be directed to particular spatial locations, or to objects that appear at anticipated points in time. While most work has focused on spatial or temporal attention in isolation, we investigated covert tracking of smoothly moving objects, which requires continuous coordination of both. We tested two propositions about the neural and cognitive basis of this operation: first that covert tracking is a right hemisphere function, and second that pre-motor components of the oculomotor system are responsible for driving covert spatial attention during tracking. We simultaneously recorded event related potentials (ERPs) and eye position while participants covertly tracked dots that moved leftward or rightward at 12 or 20°/s. ERPs were sensitive to the direction of target motion. Topographic development in the leftward motion was a mirror image of the rightward motion, suggesting that both hemispheres contribute equally to covert tracking. Small shifts in eye position were also lateralized according to the direction of target motion, implying covert activation of the oculomotor system. The data addresses two outstanding questions about the nature of visuospatial tracking. First, covert tracking is reliant upon a symmetrical frontoparietal attentional system, rather than being right lateralized. Second, this same system controls both pursuit eye movements and covert tracking.     

Directional anisotropies reveal a functional segregation of visual motion processing for perception and action

Human exhibits an anisotropy in direction perception: discrimination is superior when motion is around horizontal or vertical rather than diagonal axes. In contrast to the consistent directional anisotropy in perception, we found only small idiosyncratic anisotropies in smooth pursuit eye movements, a motor action requiring accurate discrimination of visual motion direction. Both pursuit and perceptual direction discrimination rely on signals from the middle temporal visual area (MT), yet analysis of multiple measures of MT neuronal responses in the macaque failed to provide evidence of a directional anisotropy. We conclude that MT represents different motion directions uniformly, and subsequent processing creates a directional anisotropy in pathways unique to perception. Our data support the hypothesis that, at least for visual motion, perception and action are guided by inputs from separate sensory streams. The directional anisotropy of perception appears to originate after the two streams have segregated and downstream from area MT.     

Extracting the dynamics of perceptual switching from 'noisy' behaviour: an application of hidden Markov modelling to pecking data from pigeons.

When studying animal perception, one normally has the chance of localizing perceptual events in time, that is via behavioural responses time-locked to the stimuli. With multistable stimuli, however, perceptual changes occur despite stationary stimulation. Here, the challenge is to infer these not directly observable perceptual states indirectly from the behavioural data. This estimation is complicated by the fact that an animal's performance is contaminated by errors. We propose a two-step approach to overcome this difficulty: First, one sets up a generative, stochastic model of the behavioural time series based on the relevant parameters, including the probability of errors. Second, one performs a model-based maximum-likelihood estimation on the data in order to extract the non-observable perceptual state transitions. We illustrate this methodology for data from experiments on perception of bistable apparent motion in pigeons. The observed behavioural time series is analysed and explained by a combination of a Markovian perceptual dynamics with a renewal process that governs the motor response. We propose a hidden Markov model in which non-observable states represent both the perceptual states and the states of the renewal process of the motor dynamics, while the observable states account for overt pecking performance. Showing that this constitutes an appropriate phenomenological model of the time series of observable pecking events, we use it subsequently to obtain an estimate of the internal (and thus covert) perceptual reversals. These may directly correspond to changes in the activity of mutually inhibitory populations of motion selective neurones tuned to orthogonal directions.     

High-frequency High-resolution Echocardiography: First Evidence on Non-invasive Repeated Measure of Myocardial Strain,Contractility, and Mitral Regurgitation in the Ischemia-reperfused Murine Heart

Ischemia-reperfusion (IR) was surgically performed in murine hearts which were then subjected to repeated imaging to monitor temporal changes in functional parameters of key clinical significance. Two-dimensional movies were acquired at high frame rate (8 kHz) and were utilized to estimate high-quality myocardial strain. Two-dimensional elastograms (strain images), as well as strain profiles, were visualized. Results were powerful in quantitatively assessing IR-induced changes in cardiac events including left-ventricular (LV) contraction, LV relaxation and isovolumetric phases of both pre-IR and post-IR beating hearts in intact mice. In addition, compromised sector-wise wall motion and anatomical deformation in the infarcted myocardium were visualized. The elastograms were uniquely able to provide information on the following parameters in addition to standard physiological indices that are known to be affected by myocardial infarction in the mouse: internal diameters of mitral valve orifice and aorta, effective regurgitant orifice, myocardial strain (circumferential as well as radial), turbulence in blood flow pattern as revealed by the color Doppler movies and velocity profiles, asynchrony in LV sector, and changes in the length and direction of vectors demonstrating slower and asymmetrical wall movement. This work emphasizes on the visual demonstration of how such analyses are performed.     

Three theories of stroboscopic motion detection

The three theories derive from three different paradigms. Suprathreshold judgements of perceived quality of motion in multi-flash displays are modelled by space-time Fourier analysis of the motion stimulus. Stroboscopic motion is perceived as being different from real motion to the extent that the additional Fourier components in stroboscopic motion are detectable. Stroboscopic motion of dots along conflicting paths leads to perceptual competition. The theory to describe perceptual I solution derives and proves the uniqueness of strength functions computed only from the time and from the distance between successive points on each path. Time-strength and motion-strength add to determine path-strength; only the strongest path is perceived. Motion-direction detection in continuously drifting two-flash combinations of sinusoidal gratings is described by elaborated Reichardt detectors (ERDs) that compute the covariance of temporal events in two adjacent locations. Other apparently different, detectors that account for direction-detection data are shown to be equivalent to ERDs.     

Spatially localized distortions of event time

A fundamental question about the perception of time is whether the neural mechanisms underlying temporal judgements are universal and centralized in the brain or modality specific and distributed. Time perception has traditionally been thought to be entirely dissociated from spatial vision. Here we show that the apparent duration of a dynamic stimulus can be manipulated in a local region of visual space by adapting to oscillatory motion or flicker. This implicates spatially localized temporal mechanisms in duration perception. We do not see concomitant changes in the time of onset or offset of the test patterns, demonstrating a direct local effect on duration perception rather than an indirect effect on the time course of neural processing. The effects of adaptation on duration perception can also be dissociated from motion or flicker perception per se. Although 20 Hz adaptation reduces both the apparent temporal frequency and duration of a 10 Hz test stimulus, 5 Hz adaptation increases apparent temporal frequency but has little effect on duration perception. We conclude that there is a peripheral, spatially localized, essentially visual component involved in sensing the duration of visual events.     

Expansion of direction space around the cardinal axes revealed by smooth pursuit eye movements

It is well established that perceptual direction discrimination shows an oblique effect; thresholds are higher for motion along diagonal directions than for motion along cardinal directions. Here, we compare simultaneous direction judgments and pursuit responses for the same motion stimuli and find that both pursuit and perceptual thresholds show similar anisotropies. The pursuit oblique effect is robust under a wide range of experimental manipulations, being largely resistant to changes in trajectory (radial versus tangential motion), speed (10 versus 25 deg/s), directional uncertainty (blocked versus randomly interleaved), and cognitive state (tracking alone versus concurrent tracking and perceptual tasks). Our data show that the pursuit oblique effect is caused by an effective expansion of direction space surrounding the cardinal directions and the requisite compression of space for other directions. This expansion suggests that the directions around the cardinal directions are in some way overrepresented in the visual cortical pathways that drive both smooth pursuit and perception.     

Seeing objects in motion

This paper reports estimates of the conjoint spatiotemporal tuning functions of the neural mechanisms of the human vision system which detect image motion. The functions were derived from measurements of the minimum contrast necessary to detect the direction of drift of a sinusoidal grating, in the presence of phase-reversed masking gratings of various spatial and temporal frequencies. A mask of similar spatial and temporal frequencies to the test grating reduces sensitivity considerably, whereas one differing greatly in spatial or temporal frequency has little or no effect. The results show that for test gratings drifting at 8 Hz, the tuning function is bandpass in both space and time, peaked at the temporal and spatial frequency (SF) of the test (SFs were 0.1, 1 or 5 c deg-1; c represents cycles throughout). For a grating of 5 c deg-1 drifting at 0.3 Hz, the function is bandpass in space but lowpass in time. Fourier transform of the frequency results yields a function in space-time which we term the 'spatiotemporal receptive field'. For movement detectors (bandpass in space and time) the fields comprise alternating ridges of opposing polarity, elongated in space-time along the preferred velocity axis of the detector. We suggest that this organization explains how detectors analyse form and motion concurrently and accounts, at least in part, for a variety of perceptual phenomena, including summation, reduction of motion smear, metacontrast, stroboscopic motion and spatiotemporal interpolation.     

Perceptual rivalry: reflexes reveal the gradual nature of visual awareness

Rivalry is a common tool to probe visual awareness: a constant physical stimulus evokes multiple, distinct perceptual interpretations ("percepts") that alternate over time. Percepts are typically described as mutually exclusive, suggesting that a discrete (all-or-none) process underlies changes in visual awareness. Here we follow two strategies to address whether rivalry is an all-or-none process: first, we introduce two reflexes as objective measures of rivalry, pupil dilation and optokinetic nystagmus (OKN); second, we use a continuous input device (analog joystick) to allow observers a gradual subjective report. We find that the "reflexes" reflect the percept rather than the physical stimulus. Both reflexes show a gradual dependence on the time relative to perceptual transitions. Similarly, observers' joystick deflections, which are highly correlated with the reflex measures, indicate gradual transitions. Physically simulating wave-like transitions between percepts suggest piece-meal rivalry (i.e., different regions of space belonging to distinct percepts) as one possible explanation for the gradual transitions. Furthermore, the reflexes show that dominance durations depend on whether or not the percept is actively reported. In addition, reflexes respond to transitions with shorter latencies than the subjective report and show an abundance of short dominance durations. This failure to report fast changes in dominance may result from limited access of introspection to rivalry dynamics. In sum, reflexes reveal that rivalry is a gradual process, rivalry's dynamics is modulated by the required action (response mode), and that rapid transitions in perceptual dominance can slip away from awareness.     

Temporal integration of movement: the time-course of motion streaks revealed by masking

Temporal integration in the visual system causes fast-moving objects to leave oriented 'motion streaks' in their wake, which could be used to facilitate motion direction perception. Temporal integration is thought to occur over ≈100 ms in early cortex, although this has never been tested for motion streaks. Here we compare the ability of fast-moving ('streaky') and slow-moving fields of dots to mask briefly flashed gratings either parallel or orthogonal to the motion trajectory. Gratings were presented at various asynchronies relative to motion onset (from -200 to +700 ms) to sample the time-course of the accumulating streaks. Predictions were that masking would be strongest for the fast parallel condition, and would be weak at early asynchronies and strengthen over time as integration rendered the translating dots more streaky and grating-like. The asynchrony where the masking function reached a plateau would correspond to the temporal integration period. As expected, fast-moving dots caused greater masking of parallel gratings than orthogonal gratings, and slow motion produced only modest masking of either grating orientation. Masking strength in the fast, parallel condition increased with time and reached a plateau after 77 ms, providing an estimate of the temporal integration period for mechanisms encoding motion streaks. Interestingly, the greater masking by fast motion of parallel compared with orthogonal gratings first reached significance at 48 ms before motion onset, indicating an effect of backward masking by motion streaks.     

Tracking Visual Events in Time in the Absence of Time Perception: Implicit Processing at the ms Level

Previous studies have suggested that even if subjects deem two visual stimuli less than 20 ms apart to be simultaneous, implicitly they are nonetheless distinguished in time. It is unclear, however, how information is encoded within this short timescale. We used a priming paradigm to demonstrate how successive visual stimuli are processed over time intervals of less than 20 ms. The primers were two empty square frames displayed either simultaneously or with a 17ms asynchrony. The primers were followed by the target information after a delay of 25 ms to 100 ms. The two square frames were filled in one after another with a delay of 100 ms between them, and subjects had to decide on the location of the first of the frames to be filled in. In a second version of the paradigm, only one square frame was filled in, and subjects had to decide where it was positioned. The influence of the primers is revealed through faster response times depending on the location of the first and second primers. Experiment 1 replicates earlier results, with a bias towards the side of the second primer, but only when there is a delay of 75 to 100 ms between primers and targets. The following experiments suggest this effect to be relatively independent of the task context, except for a slight effect on the time course of the biases. For the temporal order judgment task, identical results were observed when subjects have to answer to the side of the second rather than the first target, showing the effect to be independent of the hand response, and suggesting it might be related to a displacement of attention. All in all the results suggest the flow of events is followed more efficiently than suggested by explicit asynchrony judgment studies. We discuss the possible impact of these results on our understanding of the sense of time continuity.     

High temporal frequency synchrony is insufficient for perceptual grouping

We used textures of randomly moving grating patches to assess the role of fine-grain temporal synchrony in texture segregation. In the target area, patches reversed direction simultaneously. In the surround, patches changed direction at random times. Thus, phase changes in the target area were precisely synchronous, whereas those in the surround were not. In agreement with work carried out by Lee and Blake, we found that the target area was frequently visible, and that observers could discriminate its shape (horizontal versus vertical) at frame rates of 100 Hz in brief exposures (200 ms). Further experiments suggested that the length of unidirectional motion sequences in the target area, rather than synchrony, determined its visibility. To eliminate completely contrast and motion cues, we made all the background elements identical to the target elements, but with a random starting phase. Despite the presence of synchrony in the target area but not the background, the target was generally very hard to see. Targets that remained visible contained low temporal frequency modulations of direction. We conclude that the human observer can detect synchrony, but only at modest temporal frequencies once motion and contrast artefacts have been eliminated.