Bayesian calibration of simultaneity in audiovisual temporal order judgments

After repeated exposures to two successive audiovisual stimuli presented in one frequent order, participants eventually perceive a pair separated by some lag time in the same order as occurring simultaneously (lag adaptation). In contrast, we previously found that perceptual changes occurred in the opposite direction in response to tactile stimuli, conforming to bayesian integration theory (bayesian calibration). We further showed, in theory, that the effect of bayesian calibration cannot be observed when the lag adaptation was fully operational. This led to the hypothesis that bayesian calibration affects judgments regarding the order of audiovisual stimuli, but that this effect is concealed behind the lag adaptation mechanism. In the present study, we showed that lag adaptation is pitch-insensitive using two sounds at 1046 and 1480 Hz. This enabled us to cancel lag adaptation by associating one pitch with sound-first stimuli and the other with light-first stimuli. When we presented each type of stimulus (high- or low-tone) in a different block, the point of simultaneity shifted to "sound-first" for the pitch associated with sound-first stimuli, and to "light-first" for the pitch associated with light-first stimuli. These results are consistent with lag adaptation. In contrast, when we delivered each type of stimulus in a randomized order, the point of simultaneity shifted to "light-first" for the pitch associated with sound-first stimuli, and to "sound-first" for the pitch associated with light-first stimuli. The results clearly show that bayesian calibration is pitch-specific and is at work behind pitch-insensitive lag adaptation during temporal order judgment of audiovisual stimuli.     

Separable temporal metrics for time perception and anticipatory actions

Reliable estimates of time are essential for initiating interceptive actions at the right moment. However, our sense of time is surprisingly fallible. For instance, time perception can be distorted by prolonged exposure (adaptation) to movement. Here, we make use of this to determine if time perception and anticipatory actions rely on the same or on different temporal metrics. Consistent with previous reports, we find that the apparent duration of movement is mitigated by adaptation to more rapid motion, but is unchanged by adaptation to slower movement. By contrast, we find symmetrical effects of motion-adaptation on the timing of anticipatory interceptive actions, which are paralleled by changes in perceived speed for the adapted direction of motion. Our data thus reveal that anticipatory actions and perceived duration rely on different temporal metrics.     

A comprehensive model of audiovisual perception: both percept and temporal dynamics

The sparse information captured by the sensory systems is used by the brain to apprehend the environment, for example, to spatially locate the source of audiovisual stimuli. This is an ill-posed inverse problem whose inherent uncertainty can be solved by jointly processing the information, as well as introducing constraints during this process, on the way this multisensory information is handled. This process and its result--the percept--depend on the contextual conditions perception takes place in. To date, perception has been investigated and modeled on the basis of either one of two of its dimensions: the percept or the temporal dynamics of the process. Here, we extend our previously proposed audiovisual perception model to predict both these dimensions to capture the phenomenon as a whole. Starting from a behavioral analysis, we use a data-driven approach to elicit a bayesian network which infers the different percepts and dynamics of the process. Context-specific independence analyses enable us to use the model's structure to directly explore how different contexts affect the way subjects handle the same available information. Hence, we establish that, while the percepts yielded by a unisensory stimulus or by the non-fusion of multisensory stimuli may be similar, they result from different processes, as shown by their differing temporal dynamics. Moreover, our model predicts the impact of bottom-up (stimulus driven) factors as well as of top-down factors (induced by instruction manipulation) on both the perception process and the percept itself.     

Asynchrony adaptation reveals neural population code for audio-visual timing

The relative timing of auditory and visual stimuli is a critical cue for determining whether sensory signals relate to a common source and for making inferences about causality. However, the way in which the brain represents temporal relationships remains poorly understood. Recent studies indicate that our perception of multisensory timing is flexible--adaptation to a regular inter-modal delay alters the point at which subsequent stimuli are judged to be simultaneous. Here, we measure the effect of audio-visual asynchrony adaptation on the perception of a wide range of sub-second temporal relationships. We find distinctive patterns of induced biases that are inconsistent with the previous explanations based on changes in perceptual latency. Instead, our results can be well accounted for by a neural population coding model in which: (i) relative audio-visual timing is represented by the distributed activity across a relatively small number of neurons tuned to different delays; (ii) the algorithm for reading out this population code is efficient, but subject to biases owing to under-sampling; and (iii) the effect of adaptation is to modify neuronal response gain. These results suggest that multisensory timing information is represented by a dedicated population code and that shifts in perceived simultaneity following asynchrony adaptation arise from analogous neural processes to well-known perceptual after-effects.     

Spatially localized distortions of event time

A fundamental question about the perception of time is whether the neural mechanisms underlying temporal judgements are universal and centralized in the brain or modality specific and distributed. Time perception has traditionally been thought to be entirely dissociated from spatial vision. Here we show that the apparent duration of a dynamic stimulus can be manipulated in a local region of visual space by adapting to oscillatory motion or flicker. This implicates spatially localized temporal mechanisms in duration perception. We do not see concomitant changes in the time of onset or offset of the test patterns, demonstrating a direct local effect on duration perception rather than an indirect effect on the time course of neural processing. The effects of adaptation on duration perception can also be dissociated from motion or flicker perception per se. Although 20 Hz adaptation reduces both the apparent temporal frequency and duration of a 10 Hz test stimulus, 5 Hz adaptation increases apparent temporal frequency but has little effect on duration perception. We conclude that there is a peripheral, spatially localized, essentially visual component involved in sensing the duration of visual events.     

Bayesian action-perception computational model: interaction of production and recognition of cursive letters

In this paper, we study the collaboration of perception and action representations involved in cursive letter recognition and production. We propose a mathematical formulation for the whole perception-action loop, based on probabilistic modeling and bayesian inference, which we call the Bayesian Action-Perception (BAP) model. Being a model of both perception and action processes, the purpose of this model is to study the interaction of these processes. More precisely, the model includes a feedback loop from motor production, which implements an internal simulation of movement. Motor knowledge can therefore be involved during perception tasks. In this paper, we formally define the BAP model and show how it solves the following six varied cognitive tasks using bayesian inference: i) letter recognition (purely sensory), ii) writer recognition, iii) letter production (with different effectors), iv) copying of trajectories, v) copying of letters, and vi) letter recognition (with internal simulation of movements). We present computer simulations of each of these cognitive tasks, and discuss experimental predictions and theoretical developments.     

Neural Adaptation and Behavioral Measures of Temporal Processing and Speech Perception in Cochlear Implant Recipients

The objective was to determine if one of the neural temporal features, neural adaptation, can account for the across-subject variability in behavioral measures of temporal processing and speech perception performance in cochlear implant (CI) recipients. Neural adaptation is the phenomenon in which neural responses are the strongest at the beginning of the stimulus and decline following stimulus repetition (e.g., stimulus trains). It is unclear how this temporal property of neural responses relates to psychophysical measures of temporal processing (e.g., gap detection) or speech perception. The adaptation of the electrical compound action potential (ECAP) was obtained using 1000 pulses per second (pps) biphasic pulse trains presented directly to the electrode. The adaptation of the late auditory evoked potential (LAEP) was obtained using a sequence of 1-kHz tone bursts presented acoustically, through the cochlear implant. Behavioral temporal processing was measured using the Random Gap Detection Test at the most comfortable listening level. Consonant nucleus consonant (CNC) word and AzBio sentences were also tested. The results showed that both ECAP and LAEP display adaptive patterns, with a substantial across-subject variability in the amount of adaptation. No correlations between the amount of neural adaptation and gap detection thresholds (GDTs) or speech perception scores were found. The correlations between the degree of neural adaptation and demographic factors showed that CI users having more LAEP adaptation were likely to be those implanted at a younger age than CI users with less LAEP adaptation. The results suggested that neural adaptation, at least this feature alone, cannot account for the across-subject variability in temporal processing ability in the CI users. However, the finding that the LAEP adaptive pattern was less prominent in the CI group compared to the normal hearing group may suggest the important role of normal adaptation pattern at the cortical level in speech perception.     

Spike-Triggered Covariance Analysis Reveals Phenomenological Diversity of Contrast Adaptation in the Retina

When visual contrast changes, retinal ganglion cells adapt by adjusting their sensitivity as well as their temporal filtering characteristics. The latter has classically been described by contrast-induced gain changes that depend on temporal frequency. Here, we explored a new perspective on contrast-induced changes in temporal filtering by using spike-triggered covariance analysis to extract multiple parallel temporal filters for individual ganglion cells. Based on multielectrode-array recordings from ganglion cells in the isolated salamander retina, we found that contrast adaptation of temporal filtering can largely be captured by contrast-invariant sets of filters with contrast-dependent weights. Moreover, differences among the ganglion cells in the filter sets and their contrast-dependent contributions allowed us to phenomenologically distinguish three types of filter changes. The first type is characterized by newly emerging features at higher contrast, which can be reproduced by computational models that contain response-triggered gain-control mechanisms. The second type follows from stronger adaptation in the Off pathway as compared to the On pathway in On-Off-type ganglion cells. Finally, we found that, in a subset of neurons, contrast-induced filter changes are governed by particularly strong spike-timing dynamics, in particular by pronounced stimulus-dependent latency shifts that can be observed in these cells. Together, our results show that the contrast dependence of temporal filtering in retinal ganglion cells has a multifaceted phenomenology and that a multi-filter analysis can provide a useful basis for capturing the underlying signal-processing dynamics.     

Linking the computational structure of variance adaptation to biophysical mechanisms

In multiple sensory systems, adaptation to the variance of a sensory input changes the sensitivity, kinetics, and average response over timescales ranging from < 100 ms to tens of seconds. Here, we present a simple, biophysically relevant model of retinal contrast adaptation that accurately captures both the membrane potential response and all adaptive properties. The adaptive component of this model is a first-order kinetic process of the type used to describe ion channel gating and synaptic transmission. From the model, we conclude that all adaptive dynamics can be accounted for by depletion of a signaling mechanism, and that variance adaptation can be explained as adaptation to the mean of a rectified signal. The model parameters show strong similarity to known properties of bipolar cell synaptic vesicle pools. Diverse types of adaptive properties that implement theoretical principles of efficient coding can be generated by a single type of molecule or synapse with just a few microscopic states.     

Duration channels mediate human time perception

The task of deciding how long sensory events seem to last is one that the human nervous system appears to perform rapidly and, for sub-second intervals, seemingly without conscious effort. That these estimates can be performed within and between multiple sensory and motor domains suggest time perception forms one of the core, fundamental processes of our perception of the world around us. Given this significance, the current paucity in our understanding of how this process operates is surprising. One candidate mechanism for duration perception posits that duration may be mediated via a system of duration-selective 'channels', which are differentially activated depending on the match between afferent duration information and the channels' 'preferred' duration. However, this model awaits experimental validation. In the current study, we use the technique of sensory adaptation, and we present data that are well described by banks of duration channels that are limited in their bandwidth, sensory-specific, and appear to operate at a relatively early stage of visual and auditory sensory processing. Our results suggest that many of the computational principles the nervous system applies to coding visual spatial and auditory spectral information are common to its processing of temporal extent.     

Quantifying the effects of migration and mutation on adaptation and demography in spatially heterogeneous environments

How do mutation and gene flow influence population persistence, niche expansion and local adaptation in spatially heterogeneous environments? In this article, we analyse a demographic and evolutionary model of adaptation to an environment containing two habitats in equal frequencies, and we bridge the gap between different theoretical frameworks. Qualitatively, our model yields four qualitative types of outcomes: (i) global extinction of the population, (ii) adaptation to one habitat only, but also adaptation to both habitats with, (iii) specialized phenotypes or (iv) with generalized phenotypes, and we determine the conditions under which each equilibrium is reached. We derive new analytical approximations for the local densities and the distributions of traits in each habitat under a migration–selection–mutation balance, compute the equilibrium values of the means, variances and asymmetries of the local distributions of phenotypes, and contrast the effects of migration and mutation on the evolutionary outcome. We then check our analytical results by solving our model numerically, and also assess their robustness in the presence of demographic stochasticity. Although increased migration results in a decrease in local adaptation, mutation in our model does not influence the values of the local mean traits. Yet, both migration and mutation can have dramatic effects on population size and even lead to metapopulation extinction when selection is strong. Niche expansion, the ability for the population to adapt to both habitats, can also be prevented by small migration rates and a reduced evolutionary potential characterized by rare mutation events of small effects; however, niche expansion is otherwise the most likely outcome. Although our results are derived under the assumption of clonal reproduction, we finally show and discuss the links between our model and previous quantitative genetics models.     

Time‐shift experiments and patterns of adaptation across time and space

Time‐shift experiments provide measures of the mean fitness of a population in environments of different points in time. Here, we show how to use this type of data to decompose mean fitness into (1) the effect of the environment in which the population is transplanted, (2) the effect of the genetic composition of the population and (3) ‘temporal adaptation’, which measures how the population fits the environment at that time. We derive analytical results for the pattern of ‘temporal adaptation’ and show that it is in general maximal in the recent past. The link between ‘temporal adaptation’ and ‘local adaptation’ is discussed, and we show when patterns of adaptation in time and space are expected to be similar. Finally, we illustrate the potential use of this approach using a data set measuring the adaptation of HIV to the immune response of several recently infected patients.     

Temporal recalibration in vocalization induced by adaptation of delayed auditory feedback

Background

We ordinarily perceive our voice sound as occurring simultaneously with vocal production, but the sense of simultaneity in vocalization can be easily interrupted by delayed auditory feedback (DAF). DAF causes normal people to have difficulty speaking fluently but helps people with stuttering to improve speech fluency. However, the underlying temporal mechanism for integrating the motor production of voice and the auditory perception of vocal sound remains unclear. In this study, we investigated the temporal tuning mechanism integrating vocal sensory and voice sounds under DAF with an adaptation technique.

Methods and Findings

Participants produced a single voice sound repeatedly with specific delay times of DAF (0, 66, 133 ms) during three minutes to induce ‘Lag Adaptation’. They then judged the simultaneity between motor sensation and vocal sound given feedback. We found that lag adaptation induced a shift in simultaneity responses toward the adapted auditory delays. This indicates that the temporal tuning mechanism in vocalization can be temporally recalibrated after prolonged exposure to delayed vocal sounds. Furthermore, we found that the temporal recalibration in vocalization can be affected by averaging delay times in the adaptation phase.

Conclusions

These findings suggest vocalization is finely tuned by the temporal recalibration mechanism, which acutely monitors the integration of temporal delays between motor sensation and vocal sound.     

Spike-Frequency Adaptation of a Generalized Leaky Integrate-and-Fire Model Neuron

Although spike-frequency adaptation is a commonly observed property of neurons, its functional implications are still poorly understood. In this work, using a leaky integrate-and-fire neural model that includes a Ca²⁺-activated K⁺ current (I _AHP), we develop a quantitative theory of adaptation temporal dynamics and compare our results with recent in vivo intracellular recordings from pyramidal cells in the cat visual cortex. Experimentally testable relations between the degree and the time constant of spike-frequency adaptation are predicted. We also contrast the I _AHP model with an alternative adaptation model based on a dynamical firing threshold. Possible roles of adaptation in temporal computation are explored, as a a time-delayed neuronal self-inhibition mechanism. Our results include the following: (1) given the same firing rate, the variability of interspike intervals (ISIs) is either reduced or enhanced by adaptation, depending on whether the I _AHP dynamics is fast or slow compared with the mean ISI in the output spike train; (2) when the inputs are Poisson-distributed (uncorrelated), adaptation generates temporal anticorrelation between ISIs, we suggest that measurement of this negative correlation provides a probe to assess the strength of I _AHP in vivo; (3) the forward masking effect produced by the slow dynamics of I _AHP is nonlinear and effective at selecting the strongest input among competing sources of input signals.     

Inverse relationship between longevity and evolutionary rate of mitochondrial proteins in mammals and birds

Recently, an unexpected, positive correlation between the rate of evolution of mitochondrial proteins and longevity was reported. Here we re-analyze this relationship in various mammalian lineages using a bayesian phylogenetic analysis of amino-acid sequences, allowing for variable evolutionary rates across sites and species. A negative relationship between protein evolutionary rate and species longevity is reported for all oxidative phosphorylation complexes. A detailed analysis of the cytochrome b in 528 mammals reinforced this result, which contradicts previous publications. Reconducting the analysis in birds yielded similar results. We explain the discrepancy between this and previous reports by our improved taxon sampling and more appropriate methodology: unlike distance-based methods, the tree-based bayesian approach can take into account the high variation of substitution rate across amino-acid sites, and the resulting multiple substitution events. We discuss how our analysis contradicts Rottenberg’s rationale, but does not dismiss his proposal of a longevity-dependent selective pressure on mitochondrial mutation rate in mammals and birds. This is because his interpretation invokes adaptation as the single evolutionary force at work, disregarding the effects of mutation, genetic drift, and purifying selection.     

The many faces of research on face perception

Face perception is fundamental to human social interaction. Many different types of important information are visible in faces and the processes and mechanisms involved in extracting this information are complex and can be highly specialized. The importance of faces has long been recognized by a wide range of scientists. Importantly, the range of perspectives and techniques that this breadth has brought to face perception research has, in recent years, led to many important advances in our understanding of face processing. The articles in this issue on face perception each review a particular arena of interest in face perception, variously focusing on (i) the social aspects of face perception (attraction, recognition and emotion), (ii) the neural mechanisms underlying face perception (using brain scanning, patient data, direct stimulation of the brain, visual adaptation and single-cell recording), and (iii) comparative aspects of face perception (comparing adult human abilities with those of chimpanzees and children). Here, we introduce the central themes of the issue and present an overview of the articles.     

Cross-Category Adaptation: Objects Produce Gender Adaptation in the Perception of Faces

Adaptation aftereffects have been found for low-level visual features such as colour, motion and shape perception, as well as higher-level features such as gender, race and identity in domains such as faces and biological motion. It is not yet clear if adaptation effects in humans extend beyond this set of higher order features. The aim of this study was to investigate whether objects highly associated with one gender, e.g. high heels for females or electric shavers for males can modulate gender perception of a face. In two separate experiments, we adapted subjects to a series of objects highly associated with one gender and subsequently asked participants to judge the gender of an ambiguous face. Results showed that participants are more likely to perceive an ambiguous face as male after being exposed to objects highly associated to females and vice versa. A gender adaptation aftereffect was obtained despite the adaptor and test stimuli being from different global categories (objects and faces respectively). These findings show that our perception of gender from faces is highly affected by our environment and recent experience. This suggests two possible mechanisms: (a) that perception of the gender associated with an object shares at least some brain areas with those responsible for gender perception of faces and (b) adaptation to gender, which is a high-level concept, can modulate brain areas that are involved in facial gender perception through top-down processes.     

Sequence matters – selective adaptation in electroantennographic response to binary odour mixtures by the Colorado potato beetle

Chemically mediated behaviour of insects is often strongly affected by mixtures of odour stimuli and their temporal characteristics. Both sensory transduction and central processing of odour mixtures can give rise to several different kinds of interaction, which can influence how individual components are perceived and processed. In particular, odour mixtures have been examined in model experiments as premixed binary mixtures in comparison with pure odour stimuli. Only in few experiments, the influence of the temporal structure of odour mixtures on odour perception has been taken into account. Natural odour stimuli often have a pulsed structure and may in general be superimposed on a background of irrelevant or interfering compounds, which can fluctuate with different frequencies, depending on their source. To achieve a better representation of these natural conditions, our odour mixing experiments apply a new kind of stimulation protocol: odours were not premixed but superimposed with a specific time pattern; one odour stimulus was presented as a longer persisting background and the second stimulus was a superimposed short test signal. To gain an overview of odour interaction patterns in the Colorado potato beetle by causing adaptation of one receptor population at naturally occurring levels of concentration and time intervals, electroantennographic recordings were made on excised antennae. A matrix of 12 stimulus compounds led to 132 pairs of compounds tested, each in the role of background and test stimulus. In 64 cases, the interaction was significantly different, when the role of background and stimulus was exchanged. Interaction patterns ranging from no interference (independence) to suppression were found and assigned to four clearly distinguishable types. We discuss that the observed effects of the presentation sequence in odour mixtures may contribute to the mechanisms of olfactory pattern recognition and olfactory contrast perception by insects.     

Imagine Jane and identify John: face identity aftereffects induced by imagined faces

It is not known whether prolonged exposure to perceived and imagined complex visual images produces similar shifts in subsequent perception through selective adaptation. This question is important because a positive finding would suggest that perception and imagery of visual stimuli are mediated by shared neural networks. In this study, we used a selective adaptation procedure designed to induce high-level face-identity aftereffects--a phenomenon in which extended exposure to a particular face facilitates recognition of subsequent faces with opposite features while impairing recognition of all other faces. We report here that adaptation to either real or imagined faces produces a similar shift in perception and that identity boundaries represented in real and imagined faces are equivalent. Together, our results show that identity information contained in imagined and real faces produce similar behavioral outcomes. Our findings of high-level visual aftereffects induced by imagined stimuli can be taken as evidence for the involvement of shared neural networks that mediate perception and imagery of complex visual stimuli.     

On the cyclic nature of perception in vision versus audition

Does our perceptual awareness consist of a continuous stream, or a discrete sequence of perceptual cycles, possibly associated with the rhythmic structure of brain activity? This has been a long-standing question in neuroscience. We review recent psychophysical and electrophysiological studies indicating that part of our visual awareness proceeds in approximately 7–13 Hz cycles rather than continuously. On the other hand, experimental attempts at applying similar tools to demonstrate the discreteness of auditory awareness have been largely unsuccessful. We argue and demonstrate experimentally that visual and auditory perception are not equally affected by temporal subsampling of their respective input streams: video sequences remain intelligible at sampling rates of two to three frames per second, whereas audio inputs lose their fine temporal structure, and thus all significance, below 20–30 samples per second. This does not mean, however, that our auditory perception must proceed continuously. Instead, we propose that audition could still involve perceptual cycles, but the periodic sampling should happen only after the stage of auditory feature extraction. In addition, although visual perceptual cycles can follow one another at a spontaneous pace largely independent of the visual input, auditory cycles may need to sample the input stream more flexibly, by adapting to the temporal structure of the auditory inputs.