Rapid,generalized adaptation to asynchronous audiovisual speech

The brain is adaptive. The speed of propagation through air, and of low-level sensory processing, differs markedly between auditory and visual stimuli; yet the brain can adapt to compensate for the resulting cross-modal delays. Studies investigating temporal recalibration to audiovisual speech have used prolonged adaptation procedures, suggesting that adaptation is sluggish. Here, we show that adaptation to asynchronous audiovisual speech occurs rapidly. Participants viewed a brief clip of an actor pronouncing a single syllable. The voice was either advanced or delayed relative to the corresponding lip movements, and participants were asked to make a synchrony judgement. Although we did not use an explicit adaptation procedure, we demonstrate rapid recalibration based on a single audiovisual event. We find that the point of subjective simultaneity on each trial is highly contingent upon the modality order of the preceding trial. We find compelling evidence that rapid recalibration generalizes across different stimuli, and different actors. Finally, we demonstrate that rapid recalibration occurs even when auditory and visual events clearly belong to different actors. These results suggest that rapid temporal recalibration to audiovisual speech is primarily mediated by basic temporal factors, rather than higher-order factors such as perceived simultaneity and source identity.     

Pigeons' memory for sequences of light flashes: reliance on temporal properties and evidence for delay interval/gap confusion

In Experiment 1, pigeons were trained to discriminate between sequences of two and four light flashes (illumination of the feeder). Retention functions obtained with dark delays exhibited a choose-many bias at a 1-s delay and a choose-few bias at delays of 4 and 8s. Retention functions obtained with illuminated delays only displayed a slight choose-few bias. In Experiment 2, additional birds were trained with the same sample sequences. However, the intertrial interval was illuminated by the houselight for Group Light, and it was dark for Group Dark. The acquisition data suggested that multiple temporal features of the light flash sequences controlled choice responding. For both groups, the retention functions were similar to those obtained in Experiment 1. In Experiment 3, baseline training with a 1-s dark delay eliminated the choose-many bias, but a significant choose-few bias at extended dark delays was still observed. Pigeons discriminate light flash sequences by relying on temporal properties of the sequence rather than using an event switch to count flashes. The biased-forgetting effects obtained in these studies appear to be primarily due to confusion between the delay interval and the gap between light flashes.     

Effect before Cause: Supramodal Recalibration of Sensorimotor Timing

Background

Our motor actions normally generate sensory events, but how do we know which events were self generated and which have external causes? Here we use temporal adaptation to investigate the processing stage and generality of our sensorimotor timing estimates.

Methodology/Principal Findings

Adaptation to artificially-induced delays between action and event can produce a startling percept—upon removal of the delay it feels as if the sensory event precedes its causative action. This temporal recalibration of action and event occurs in a quantitatively similar manner across the sensory modalities. Critically, it is robust to the replacement of one sense during the adaptation phase with another sense during the test judgment.

Conclusions/Significance

Our findings suggest a high-level, supramodal recalibration mechanism. The effects are well described by a simple model which attempts to preserve the expected synchrony between action and event, but only when causality indicates it is reasonable to do so. We further demonstrate that this model successfully characterises related adaptation data from outside the sensorimotor domain.     

The role of the rat medial prefrontal cortex in adapting to changes in instrumental contingency

In order to select actions appropriate to current needs, a subject must identify relationships between actions and events. Control over the environment is determined by the degree to which action consequences can be predicted, as described by action-outcome contingencies--i.e. performing an action should affect the probability of the outcome. We evaluated in a first experiment adaptation to contingency changes in rats with neurotoxic lesions of the medial prefrontal cortex. Results indicate that this brain region is not critical to adjust instrumental responding to a negative contingency where the rats must refrain from pressing a lever, as this action prevents reward delivery. By contrast, this brain region is required to reduce responding in a non-contingent situation where the same number of rewards is freely delivered and actions do not affect the outcome any more. In a second experiment, we determined that this effect does not result from a different perception of temporal relationships between actions and outcomes since lesioned rats adapted normally to gradually increasing delays in reward delivery. These data indicate that the medial prefrontal cortex is not directly involved in evaluating the correlation between action--and reward--rates or in the perception of reward delays. The deficit in lesioned rats appears to consist of an abnormal response to the balance between contingent and non-contingent rewards. By highlighting the role of prefrontal regions in adapting to the causal status of actions, these data contribute to our understanding of the neural basis of choice tasks.     

Temporal recalibration in vocalization induced by adaptation of delayed auditory feedback

Background

We ordinarily perceive our voice sound as occurring simultaneously with vocal production, but the sense of simultaneity in vocalization can be easily interrupted by delayed auditory feedback (DAF). DAF causes normal people to have difficulty speaking fluently but helps people with stuttering to improve speech fluency. However, the underlying temporal mechanism for integrating the motor production of voice and the auditory perception of vocal sound remains unclear. In this study, we investigated the temporal tuning mechanism integrating vocal sensory and voice sounds under DAF with an adaptation technique.

Methods and Findings

Participants produced a single voice sound repeatedly with specific delay times of DAF (0, 66, 133 ms) during three minutes to induce ‘Lag Adaptation’. They then judged the simultaneity between motor sensation and vocal sound given feedback. We found that lag adaptation induced a shift in simultaneity responses toward the adapted auditory delays. This indicates that the temporal tuning mechanism in vocalization can be temporally recalibrated after prolonged exposure to delayed vocal sounds. Furthermore, we found that the temporal recalibration in vocalization can be affected by averaging delay times in the adaptation phase.

Conclusions

These findings suggest vocalization is finely tuned by the temporal recalibration mechanism, which acutely monitors the integration of temporal delays between motor sensation and vocal sound.     

Adaptation to visual feedback delay influences visuomotor learning

Computational theory of motor control suggests that the brain continuously monitors motor commands, to predict their sensory consequences before actual sensory feedback becomes available. Such prediction error is a driving force of motor learning, and therefore appropriate associations between motor commands and delayed sensory feedback signals are crucial. Indeed, artificially introduced delays in visual feedback have been reported to degrade motor learning. However, considering our perceptual ability to causally bind our own actions with sensory feedback, demonstrated by the decrease in the perceived time delay following repeated exposure to an artificial delay, we hypothesized that such perceptual binding might alleviate deficits of motor learning associated with delayed visual feedback. Here, we evaluated this hypothesis by investigating the ability of human participants to adapt their reaching movements in response to a novel visuomotor environment with 3 visual feedback conditions--no-delay, sudden-delay, and adapted-delay. To introduce novelty into the trials, the cursor position, which originally indicated the hand position in baseline trials, was rotated around the starting position. In contrast to the no-delay condition, a 200-ms delay was artificially introduced between the cursor and hand positions during the presence of visual rotation (sudden-delay condition), or before the application of visual rotation (adapted-delay condition). We compared the learning rate (representing how the movement error modifies the movement direction in the subsequent trial) between the 3 conditions. In comparison with the no-delay condition, the learning rate was significantly degraded for the sudden-delay condition. However, this degradation was significantly alleviated by prior exposure to the delay (adapted-delay condition). Our data indicate the importance of appropriate temporal associations between motor commands and sensory feedback in visuomotor learning. Moreover, they suggest that the brain is able to account for such temporal associations in a flexible manner.     

Motor cortex representation of the upper-limb in individuals born without a hand

The body schema is an action-related representation of the body that arises from activity in a network of multiple brain areas. While it was initially thought that the body schema developed with experience, the existence of phantom limbs in individuals born without a limb (amelics) led to the suggestion that it was innate. The problem with this idea, however, is that the vast majority of amelics do not report the presence of a phantom limb. Transcranial magnetic stimulation (TMS) applied over the primary motor cortex (M1) of traumatic amputees can evoke movement sensations in the phantom, suggesting that traumatic amputation does not delete movement representations of the missing hand. Given this, we asked whether the absence of a phantom limb in the majority of amelics means that the motor cortex does not contain a cortical representation of the missing limb, or whether it is present but has been deactivated by the lack of sensorimotor experience. In four upper-limb amelic subjects we directly stimulated the arm/hand region of M1 to see 1) whether we could evoke phantom sensations, and 2) whether muscle representations in the two cortices were organised asymmetrically. TMS applied over the motor cortex contralateral to the missing limb evoked contractions in stump muscles but did not evoke phantom movement sensations. The location and extent of muscle maps varied between hemispheres but did not reveal any systematic asymmetries. In contrast, forearm muscle thresholds were always higher for the missing limb side. We suggest that phantom movement sensations reported by some upper limb amelics are mostly driven by vision and not by the persistence of motor commands to the missing limb within the sensorimotor cortex. We propose that prewired movement representations of a limb need the experience of movement to be expressed within the primary motor cortex.     

Both of us disgusted in My insula: the common neural basis of seeing and feeling disgust

What neural mechanism underlies the capacity to understand the emotions of others? Does this mechanism involve brain areas normally involved in experiencing the same emotion? We performed an fMRI study in which participants inhaled odorants producing a strong feeling of disgust. The same participants observed video clips showing the emotional facial expression of disgust. Observing such faces and feeling disgust activated the same sites in the anterior insula and to a lesser extent in the anterior cingulate cortex. Thus, as observing hand actions activates the observer's motor representation of that action, observing an emotion activates the neural representation of that emotion. This finding provides a unifying mechanism for understanding the behaviors of others.     

Delays in Human-Computer Interaction and Their Effects on Brain Activity

The temporal contingency of feedback is an essential requirement of successful human-computer interactions. The timing of feedback not only affects the behavior of a user but is also accompanied by changes in psychophysiology and neural activity. In three fMRI experiments we systematically studied the impact of delayed feedback on brain activity while subjects performed an auditory categorization task. In the first fMRI experiment, we analyzed the effects of rare and thus unexpected delays of different delay duration on brain activity. In the second experiment, we investigated if users can adapt to frequent delays. Therefore, delays were presented as often as immediate feedback. In a third experiment, the influence of interaction outage was analyzed by measuring the effect of infrequent omissions of feedback on brain activity. The results show that unexpected delays in feedback presentation compared to immediate feedback stronger activate inter alia bilateral the anterior insular cortex, the posterior medial frontal cortex, the left inferior parietal lobule and the right inferior frontal junction. The strength of this activation increases with the duration of the delay. Thus, delays interrupt the course of an interaction and trigger an orienting response that in turn activates brain regions of action control. If delays occur frequently, users can adapt, delays become expectable, and the brain activity in the observed network diminishes over the course of the interaction. However, introducing rare omissions of expected feedback reduces the system’s trustworthiness which leads to an increase in brain activity not only in response to such omissions but also following frequently occurring and thus expected delays.     

Conditioning and time representation in long short-term memory networks

Dopaminergic models based on the temporal-difference learning algorithm usually do not differentiate trace from delay conditioning. Instead, they use a fixed temporal representation of elapsed time since conditioned stimulus onset. Recently, a new model was proposed in which timing is learned within a long short-term memory (LSTM) artificial neural network representing the cerebral cortex (Rivest et al. in J Comput Neurosci 28(1):107–130, 2010). In this paper, that model’s ability to reproduce and explain relevant data, as well as its ability to make interesting new predictions, are evaluated. The model reveals a strikingly different temporal representation between trace and delay conditioning since trace conditioning requires working memory to remember the past conditioned stimulus while delay conditioning does not. On the other hand, the model predicts no important difference in DA responses between those two conditions when trained on one conditioning paradigm and tested on the other. The model predicts that in trace conditioning, animal timing starts with the conditioned stimulus offset as opposed to its onset. In classical conditioning, it predicts that if the conditioned stimulus does not disappear after the reward, the animal may expect a second reward. Finally, the last simulation reveals that the buildup of activity of some units in the networks can adapt to new delays by adjusting their rate of integration. Most importantly, the paper shows that it is possible, with the proposed architecture, to acquire discharge patterns similar to those observed in dopaminergic neurons and in the cerebral cortex on those tasks simply by minimizing a predictive cost function.     

Reconstructing speech from human auditory cortex

How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.     

Light flashes of different durations (0.063-3.33 msec) phase shift the circadian flight activity of a bat

The phase-response curve (PRC) for the circadian rhythm in the flight activity of a cave-dwelling bat, Hipposideros speoris, constructed with 0.063-msec light flashes, reported here is the first of its kind for any circadian system and is unlike any other phase-response curves constructed for other nocturnal animals. The phase responding with maximal advances (90 degrees) and the phase responding with maximal delays (0 degree) of this PRC were exposed to light flashes of systematically varying durations from 0.083 to 3.33-msec. For 0 degree phase, the flashes of 0.063-3.33 msec effected delay phase shifts of comparable magnitude. For 90 degrees phase, the flashes of 0.063-1.0 msec effected advance phase shifts, whereas 3.33-msec flashes effected unmistakable delay phase shifts with advancing transients. Phase shifts evoked with such light flashes are further compared with phase shifts evoked with pulses of longer durations (15 min to 2.8 hr) for H. speoris.     

Causal inference regulates audiovisual spatial recalibration via its influence on audiovisual perception

To obtain a coherent perception of the world, our senses need to be in alignment. When we encounter misaligned cues from two sensory modalities, the brain must infer which cue is faulty and recalibrate the corresponding sense. We examined whether and how the brain uses cue reliability to identify the miscalibrated sense by measuring the audiovisual ventriloquism aftereffect for stimuli of varying visual reliability. To adjust for modality-specific biases, visual stimulus locations were chosen based on perceived alignment with auditory stimulus locations for each participant. During an audiovisual recalibration phase, participants were presented with bimodal stimuli with a fixed perceptual spatial discrepancy; they localized one modality, cued after stimulus presentation. Unimodal auditory and visual localization was measured before and after the audiovisual recalibration phase. We compared participants’ behavior to the predictions of three models of recalibration: (a) Reliability-based: each modality is recalibrated based on its relative reliability—less reliable cues are recalibrated more; (b) Fixed-ratio: the degree of recalibration for each modality is fixed; (c) Causal-inference: recalibration is directly determined by the discrepancy between a cue and its estimate, which in turn depends on the reliability of both cues, and inference about how likely the two cues derive from a common source. Vision was hardly recalibrated by audition. Auditory recalibration by vision changed idiosyncratically as visual reliability decreased: the extent of auditory recalibration either decreased monotonically, peaked at medium visual reliability, or increased monotonically. The latter two patterns cannot be explained by either the reliability-based or fixed-ratio models. Only the causal-inference model of recalibration captures the idiosyncratic influences of cue reliability on recalibration. We conclude that cue reliability, causal inference, and modality-specific biases guide cross-modal recalibration indirectly by determining the perception of audiovisual stimuli.     

The Effect of Visual Apparent Motion on Audiovisual Simultaneity

Visual motion information from dynamic environments is important in multisensory temporal perception. However, it is unclear how visual motion information influences the integration of multisensory temporal perceptions. We investigated whether visual apparent motion affects audiovisual temporal perception. Visual apparent motion is a phenomenon in which two flashes presented in sequence in different positions are perceived as continuous motion. Across three experiments, participants performed temporal order judgment (TOJ) tasks. Experiment 1 was a TOJ task conducted in order to assess audiovisual simultaneity during perception of apparent motion. The results showed that the point of subjective simultaneity (PSS) was shifted toward a sound-lead stimulus, and the just noticeable difference (JND) was reduced compared with a normal TOJ task with a single flash. This indicates that visual apparent motion affects audiovisual simultaneity and improves temporal discrimination in audiovisual processing. Experiment 2 was a TOJ task conducted in order to remove the influence of the amount of flash stimulation from Experiment 1. The PSS and JND during perception of apparent motion were almost identical to those in Experiment 1, but differed from those for successive perception when long temporal intervals were included between two flashes without motion. This showed that the result obtained under the apparent motion condition was unaffected by the amount of flash stimulation. Because apparent motion was produced by a constant interval between two flashes, the results may be accounted for by specific prediction. In Experiment 3, we eliminated the influence of prediction by randomizing the intervals between the two flashes. However, the PSS and JND did not differ from those in Experiment 1. It became clear that the results obtained for the perception of visual apparent motion were not attributable to prediction. Our findings suggest that visual apparent motion changes temporal simultaneity perception and improves temporal discrimination in audiovisual processing.     

Dispersed activation in the left temporal cortex for speech-reading in congenitally deaf people

Does the lateral temporal cortex require acoustic exposure in order to become specialized for speech processing? Six hearing participants and six congenitally deaf participants, all with spoken English as their first langugage, were scanned using functional magnetic resonance imaging while performing a simple speech-reading task. Focal activation of the left lateral temporal cortex was significantly reduced in the deaf group compared with the hearing group. Activation within this region was present in individual deaf participants, but varied in location from person to person. Early acoustic experience may be required for regions within the left temporal cortex in order to develop into a coherent network with subareas devoted to specific speech analysis functions.     

Schizotypal perceptual aberrations of time: correlation between score, behavior and brain activity

A fundamental trait of the human self is its continuum experience of space and time. Perceptual aberrations of this spatial and temporal continuity is a major characteristic of schizophrenia spectrum disturbances--including schizophrenia, schizotypal personality disorder and schizotypy. We have previously found the classical Perceptual Aberration Scale (PAS) scores, related to body and space, to be positively correlated with both behavior and temporo-parietal activation in healthy participants performing a task involving self-projection in space. However, not much is known about the relationship between temporal perceptual aberration, behavior and brain activity. To this aim, we composed a temporal Perceptual Aberration Scale (tPAS) similar to the traditional PAS. Testing on 170 participants suggested similar performance for PAS and tPAS. We then correlated tPAS and PAS scores to participants' performance and neural activity in a task of self-projection in time. tPAS scores correlated positively with reaction times across task conditions, as did PAS scores. Evoked potential mapping and electrical neuroimaging showed self-projection in time to recruit a network of brain regions at the left anterior temporal cortex, right temporo-parietal junction, and occipito-temporal cortex, and duration of activation in this network positively correlated with tPAS and PAS scores. These data demonstrate that schizotypal perceptual aberrations of both time and space, as reflected by tPAS and PAS scores, are positively correlated with performance and brain activation during self-projection in time in healthy individuals along the schizophrenia spectrum.     

Representation of Cognitive Reappraisal Goals in Frontal Gamma Oscillations

Recently, numerous efforts have been made to understand the neural mechanisms underlying cognitive regulation of emotion, such as cognitive reappraisal. Many studies have reported that cognitive control of emotion induces increases in neural activity of the control system, including the prefrontal cortex and the dorsal anterior cingulate cortex, and increases or decreases (depending upon the regulation goal) in neural activity of the appraisal system, including the amygdala and the insula. It has been hypothesized that information about regulation goals needs to be processed through interactions between the control and appraisal systems in order to support cognitive reappraisal. However, how this information is represented in the dynamics of cortical activity remains largely unknown. To address this, we investigated temporal changes in gamma band activity (35–55 Hz) in human electroencephalograms during a cognitive reappraisal task that was comprised of three reappraisal goals: to decease, maintain, or increase emotional responses modulated by affect-laden pictures. We examined how the characteristics of gamma oscillations, such as spectral power and large-scale phase synchronization, represented cognitive reappraisal goals. We found that left frontal gamma power decreased, was sustained, or increased when the participants suppressed, maintained, or amplified their emotions, respectively. This change in left frontal gamma power appeared during an interval of 1926 to 2453 ms after stimulus onset. We also found that the number of phase-synchronized pairs of gamma oscillations over the entire brain increased when participants regulated their emotions compared to when they maintained their emotions. These results suggest that left frontal gamma power may reflect cortical representation of emotional states modulated by cognitive reappraisal goals and gamma phase synchronization across whole brain regions may reflect emotional regulatory efforts to achieve these goals. Our study may provide the basis for an electroencephalogram-based neurofeedback system for the cognitive regulation of emotion.     

Subjective Somatosensory Experiences Disclosed by Focused Attention: Cortical-Hippocampal-Insular and Amygdala Contributions

In order to explore the neurobiological foundations of qualitative subjective experiences, the present study was designed to correlate objective third-person brain fMRI measures with subjective first-person identification and scaling of local, subtle, and specific somatosensory sensations, obtained directly after the imaging procedure. Thus, thirty-four volunteers were instructed to focus and sustain their attention to either provoked or spontaneous sensations of each thumb during the fMRI procedure. By means of a Likert scale applied immediately afterwards, the participants recalled and evaluated the intensity of their attention and identified specific somatosensory sensations (e.g. pulsation, vibration, heat). Using the subject''s subjective scores as covariates to model both attention intensity and general somatosensory experiences regressors, the whole-brain random effect analyses revealed activations in the frontopolar prefrontal cortex (BA10), primary somatosensory cortex (BA1), premotor cortex (BA 6), precuneus (BA 7), temporopolar cortex (BA 38), inferior parietal lobe (BA 39), hippocampus, insula and amygdala. Furthermore, BA10 showed differential activity, with ventral BA10 correlating exclusively with attention (r(32) = 0.54, p = 0.0013) and dorsal BA10 correlating exclusively with somatosensory sensation (r(32) = 0.46, p = 0.007). All other reported brain areas showed significant positive correlations solely with subjective somatosensory experiences reports. These results provide evidence that the frontopolar prefrontal cortex has dissociable functions depending on specific cognitive demands; i.e. the dorsal portion of the frontopolar prefrontal cortex in conjunction with primary somatosensory cortex, temporopolar cortex, inferior parietal lobe, hippocampus, insula and amygdala are involved in the processing of spontaneous general subjective somatosensory experiences disclosed by focused and sustained attention.     

Alternative time representation in dopamine models

Dopaminergic neuron activity has been modeled during learning and appetitive behavior, most commonly using the temporal-difference (TD) algorithm. However, a proper representation of elapsed time and of the exact task is usually required for the model to work. Most models use timing elements such as delay-line representations of time that are not biologically realistic for intervals in the range of seconds. The interval-timing literature provides several alternatives. One of them is that timing could emerge from general network dynamics, instead of coming from a dedicated circuit. Here, we present a general rate-based learning model based on long short-term memory (LSTM) networks that learns a time representation when needed. Using a naïve network learning its environment in conjunction with TD, we reproduce dopamine activity in appetitive trace conditioning with a constant CS-US interval, including probe trials with unexpected delays. The proposed model learns a representation of the environment dynamics in an adaptive biologically plausible framework, without recourse to delay lines or other special-purpose circuits. Instead, the model predicts that the task-dependent representation of time is learned by experience, is encoded in ramp-like changes in single-neuron activity distributed across small neural networks, and reflects a temporal integration mechanism resulting from the inherent dynamics of recurrent loops within the network. The model also reproduces the known finding that trace conditioning is more difficult than delay conditioning and that the learned representation of the task can be highly dependent on the types of trials experienced during training. Finally, it suggests that the phasic dopaminergic signal could facilitate learning in the cortex.     

Recruitment of occipital cortex during sensory substitution training linked to subjective experience of seeing in people with blindness

Over three months of intensive training with a tactile stimulation device, 18 blind and 10 blindfolded seeing subjects improved in their ability to identify geometric figures by touch. Seven blind subjects spontaneously reported 'visual qualia', the subjective sensation of seeing flashes of light congruent with tactile stimuli. In the latter subjects tactile stimulation evoked activation of occipital cortex on electroencephalography (EEG). None of the blind subjects who failed to experience visual qualia, despite identical tactile stimulation training, showed EEG recruitment of occipital cortex. None of the blindfolded seeing humans reported visual-like sensations during tactile stimulation. These findings support the notion that the conscious experience of seeing is linked to the activation of occipital brain regions in people with blindness. Moreover, the findings indicate that provision of visual information can be achieved through non-visual sensory modalities which may help to minimize the disability of blind individuals, affording them some degree of object recognition and navigation aid.