Opposite Distortions in Interval Timing Perception for Visual and Auditory Stimuli with Temporal Modulations

When an object is presented visually and moves or flickers, the perception of its duration tends to be overestimated. Such an overestimation is called time dilation. Perceived time can also be distorted when a stimulus is presented aurally as an auditory flutter, but the mechanisms and their relationship to visual processing remains unclear. In the present study, we measured interval timing perception while modulating the temporal characteristics of visual and auditory stimuli, and investigated whether the interval times of visually and aurally presented objects shared a common mechanism. In these experiments, participants compared the durations of flickering or fluttering stimuli to standard stimuli, which were presented continuously. Perceived durations for auditory flutters were underestimated, while perceived durations of visual flickers were overestimated. When auditory flutters and visual flickers were presented simultaneously, these distortion effects were cancelled out. When auditory flutters were presented with a constantly presented visual stimulus, the interval timing perception of the visual stimulus was affected by the auditory flutters. These results indicate that interval timing perception is governed by independent mechanisms for visual and auditory processing, and that there are some interactions between the two processing systems.     

Activation of duration-sensitive auditory cortical fields in humans

The influence of stimulus duration on auditory evoked potentials (AEPs) was examined for tones varying randomly in duration, location, and frequency in an auditory selective attention task. Stimulus duration effects were isolated as duration difference waves by subtracting AEPs to short duration tones from AEPs to longer duration tones of identical location, frequency and rise time. This analysis revealed that AEP components generally increased in amplitude and decreased in latency with increments in signal duration, with evidence of longer temporal integration times for lower frequency tones. Different temporal integration functions were seen for different N1 subcomponents. The results suggest that different auditory cortical areas have different temporal integration times, and that these functions vary as a function of tone frequency.     

Naturalistic Stimulus Structure Determines the Integration of Audiovisual Looming Signals in Binocular Rivalry

Rapid integration of biologically relevant information is crucial for the survival of an organism. Most prominently, humans should be biased to attend and respond to looming stimuli that signal approaching danger (e.g. predator) and hence require rapid action. This psychophysics study used binocular rivalry to investigate the perceptual advantage of looming (relative to receding) visual signals (i.e. looming bias) and how this bias can be influenced by concurrent auditory looming/receding stimuli and the statistical structure of the auditory and visual signals.Subjects were dichoptically presented with looming/receding visual stimuli that were paired with looming or receding sounds. The visual signals conformed to two different statistical structures: (1) a ‘simple’ random-dot kinematogram showing a starfield and (2) a “naturalistic” visual Shepard stimulus. Likewise, the looming/receding sound was (1) a simple amplitude- and frequency-modulated (AM-FM) tone or (2) a complex Shepard tone. Our results show that the perceptual looming bias (i.e. the increase in dominance times for looming versus receding percepts) is amplified by looming sounds, yet reduced and even converted into a receding bias by receding sounds. Moreover, the influence of looming/receding sounds on the visual looming bias depends on the statistical structure of both the visual and auditory signals. It is enhanced when audiovisual signals are Shepard stimuli.In conclusion, visual perception prioritizes processing of biologically significant looming stimuli especially when paired with looming auditory signals. Critically, these audiovisual interactions are amplified for statistically complex signals that are more naturalistic and known to engage neural processing at multiple levels of the cortical hierarchy.     

Auditory Discrimination Learning: Role of Working Memory

Perceptual training is generally assumed to improve perception by modifying the encoding or decoding of sensory information. However, this assumption is incompatible with recent demonstrations that transfer of learning can be enhanced by across-trial variation of training stimuli or task. Here we present three lines of evidence from healthy adults in support of the idea that the enhanced transfer of auditory discrimination learning is mediated by working memory (WM). First, the ability to discriminate small differences in tone frequency or duration was correlated with WM measured with a tone n-back task. Second, training frequency discrimination around a variable frequency transferred to and from WM learning, but training around a fixed frequency did not. The transfer of learning in both directions was correlated with a reduction of the influence of stimulus variation in the discrimination task, linking WM and its improvement to across-trial stimulus interaction in auditory discrimination. Third, while WM training transferred broadly to other WM and auditory discrimination tasks, variable-frequency training on duration discrimination did not improve WM, indicating that stimulus variation challenges and trains WM only if the task demands stimulus updating in the varied dimension. The results provide empirical evidence as well as a theoretic framework for interactions between cognitive and sensory plasticity during perceptual experience.     

The effect of study-test modalities on the remembrance of subjective duration from long-term memory

It was examined whether stimulus modality (auditory vs. visual) affects the retrieval of subjective duration from memory. In two experiments the temporal generalization paradigm was used. Participants had to decide whether the previously learned standard duration (400 ms) occurred in the context of comparison stimuli. Two major results were found. (1) Discrimination was more accurate if the training and testing stimuli were of the same modality than if they were of opposite modalities. (2) If both modality of learning and modality of testing were different, subjects systematically underestimated the test durations, i.e. temporal generalization gradients (the proportion of identifications of a stimulus as the standard, plotted against stimulus duration) shifted to the right. The observed shift is interpreted as a result of a delayed timing process.     

Distortions of subjective time perception within and across senses

Background

The ability to estimate the passage of time is of fundamental importance for perceptual and cognitive processes. One experience of time is the perception of duration, which is not isomorphic to physical duration and can be distorted by a number of factors. Yet, the critical features generating these perceptual shifts in subjective duration are not understood.

Methodology/Findings

We used prospective duration judgments within and across sensory modalities to examine the effect of stimulus predictability and feature change on the perception of duration. First, we found robust distortions of perceived duration in auditory, visual and auditory-visual presentations despite the predictability of the feature changes in the stimuli. For example, a looming disc embedded in a series of steady discs led to time dilation, whereas a steady disc embedded in a series of looming discs led to time compression. Second, we addressed whether visual (auditory) inputs could alter the perception of duration of auditory (visual) inputs. When participants were presented with incongruent audio-visual stimuli, the perceived duration of auditory events could be shortened or lengthened by the presence of conflicting visual information; however, the perceived duration of visual events was seldom distorted by the presence of auditory information and was never perceived shorter than their actual durations.

Conclusions/Significance

These results support the existence of multisensory interactions in the perception of duration and, importantly, suggest that vision can modify auditory temporal perception in a pure timing task. Insofar as distortions in subjective duration can neither be accounted for by the unpredictability of an auditory, visual or auditory-visual event, we propose that it is the intrinsic features of the stimulus that critically affect subjective time distortions.     

Stimulus-dependent processing of temporal order

Two distinct conceptualisations of processing mechanisms have been proposed in the research on the perception of temporal order, one that assumes a central-timing mechanism that is involved in the detection of temporal order independent of modality and stimulus type, another one assuming feature-specific mechanisms that are dependent on stimulus properties. In the present study, four different temporal-order judgement tasks were compared to test these two conceptualisations, that is, to determine whether common processes underlie temporal-order thresholds over different modalities and stimulus types or whether distinct processes are related to each task. Measurements varied regarding modality (visual and auditory) and stimulus properties (auditory modality: clicks and tones; visual modality: colour and position). Results indicate that the click and the tone paradigm, as well as the colour and position paradigm, correlate with each other. Besides these intra-modal relationships, cross-modal correlations show dependencies between the click, the colour and the position tasks. Both processing mechanisms seem to influence the detection of temporal order. While two different tones are integrated and processed by a more independent, possibly feature-specific mechanism, a more central, modality-independent timing mechanism contributes to the click, colour and position condition.     

Sensory modality and time perception in children and adults

This experiment investigated the effect of signal modality on time perception in 5- and 8-year-old children as well as young adults using a duration bisection task in which auditory and visual signals were presented in the same test session and shared common anchor durations. Durations were judged shorter for visual than for auditory signals by all age groups. However, the magnitude of this modality difference was larger in the children than in the adults. Sensitivity to time was also observed to increase with age for both modalities. Taken together, these two observations suggest that the greater modality effect on duration judgments for the children, for whom attentional abilities are considered limited, is the result of visual signals requiring more attentional resources than are needed for the processing of auditory signals. Within the framework of the information-processing model of Scalar Timing Theory, these effects are consistent with a developmental difference in the operation of the "attentional switch" used to transfer pulses from the pacemaker into the accumulator. Specifically, although timing is more automatic for auditory than visual signals in both children and young adults, children have greater difficulty in keeping the switch in the closed state during the timing of visual signals.     

The neuromagnetic dynamics of time perception

Examining real-time cortical dynamics is crucial for understanding time perception. Using magnetoencephalography we studied auditory duration discrimination of short (<.5 s) versus long tones (>.5 s) versus a pitch control. Time-frequency analysis of event-related fields showed widespread beta-band (13-30 Hz) desynchronization during all tone presentations. Synthetic aperture magnetometry indicated automatic primarily sensorimotor responses in short and pitch conditions, with activation specific to timing in bilateral inferior frontal gyrus. In the long condition, a right lateralized network was active, including lateral prefrontal cortices, inferior frontal gyrus, supramarginal gyrus and secondary auditory areas. Activation in this network peaked just after attention to tone duration was no longer necessary, suggesting a role in sustaining representation of the interval. These data expand our understanding of time perception by revealing its complex cortical spatiotemporal signature.     

Multi-Variate EEG Analysis as a Novel Tool to Examine Brain Responses to Naturalistic Music Stimuli

Note onsets in music are acoustic landmarks providing auditory cues that underlie the perception of more complex phenomena such as beat, rhythm, and meter. For naturalistic ongoing sounds a detailed view on the neural representation of onset structure is hard to obtain, since, typically, stimulus-related EEG signatures are derived by averaging a high number of identical stimulus presentations. Here, we propose a novel multivariate regression-based method extracting onset-related brain responses from the ongoing EEG. We analyse EEG recordings of nine subjects who passively listened to stimuli from various sound categories encompassing simple tone sequences, full-length romantic piano pieces and natural (non-music) soundscapes. The regression approach reduces the 61-channel EEG to one time course optimally reflecting note onsets. The neural signatures derived by this procedure indeed resemble canonical onset-related ERPs, such as the N1-P2 complex. This EEG projection was then utilized to determine the Cortico-Acoustic Correlation (CACor), a measure of synchronization between EEG signal and stimulus. We demonstrate that a significant CACor (i) can be detected in an individual listener''s EEG of a single presentation of a full-length complex naturalistic music stimulus, and (ii) it co-varies with the stimuli’s average magnitudes of sharpness, spectral centroid, and rhythmic complexity. In particular, the subset of stimuli eliciting a strong CACor also produces strongly coordinated tension ratings obtained from an independent listener group in a separate behavioral experiment. Thus musical features that lead to a marked physiological reflection of tone onsets also contribute to perceived tension in music.     

Perception of the emotional component of vocal signals at different durations of the stimulus

The effect of the temporal structure of a vocal stimulus on the perception of the emotional component of the signal was studied in several age groups (7–10, 11–13, and 14–17 years). The experiments were performed at different durations of the stimulus (0.5, 1, 1.5, 2, and 3 s). ANOVA of the recognition efficiency and response time showed that the stimulus duration and the interactions of this factor with two others (stimulus duration × age and stimulus duration × emotion type) were highly significant for the recognition of emotions. The effects of the temporal structure of the signal on the recognition efficiency and response time were the strongest in the cases of neutral and negative emotional intonations and on going from the youngest to the middle age group. The minimal stimulus duration at which the threshold recognition of the emotion type occurred changed with age (from 2 s in the youngest age group to 0.5 s in the oldest). The capacity of the sensory acoustic memory was evaluated in children and adolescents of different ages.     

Effect of contingent auditory stimuli on concurrent schedule performance: an alternative punisher to electric shock

This study explored whether load auditory stimuli could be used as functional punishing stimuli in place of electric shock. Three experiments examined the effect of a loud auditory stimulus on rats’ responding maintained by a concurrent reinforcement schedule. In Experiment 1, overall response rate decreased when a concurrent 1.5 s tone presentation schedule was superimposed on the concurrent variable interval (VI) 180-s, VI 180-s reinforcement schedule. On the contrary, response rate increased when a click presentation schedule was added. In Experiment 2, the extent of the response suppression with a 1.5 s tone presentation varied as a function of the frequency of the reinforcement schedule maintaining responses; the leaner the schedule employed, the greater the response suppression. In Experiment 3, response suppression was observed to be inversely related to the duration of the tone; response facilitation was observed when a 3.0-s tone was used. In Experiments 1 and 2, a preference shift towards the alternative with richer reinforcement was observed when the tone schedule was added. In contrast, the preference shifted towards the leaner alternative when the click or longer duration stimulus was used. These results imply that both the type and duration of a loud auditory stimulus, as well as the reinforcement schedule maintaining responses, have a critical role in determining the effect of the stimuli on responding. They also suggest that a loud auditory stimulus can be used as a positive punisher in a choice situation for rats, when the duration of the tone is brief, and the reinforcement schedule maintaining responses is lean.     

An auditory neural correlate suggests a mechanism underlying holistic pitch perception

Current theories of auditory pitch perception propose that cochlear place (spectral) and activity timing pattern (temporal) information are somehow combined within the brain to produce holistic pitch percepts, yet the neural mechanisms for integrating these two kinds of information remain obscure. To examine this process in more detail, stimuli made up of three pure tones whose components are individually resolved by the peripheral auditory system, but that nonetheless elicit a holistic, "missing fundamental" pitch percept, were played to human listeners. A technique was used to separate neural timing activity related to individual components of the tone complexes from timing activity related to an emergent feature of the complex (the envelope), and the region of the tonotopic map where information could originate from was simultaneously restricted by masking noise. Pitch percepts were mirrored to a very high degree by a simple combination of component-related and envelope-related neural responses with similar timing that originate within higher-frequency regions of the tonotopic map where stimulus components interact. These results suggest a coding scheme for holistic pitches whereby limited regions of the tonotopic map (spectral places) carrying envelope- and component-related activity with similar timing patterns selectively provide a key source of neural pitch information. A similar mechanism of integration between local and emergent object properties may contribute to holistic percepts in a variety of sensory systems.     

The role of superior temporal cortex in auditory timing

Recently, there has been upsurge of interest in the neural mechanisms of time perception. A central question is whether the representation of time is distributed over brain regions as a function of stimulus modality, task and length of the duration used or whether it is centralized in a single specific and supramodal network. The answers seem to be converging on the former, and many areas not primarily considered as temporal processing areas remain to be investigated in the temporal domain. Here we asked whether the superior temporal gyrus, an auditory modality specific area, is involved in processing of auditory timing. Repetitive transcranial magnetic stimulation was applied over left and right superior temporal gyri while participants performed either a temporal or a frequency discrimination task of single tones. A significant decrease in performance accuracy was observed after stimulation of the right superior temporal gyrus, in addition to an increase in response uncertainty as measured by the Just Noticeable Difference. The results are specific to auditory temporal processing and performance on the frequency task was not affected. Our results further support the idea of distributed temporal processing and speak in favor of the existence of modality specific temporal regions in the human brain.     

Mediation of timing accuracy by operant behavior

We attempted to demonstrate that timing performance on a temporal discrimination would be enhanced if rats were required to fill a duration with behavior than when they were not required to respond. Six rats were trained to discriminate between a 3- and 9-s stimulus in a symbolic-matching-to-sample task. In two conditions, a tone was used to signal the sample, and in the other two conditions, a light was used to signal the sample. In two conditions, the rats were required to respond on a lever mounted on the rear wall of the experimental chamber before making their discriminative response to one of the two levers mounted on the front wall of the experimental chamber. In the other two conditions, the rear lever was not presented during sample presentation, and no response was required. Consistent with our predictions, timing performance was significantly better when a lever-response was required during sample presentation than when no response was required.     

Cross-modal distortion of time perception: demerging the effects of observed and performed motion

Temporal information is often contained in multi-sensory stimuli, but it is currently unknown how the brain combines e.g. visual and auditory cues into a coherent percept of time. The existing studies of cross-modal time perception mainly support the "modality appropriateness hypothesis", i.e. the domination of auditory temporal cues over visual ones because of the higher precision of audition for time perception. However, these studies suffer from methodical problems and conflicting results. We introduce a novel experimental paradigm to examine cross-modal time perception by combining an auditory time perception task with a visually guided motor task, requiring participants to follow an elliptic movement on a screen with a robotic manipulandum. We find that subjective duration is distorted according to the speed of visually observed movement: The faster the visual motion, the longer the perceived duration. In contrast, the actual execution of the arm movement does not contribute to this effect, but impairs discrimination performance by dual-task interference. We also show that additional training of the motor task attenuates the interference, but does not affect the distortion of subjective duration. The study demonstrates direct influence of visual motion on auditory temporal representations, which is independent of attentional modulation. At the same time, it provides causal support for the notion that time perception and continuous motor timing rely on separate mechanisms, a proposal that was formerly supported by correlational evidence only. The results constitute a counterexample to the modality appropriateness hypothesis and are best explained by Bayesian integration of modality-specific temporal information into a centralized "temporal hub".     

Differential effects of empty and filled intervals on duration estimation by pigeons: tests of an attention-sharing explanation

Pigeons were trained in a within-subjects design to discriminate durations of an empty interval and a filled interval. Even when different stimuli were used to mark empty intervals and to signal filled intervals, pigeons judged empty intervals to be longer than equal-length filled intervals. This timing difference was not a result of pigeons timing marker duration on empty interval trials. Increasing marker duration did not produce an overestimation of the empty time intervals. It was suggested that this timing difference could be due to a reduction in attention to temporal processing on filled interval trials when visual stimuli are used. Consistent with this hypothesis, it was found that empty intervals were judged longer than filled intervals when testing occurred in a darkened test room, but not when the test room was illuminated. In addition, no timing difference was observed when different auditory stimuli were used as markers for empty intervals and as signals for filled intervals.     

Influence of Rhythmic Grouping on Duration Perception: A Novel Auditory Illusion

This study investigated a potential auditory illusion in duration perception induced by rhythmic temporal contexts. Listeners with or without musical training performed a duration discrimination task for a silent period in a rhythmic auditory sequence. The critical temporal interval was presented either within a perceptual group or between two perceptual groups. We report the just-noticeable difference (difference limen, DL) for temporal intervals and the point of subjective equality (PSE) derived from individual psychometric functions based on performance of a two-alternative forced choice task. In musically untrained individuals, equal temporal intervals were perceived as significantly longer when presented between perceptual groups than within a perceptual group (109.25% versus 102.5% of the standard duration). Only the perceived duration of the between-group interval was significantly longer than its objective duration. Musically trained individuals did not show this effect. However, in both musically trained and untrained individuals, the relative difference limens for discriminating the comparison interval from the standard interval were larger in the between-groups condition than in the within-group condition (7.3% vs. 5.6% of the standard duration). Thus, rhythmic grouping affected sensitivity to duration changes in all listeners, with duration differences being harder to detect at boundaries of rhythm groups than within rhythm groups. Our results show for the first time that temporal Gestalt induces auditory duration illusions in typical listeners, but that musical experts are not susceptible to this effect of rhythmic grouping.     

Sequential grouping of tone sequence as reflected by the mismatch negativity

The human sequential grouping that organizes parts of tones into a group was examined by the mismatch negativity (MMN), a component of event-related potentials that reveals the sensory memory process. The sequential grouping is accomplished by the combinations of some factors, e.g., temporal and frequency proximity principles. In this study, auditory oddball stimuli in which each of the stimuli consisted of series of tone bursts, were applied to the subjects, and the MMN elicited by the deviation of the frequency of the last tone in the stimulus was investigated. The relationship between the expected phenomena of sequential grouping of tones and observed magnitudes of MMN was evaluated. It was shown that the magnitudes of MMN changed according to the configuration (number of tones, frequency) of tone sequence to be stored. This result suggested that the sequential grouping of presented tones was achieved on the preattentive auditory sensory memory process. It was also shown that the relative change of MMN magnitudes corresponded to the conditions of sequential grouping, which had been proposed by the auditory psychophysical studies. The investigation of MMN properties could reveal the nature of auditory sequential grouping.This study was approved by the Ethics Committee on Clinical Investigation, Graduate School of Engineering, Tohoku University and was carried out in accordance with the policy of the Declaration of Helsinki.     

Memory strength and repetition suppression: multimodal imaging of medial temporal cortical contributions to recognition

Declarative memory permits an organism to recognize stimuli that have been previously encountered, discriminating them from those that are novel. One basis for recognition is item memory strength, which may support the perception of stimulus familiarity. Though the medial temporal lobes are known to be critical for declarative memory, at present the neural mechanisms supporting perceived differences in memory strength remain poorly specified. Here, functional MRI (fMRI) and anatomically constrained magnetoencephalography (MEG) indexed correlates of graded memory strength in the human brain, focusing on medial temporal cortex. fMRI revealed a decrease in medial temporal cortical activation that tracked parametric levels of perceived memory strength. Anatomically constrained MEG current estimates revealed that strength-dependent signal reductions onset within 150-300 ms. Memory strength appears to be rapidly signaled by medial temporal cortex through repetition suppression (activation reductions), providing a basis for the subjective perception of stimulus familiarity or novelty.