Perceptual organization of sound begins in the auditory periphery

Segmenting the complex acoustic mixture that makes a typical auditory scene into relevant perceptual objects is one of the main challenges of the auditory system [1], for both human and nonhuman species. Several recent studies indicate that perceptual auditory object formation, or "streaming," may be based on neural activity within the auditory cortex and beyond [2, 3]. Here, we find that scene analysis starts much earlier in the auditory pathways. Single units were recorded from a peripheral structure of the mammalian auditory brainstem, the cochlear nucleus. Peripheral responses were similar to cortical responses and displayed all of the functional properties required for streaming, including multisecond adaptation. Behavioral streaming was also measured in human listeners. Neurometric functions derived from the peripheral responses predicted accurately behavioral streaming. This reveals that subcortical structures may already contribute to the analysis of auditory scenes. This finding is consistent with the observation that species lacking a neocortex can still achieve and benefit from behavioral streaming [4]. For humans, we argue that auditory scene analysis of complex scenes is probably based on interactions between subcortical and cortical neural processes, with the relative contribution of each stage depending on the nature of the acoustic cues forming the streams.     

Detection of Appearing and Disappearing Objects in Complex Acoustic Scenes

The ability to detect sudden changes in the environment is critical for survival. Hearing is hypothesized to play a major role in this process by serving as an “early warning device,” rapidly directing attention to new events. Here, we investigate listeners'' sensitivity to changes in complex acoustic scenes—what makes certain events “pop-out” and grab attention while others remain unnoticed? We use artificial “scenes” populated by multiple pure-tone components, each with a unique frequency and amplitude modulation rate. Importantly, these scenes lack semantic attributes, which may have confounded previous studies, thus allowing us to probe low-level processes involved in auditory change perception. Our results reveal a striking difference between “appear” and “disappear” events. Listeners are remarkably tuned to object appearance: change detection and identification performance are at ceiling; response times are short, with little effect of scene-size, suggesting a pop-out process. In contrast, listeners have difficulty detecting disappearing objects, even in small scenes: performance rapidly deteriorates with growing scene-size; response times are slow, and even when change is detected, the changed component is rarely successfully identified. We also measured change detection performance when a noise or silent gap was inserted at the time of change or when the scene was interrupted by a distractor that occurred at the time of change but did not mask any scene elements. Gaps adversely affected the processing of item appearance but not disappearance. However, distractors reduced both appearance and disappearance detection. Together, our results suggest a role for neural adaptation and sensitivity to transients in the process of auditory change detection, similar to what has been demonstrated for visual change detection. Importantly, listeners consistently performed better for item addition (relative to deletion) across all scene interruptions used, suggesting a robust perceptual representation of item appearance.     

Neural processing of auditory looming in the human brain

Acoustic intensity change, along with interaural, spectral, and reverberation information, is an important cue for the perception of auditory motion. Approaching sound sources produce increases in intensity, and receding sound sources produce corresponding decreases. Human listeners typically overestimate increasing compared to equivalent decreasing sound intensity and underestimate the time to contact of approaching sound sources. These characteristics could provide a selective advantage by increasing the margin of safety for response to looming objects. Here, we used dynamic intensity and functional magnetic resonance imaging to examine the neural underpinnings of the perceptual priority for rising intensity. We found that, consistent with activation by horizontal and vertical auditory apparent motion paradigms, rising and falling intensity activated the right temporal plane more than constant intensity. Rising compared to falling intensity activated a distributed neural network subserving space recognition, auditory motion perception, and attention and comprising the superior temporal sulci and the middle temporal gyri, the right temporoparietal junction, the right motor and premotor cortices, the left cerebellar cortex, and a circumscribed region in the midbrain. This anisotropic processing of acoustic intensity change may reflect the salience of rising intensity produced by looming sources in natural environments.     

Auditory cortical contrast enhancing by global winner-take-all inhibitory interactions

Brains decompose the world into discrete objects of perception, thereby facing the problem of how to segregate and selectively address similar objects that are concurrently present in a scene. Theoretical models propose that this could be achieved by neuronal implementations of so-called winner-take-all algorithms where neuronal representations of objects or object features interact in a competitive manner. Here we present evidence for the existence of such a mechanism in an animal species. We present electrophysiological, neuropharmacological and neuroanatomical data which suggest a novel view of the role of GABA(A)-mediated inhibition in primary auditory cortex (AI), where intracortical GABA(A)-mediated inhibition operates on a global scale within a circular map of sound periodicity representation in AI, with functionally inhibitory projections of similar effect from any location throughout the whole map. These interactions could underlie the proposed competitive "winner-take-all" algorithm to support object segregation, e.g., segregation of different speakers in cocktail-party situations.     

Stochastic correlative firing for figure-ground segregation

Segregation of sensory inputs into separate objects is a central aspect of perception and arises in all sensory modalities. The figure-ground segregation problem requires identifying an object of interest in a complex scene, in many cases given binaural auditory or binocular visual observations. The computations required for visual and auditory figure-ground segregation share many common features and can be cast within a unified framework. Sensory perception can be viewed as a problem of optimizing information transmission. Here we suggest a stochastic correlative firing mechanism and an associative learning rule for figure-ground segregation in several classic sensory perception tasks, including the cocktail party problem in binaural hearing, binocular fusion of stereo images, and Gestalt grouping in motion perception.     

Cortical signatures of auditory object binding in children with autism spectrum disorder are anomalous in concordance with behavior and diagnosis

Organizing sensory information into coherent perceptual objects is fundamental to everyday perception and communication. In the visual domain, indirect evidence from cortical responses suggests that children with autism spectrum disorder (ASD) have anomalous figure–ground segregation. While auditory processing abnormalities are common in ASD, especially in environments with multiple sound sources, to date, the question of scene segregation in ASD has not been directly investigated in audition. Using magnetoencephalography, we measured cortical responses to unattended (passively experienced) auditory stimuli while parametrically manipulating the degree of temporal coherence that facilitates auditory figure–ground segregation. Results from 21 children with ASD (aged 7–17 years) and 26 age- and IQ-matched typically developing children provide evidence that children with ASD show anomalous growth of cortical neural responses with increasing temporal coherence of the auditory figure. The documented neurophysiological abnormalities did not depend on age, and were reflected both in the response evoked by changes in temporal coherence of the auditory scene and in the associated induced gamma rhythms. Furthermore, the individual neural measures were predictive of diagnosis (83% accuracy) and also correlated with behavioral measures of ASD severity and auditory processing abnormalities. These findings offer new insight into the neural mechanisms underlying auditory perceptual deficits and sensory overload in ASD, and suggest that temporal-coherence-based auditory scene analysis and suprathreshold processing of coherent auditory objects may be atypical in ASD.

To navigate everyday environments, the auditory system must analyze the temporal coherence of sound elements scattered across different frequencies to organize them into discrete perceptual objects. This neuroimaging study reveals that this process is altered in autism, potentially explaining the often-experienced sensory overload.     

Competing sound sources reveal spatial effects in cortical processing

Why is spatial tuning in auditory cortex weak, even though location is important to object recognition in natural settings? This question continues to vex neuroscientists focused on linking physiological results to auditory perception. Here we show that the spatial locations of simultaneous, competing sound sources dramatically influence how well neural spike trains recorded from the zebra finch field L (an analog of mammalian primary auditory cortex) encode source identity. We find that the location of a birdsong played in quiet has little effect on the fidelity of the neural encoding of the song. However, when the song is presented along with a masker, spatial effects are pronounced. For each spatial configuration, a subset of neurons encodes song identity more robustly than others. As a result, competing sources from different locations dominate responses of different neural subpopulations, helping to separate neural responses into independent representations. These results help elucidate how cortical processing exploits spatial information to provide a substrate for selective spatial auditory attention.     

Cortical motion deafness

The extent to which the auditory system, like the visual system, processes spatial stimulus characteristics such as location and motion in separate specialized neuronal modules or in one homogeneously distributed network is unresolved. Here we present a patient with a selective deficit for the perception and discrimination of auditory motion following resection of the right anterior temporal lobe and the right posterior superior temporal gyrus (STG). Analysis of stimulus identity and location within the auditory scene remained intact. In addition, intracranial auditory evoked potentials, recorded preoperatively, revealed motion-specific responses selectively over the resected right posterior STG, and electrical cortical stimulation of this region was experienced by the patient as incoming moving sounds. Collectively, these data present a patient with cortical motion deafness, providing evidence that cortical processing of auditory motion is performed in a specialized module within the posterior STG.     

Contextual Effects of Scene on the Visual Perception of Object Orientation in Depth

We investigated the effect of background scene on the human visual perception of depth orientation (i.e., azimuth angle) of three-dimensional common objects. Participants evaluated the depth orientation of objects. The objects were surrounded by scenes with an apparent axis of the global reference frame, such as a sidewalk scene. When a scene axis was slightly misaligned with the gaze line, object orientation perception was biased, as if the gaze line had been assimilated into the scene axis (Experiment 1). When the scene axis was slightly misaligned with the object, evaluated object orientation was biased, as if it had been assimilated into the scene axis (Experiment 2). This assimilation may be due to confusion between the orientation of the scene and object axes (Experiment 3). Thus, the global reference frame may influence object orientation perception when its orientation is similar to that of the gaze-line or object.     

Attribute Pair-Based Visual Recognition and Memory

Background

In the human visual system, different attributes of an object, such as shape, color, and motion, are processed separately in different areas of the brain. This raises a fundamental question of how are these attributes integrated to produce a unified perception and a specific response. This “binding problem” is computationally difficult because all attributes are assumed to be bound together to form a single object representation. However, there is no firm evidence to confirm that such representations exist for general objects.

Methodology/Principal Findings

Here we propose a paired-attribute model in which cognitive processes are based on multiple representations of paired attributes. In line with the model''s prediction, we found that multiattribute stimuli can produce an illusory perception of a multiattribute object arising from erroneous integration of attribute pairs, implying that object recognition is based on parallel perception of paired attributes. Moreover, in a change-detection task, a feature change in a single attribute frequently caused an illusory perception of change in another attribute, suggesting that multiple pairs of attributes are stored in memory.

Conclusions/Significance

The paired-attribute model can account for some novel illusions and controversial findings on binocular rivalry and short-term memory. Our results suggest that many cognitive processes are performed at the level of paired attributes rather than integrated objects, which greatly facilitates the binding problem and provides simpler solutions for it.     

Acoustic facilitation of object movement detection during self-motion

In humans, as well as most animal species, perception of object motion is critical to successful interaction with the surrounding environment. Yet, as the observer also moves, the retinal projections of the various motion components add to each other and extracting accurate object motion becomes computationally challenging. Recent psychophysical studies have demonstrated that observers use a flow-parsing mechanism to estimate and subtract self-motion from the optic flow field. We investigated whether concurrent acoustic cues for motion can facilitate visual flow parsing, thereby enhancing the detection of moving objects during simulated self-motion. Participants identified an object (the target) that moved either forward or backward within a visual scene containing nine identical textured objects simulating forward observer translation. We found that spatially co-localized, directionally congruent, moving auditory stimuli enhanced object motion detection. Interestingly, subjects who performed poorly on the visual-only task benefited more from the addition of moving auditory stimuli. When auditory stimuli were not co-localized to the visual target, improvements in detection rates were weak. Taken together, these results suggest that parsing object motion from self-motion-induced optic flow can operate on multisensory object representations.     

听觉选择性注意的可塑性及其神经机制研究

本文旨在研究音乐训练是否能增强基于音高和空间位置的听觉选择性注意力,以及音乐训练对听觉可塑性的神经机制.听觉感知实验中,受试者根据音高差异或空间位置差异,选择两个同时播放的数字之一.听觉认知实验在安静和噪声环境中播放频率分辨率不同的复合音,记录受试者听觉脑干频率跟随响应(frequency-following responses,FFRs).本文提出分析FFR的四种方法,即包络相关频率跟随响应(envelope-related frequency-following response,FFRENV)的短时锁相值、瞬时相位差极性图、相位差均值矢量以及时间细节结构相关频率跟随响应(temporal-fine-structure-related frequency-following response,FFRTFS)的幅度谱信噪比.实验结果表明,在完成基于音高的任务时,受过音乐训练的受试者准确率更高、反应时间更短.外界噪声不影响两组人群在基频(fundamental frequency,F0)的神经元锁相能力,但是显著降低了谐波处的神经元锁相能力.受过音乐训练的受试者的神经元在基频处的锁相能力和谐波处抗噪能力均增强,且其FFRTFS幅度谱信噪比与基于音高的行为学准确率呈正相关.因此,受过音乐训练的受试者其音高选择性注意感知能力的提高取决于认知神经能力的增强,经过音乐训练后,F0处FFRENV的锁相能力、谐波处FFRTFS的抗噪和持续锁相能力以及谐波处FFRTFS幅度谱信噪比均明显增强.音乐训练对听觉选择性注意具有显著的可塑性.     

Attention-memory interactions in scene perception

The perception of natural scenes relies on the integration of pre-existing knowledge with the immediate results of attentional processing, and what can be remembered from a scene depends in turn on how that scene is perceived and understood. However, there are conflicting results in the literature as to whether people are more likely to remember those objects that are consistent with the scene or those that are not. Moreover, whether any discrepancy between the likelihood of remembering schema-consistent or schema-inconsistent objects should be attributed to the schematic effects on attention or on memory remains unclear. To address this issue, the current study attempted to directly manipulate attention allocation by requiring participants to look at (i) schema-consistent objects, (ii) schema-inconsistent objects, or (iii) to share attention equally across both. Regardless of the differential allocation of attention or object fixation, schema-consistent objects were better recalled whereas recognition was independent of schema-consistency, but depended on task instruction. These results suggest that attention is important both for remembering low-level object properties, and information whose retrieval is not supported by the currently active schema. Specific knowledge of the scenes being viewed can result in the recall of non-fixated objects, but without such knowledge attention is required to encode sufficient detail for subsequent recognition. Our results demonstrate therefore that attention is not critical for the retrieval of objects that are consistent with a scene's schematic content.     

Visual saliency and spike timing in the ventral visual pathway.

Visual saliency is a fundamental yet hard to define property of objects or locations in the visual world. In a context where objects and their representations compete to dominate our perception, saliency can be thought of as the "juice" that makes objects win the race. It is often assumed that saliency is extracted and represented in an explicit saliency map, which serves to determine the location of spatial attention at any given time. It is then by drawing attention to a salient object that it can be recognized or categorized. I argue against this classical view that visual "bottom-up" saliency automatically recruits the attentional system prior to object recognition. A number of visual processing tasks are clearly performed too fast for such a costly strategy to be employed. Rather, visual attention could simply act by biasing a saliency-based object recognition system. Under natural conditions of stimulation, saliency can be represented implicitly throughout the ventral visual pathway, independent of any explicit saliency map. At any given level, the most activated cells of the neural population simply represent the most salient locations. The notion of saliency itself grows increasingly complex throughout the system, mostly based on luminance contrast until information reaches visual cortex, gradually incorporating information about features such as orientation or color in primary visual cortex and early extrastriate areas, and finally the identity and behavioral relevance of objects in temporal cortex and beyond. Under these conditions the object that dominates perception, i.e. the object yielding the strongest (or the first) selective neural response, is by definition the one whose features are most "salient"--without the need for any external saliency map. In addition, I suggest that such an implicit representation of saliency can be best encoded in the relative times of the first spikes fired in a given neuronal population. In accordance with our subjective experience that saliency and attention do not modify the appearance of objects, the feed-forward propagation of this first spike wave could serve to trigger saliency-based object recognition outside the realm of awareness, while conscious perceptions could be mediated by the remaining discharges of longer neuronal spike trains.     

Interaction between Attention and Bottom-Up Saliency Mediates the Representation of Foreground and Background in an Auditory Scene

The mechanism by which a complex auditory scene is parsed into coherent objects depends on poorly understood interactions between task-driven and stimulus-driven attentional processes. We illuminate these interactions in a simultaneous behavioral–neurophysiological study in which we manipulate participants' attention to different features of an auditory scene (with a regular target embedded in an irregular background). Our experimental results reveal that attention to the target, rather than to the background, correlates with a sustained (steady-state) increase in the measured neural target representation over the entire stimulus sequence, beyond auditory attention's well-known transient effects on onset responses. This enhancement, in both power and phase coherence, occurs exclusively at the frequency of the target rhythm, and is only revealed when contrasting two attentional states that direct participants' focus to different features of the acoustic stimulus. The enhancement originates in auditory cortex and covaries with both behavioral task and the bottom-up saliency of the target. Furthermore, the target's perceptual detectability improves over time, correlating strongly, within participants, with the target representation's neural buildup. These results have substantial implications for models of foreground/background organization, supporting a role of neuronal temporal synchrony in mediating auditory object formation.     

Adaptive vocal behavior drives perception by echolocation in bats

Echolocation operates through adaptive sensorimotor systems that collectively enable the bat to localize and track sonar objects as it flies. The features of sonar signals used by a bat to probe its surroundings determine the information available to its acoustic imaging system. In turn, the bat's perception of a complex scene guides its active adjustments in the features of subsequent sonar vocalizations. Here, we propose that the bat's active vocal-motor behaviors play directly into its representation of a dynamic auditory scene.     

Can Attention Be Confined to Just Part of a Moving Object? Revisiting Target-Distractor Merging in Multiple Object Tracking

While it was initially thought that attention was space-based, more recent work has shown that attention can also be object-based, in that observers find it easier to attend to different parts of the same object than to different parts of different objects. Such studies have shown that attention more easily spreads throughout an object than between objects. However, it is not known to what extent attention can be confined to just part of an object and to what extent attending to part of an object necessarily causes the entire object to be attended. We have investigated this question in the context of the multiple object tracking paradigm in which subjects are shown a scene containing a number of identical moving objects and asked to mentally track a subset of them, the targets, while not tracking the remainder, the distractors. Previous work has shown that joining each target to a distractor by a solid connector so that each target-distractor pair forms a single physical object, a technique known as target-distractor merging, makes it hard to track the targets, suggesting that attention cannot be restricted to just parts of objects. However, in that study the target-distractor pairs continuously changed length, which in itself would have made tracking difficult. Here we show that it remains difficult to track the targets even when the target-distractor pairs do not change length and even when the targets can be differentiated from the connectors that join them to the distractors. Our experiments suggest that it is hard to confine attention to just parts of objects, at least in the case of moving objects.     

Encoding of illusory continuity in primary auditory cortex

When interfering objects occlude a scene, the visual system restores the occluded information. Similarly, when a sound of interest (a "foreground" sound) is interrupted (occluded) by loud noise, the auditory system restores the occluded information. This process, called auditory induction, can be exploited to create a continuity illusion. When a segment of a foreground sound is deleted and loud noise fills the missing portion, listeners incorrectly report hearing the foreground continuing through the noise. Here we reveal the neurophysiological underpinnings of illusory continuity in single-neuron responses from awake macaque monkeys' primary auditory cortex (A1). A1 neurons represented the missing segment of occluded tonal foregrounds by responding to discontinuous foregrounds interrupted by intense noise as if they were responding to the complete foregrounds. By comparison, simulated peripheral responses represented only the noise and not the occluded foreground. The results reveal that many A1 single-neuron responses closely follow the illusory percept.     

Auditory motion aftereffects of approaching and withdrawing sound sources

Auditory motion aftereffects of approaching and withdrawing sound sources were investigated in the free field. The approaching and withdrawing of a sound source were simulated by means of differently directed changes in the amplitude of impulses of broadband noise (from 20 Hz to 20 kHz) through two loudspeakers placed 1.1 and 4.5 m away from the listener. Presentation of the adapting approaching and withdrawing stimuli changed the perception of test signals following them: a stationary test signal was perceived by listeners as moving in the direction opposite to one of the movement of the adapting stimulus, whereas a test stimulus slowly moving in same direction as the adapting signal was perceived as stationary. The specific features of the auditory aftereffect of signals moving in a radial direction were similar to those of sound sources moving in a horizontal plane.     

Multisensory Tracking of Objects in Darkness: Capture of Positive Afterimages by the Tactile and Proprioceptive Senses

This paper reports on three experiments investigating the contribution of different sensory modalities to the tracking of objects moved in total darkness. Participants sitting in the dark were exposed to a brief, bright flash which reliably induced a positive visual afterimage of the scene so illuminated. If the participants subsequently move their hand in the darkness, the visual afterimage of that hand fades or disappears; this is presumably due to conflict between the illusory visual afterimage (of the hand in its original location) and other information (e.g., proprioceptive) from a general mechanism for tracking body parts. This afterimage disappearance effect also occurs for held objects which are moved in the dark, and some have argued that this represents a case of body schema extension, i.e. the rapid incorporation of held external objects into the body schema. We demonstrate that the phenomenon is not limited to held objects and occurs in conditions where incorporation into the body schema is unlikely. Instead, we propose that the disappearance of afterimages of objects moved in darkness comes from a general mechanism for object tracking which integrates input from multiple sensory systems. This mechanism need not be limited to tracking body parts, and thus we need not invoke body schema extension to explain the afterimage disappearance. In this series of experiments, we test whether auditory feedback of object movement can induce afterimage disappearance, demonstrate that the disappearance effect scales with the magnitude of proprioceptive feedback, and show that tactile feedback alone is sufficient for the effect. Together, these data demonstrate that the visual percept of a positive afterimage is constructed not just from visual input of the scene when light reaches the eyes, but in conjunction with input from multiple other senses.