Competing neural responses for auditory and visual decisions

Why is it hard to divide attention between dissimilar activities, such as reading and listening to a conversation? We used functional magnetic resonance imaging (fMRI) to study interference between simple auditory and visual decisions, independently of motor competition. Overlapping activity for auditory and visual tasks performed in isolation was found in lateral prefrontal regions, middle temporal cortex and parietal cortex. When the visual stimulus occurred during the processing of the tone, its activation in prefrontal and middle temporal cortex was suppressed. Additionally, reduced activity was seen in modality-specific visual cortex. These results paralleled impaired awareness of the visual event. Even without competing motor responses, a simple auditory decision interferes with visual processing on different neural levels, including prefrontal cortex, middle temporal cortex and visual regions.     

An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory Cortex

Research on the neural basis of speech-reading implicates a network of auditory language regions involving inferior frontal cortex, premotor cortex and sites along superior temporal cortex. In audiovisual speech studies, neural activity is consistently reported in posterior superior temporal Sulcus (pSTS) and this site has been implicated in multimodal integration. Traditionally, multisensory interactions are considered high-level processing that engages heteromodal association cortices (such as STS). Recent work, however, challenges this notion and suggests that multisensory interactions may occur in low-level unimodal sensory cortices. While previous audiovisual speech studies demonstrate that high-level multisensory interactions occur in pSTS, what remains unclear is how early in the processing hierarchy these multisensory interactions may occur. The goal of the present fMRI experiment is to investigate how visual speech can influence activity in auditory cortex above and beyond its response to auditory speech. In an audiovisual speech experiment, subjects were presented with auditory speech with and without congruent visual input. Holding the auditory stimulus constant across the experiment, we investigated how the addition of visual speech influences activity in auditory cortex. We demonstrate that congruent visual speech increases the activity in auditory cortex.     

Negative BOLD differentiates visual imagery and perception

Recent studies emphasize the overlap between the neural substrates of visual perception and visual imagery. However, the subjective experiences of imagining and seeing are clearly different. Here we demonstrate that deactivation of auditory cortex (and to some extent of somatosensory and subcortical visual structures) as measured by BOLD functional magnetic resonance imaging unequivocally differentiates visual imagery from visual perception. During visual imagery, auditory cortex deactivation negatively correlates with activation in visual cortex and with the score in the subjective vividness of visual imagery questionnaire (VVIQ). Perception of the world requires the merging of multisensory information so that, during seeing, information from other sensory systems modifies visual cortical activity and shapes experience. We suggest that pure visual imagery corresponds to the isolated activation of visual cortical areas with concurrent deactivation of "irrelevant" sensory processing that could disrupt the image created by our "mind's eye."     

Extensive cochleotopic mapping of human auditory cortical fields obtained with phase-encoding FMRI

The primary sensory cortices are characterized by a topographical mapping of basic sensory features which is considered to deteriorate in higher-order areas in favor of complex sensory features. Recently, however, retinotopic maps were also discovered in the higher-order visual, parietal and prefrontal cortices. The discovery of these maps enabled the distinction between visual regions, clarified their function and hierarchical processing. Could such extension of topographical mapping to high-order processing regions apply to the auditory modality as well? This question has been studied previously in animal models but only sporadically in humans, whose anatomical and functional organization may differ from that of animals (e.g. unique verbal functions and Heschl''s gyrus curvature). Here we applied fMRI spectral analysis to investigate the cochleotopic organization of the human cerebral cortex. We found multiple mirror-symmetric novel cochleotopic maps covering most of the core and high-order human auditory cortex, including regions considered non-cochleotopic, stretching all the way to the superior temporal sulcus. These maps suggest that topographical mapping persists well beyond the auditory core and belt, and that the mirror-symmetry of topographical preferences may be a fundamental principle across sensory modalities.     

Posterior Parietal Cortex Drives Inferotemporal Activations During Three-Dimensional Object Vision

The primate visual system consists of a ventral stream, specialized for object recognition, and a dorsal visual stream, which is crucial for spatial vision and actions. However, little is known about the interactions and information flow between these two streams. We investigated these interactions within the network processing three-dimensional (3D) object information, comprising both the dorsal and ventral stream. Reversible inactivation of the macaque caudal intraparietal area (CIP) during functional magnetic resonance imaging (fMRI) reduced fMRI activations in posterior parietal cortex in the dorsal stream and, surprisingly, also in the inferotemporal cortex (ITC) in the ventral visual stream. Moreover, CIP inactivation caused a perceptual deficit in a depth-structure categorization task. CIP-microstimulation during fMRI further suggests that CIP projects via posterior parietal areas to the ITC in the ventral stream. To our knowledge, these results provide the first causal evidence for the flow of visual 3D information from the dorsal stream to the ventral stream, and identify CIP as a key area for depth-structure processing. Thus, combining reversible inactivation and electrical microstimulation during fMRI provides a detailed view of the functional interactions between the two visual processing streams.     

听觉皮层信号处理

听觉系统和视觉系统的不同之处在于：听觉系统在外周感受器和听皮层间具有更长的皮层下通路和更多的突触联系。该特殊结构反应了听觉系统从复杂听觉环境中提取与行为相关信号的机制与其他感觉系统不同。听皮层神经信号处理包括两种重要的转换机制,声音信号的非同构转换以及从声音感受到知觉层面的转换。听觉皮层神经编码机制同时也受到听觉反馈和语言或发声过程中发声信号的调控。听觉神经科学家和生物医学工程师所面临的挑战便是如何去理解大脑中这些转换的编码机制。我将会用我实验室最近的一些发现来阐述听觉信号是如何在原听皮层中进行处理的,并讨论其对于言语和音乐在大脑中的处理机制以及设计神经替代装置诸如电子耳蜗的意义。我们使用了结合神经电生理技术和量化工程学的方法来研究这些问题。     

Neural basis of the ventriloquist illusion

The ventriloquist creates the illusion that his or her voice emerges from the visibly moving mouth of the puppet [1]. This well-known illusion exemplifies a basic principle of how auditory and visual information is integrated in the brain to form a unified multimodal percept. When auditory and visual stimuli occur simultaneously at different locations, the more spatially precise visual information dominates the perceived location of the multimodal event. Previous studies have examined neural interactions between spatially disparate auditory and visual stimuli [2-5], but none has found evidence for a visual influence on the auditory cortex that could be directly linked to the illusion of a shifted auditory percept. Here we utilized event-related brain potentials combined with event-related functional magnetic resonance imaging to demonstrate on a trial-by-trial basis that a precisely timed biasing of the left-right balance of auditory cortex activity by the discrepant visual input underlies the ventriloquist illusion. This cortical biasing may reflect a fundamental mechanism for integrating the auditory and visual components of environmental events, which ensures that the sounds are adaptively localized to the more reliable position provided by the visual input.     

Unveiling functions of the visual cortex using task-specific deep neural networks

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.     

Preserved functional specialization for spatial processing in the middle occipital gyrus of the early blind

The occipital cortex (OC) of early-blind humans is activated during various nonvisual perceptual and cognitive tasks, but little is known about its modular organization. Using functional MRI we tested whether processing of auditory versus tactile and spatial versus nonspatial information was dissociated in the OC of the early blind. No modality-specific OC activation was observed. However, the right middle occipital gyrus (MOG) showed a preference for spatial over nonspatial processing of both auditory and tactile stimuli. Furthermore, MOG activity was correlated with accuracy of individual sound localization performance. In sighted controls, most of extrastriate OC, including the MOG, was deactivated during auditory and tactile conditions, but the right MOG was more activated during spatial than nonspatial visual tasks. Thus, although the sensory modalities driving the neurons in the reorganized OC of blind individuals are altered, the functional specialization of extrastriate cortex is retained regardless of visual experience.     

The Development of Visual Areas Depends Differently on Visual Experience

Visual experience plays an important role in the development of the visual cortex; however, recent functional imaging studies have shown that the functional organization is preserved in several higher-tier visual areas in congenitally blind subjects, indicating that maturation of visual areas depend unequally on visual experience. In this study, we aim to validate this hypothesis using a multimodality MRI approach. We found increased cortical thickness in the congenitally blind was present in the early visual areas and absent in the higher-tier ones, suggesting that the structural development of the visual cortex depends hierarchically on visual experience. In congenitally blind subjects, the decreased resting-state functional connectivity with the primary somatosensory cortex was more prominent in the early visual areas than in the higher-tier ones and were more pronounced in the ventral stream than in the dorsal one, suggesting that the development of functional organization of the visual cortex also depends differently on visual experience. Moreover, congenitally blind subjects showed normal or increased functional connectivity between ipsilateral higher-tier and early visual areas, suggesting an indirect corticocortical pathway through which somatosenroy information can reach the early visual areas. These findings support our hypothesis that the development of visual areas depends differently on visual experience.     

A cognitive set and top-down processes in the functional organization of visual recognition

A top-down processing model and the role of frontal and prefrontal cortex in the neural mechanisms of recognition of verbal and non-verbal visual stimuli, are discussed. Stimulus recognition is considered to be a result of integration of the bottom-up and top-down streams. The cognitive set phenomenon can be used for studying the downward control in the neurocognitive visual system and in functional organisation of stimuli recognition.     

Functional specialization of seven mouse visual cortical areas

To establish the mouse as a genetically tractable model for high-order visual processing, we characterized fine-scale retinotopic organization of visual cortex and determined functional specialization of layer 2/3 neuronal populations in seven retinotopically identified areas. Each area contains a distinct visuotopic representation and encodes a unique combination of spatiotemporal features. Areas LM, AL, RL, and AM prefer up to three times faster temporal frequencies and significantly lower spatial frequencies than V1, while V1 and PM prefer high spatial and low temporal frequencies. LI prefers both high spatial and temporal frequencies. All extrastriate areas except LI increase orientation selectivity compared to V1, and three areas are significantly more direction selective (AL, RL, and AM). Specific combinations of spatiotemporal representations further distinguish areas. These results reveal that mouse higher visual areas are functionally distinct, and separate groups of areas may be specialized for motion-related versus pattern-related computations, perhaps forming pathways analogous to dorsal and ventral streams in other species.     

See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex

Our understanding of multisensory integration has advanced because of recent functional neuroimaging studies of three areas in human lateral occipito-temporal cortex: superior temporal sulcus, area LO and area MT (V5). Superior temporal sulcus is activated strongly in response to meaningful auditory and visual stimuli, but responses to tactile stimuli have not been well studied. Area LO shows strong activation in response to both visual and tactile shape information, but not to auditory representations of objects. Area MT, an important region for processing visual motion, also shows weak activation in response to tactile motion, and a signal that drops below resting baseline in response to auditory motion. Within superior temporal sulcus, a patchy organization of regions is activated in response to auditory, visual and multisensory stimuli. This organization appears similar to that observed in polysensory areas in macaque superior temporal sulcus, suggesting that it is an anatomical substrate for multisensory integration. A patchy organization might also be a neural mechanism for integrating disparate representations within individual sensory modalities, such as representations of visual form and visual motion.     

Action insight: the role of the dorsal stream in the perception of grasping

In this issue of Neuron, Shmuelof and Zohary use functional magnetic resonance imaging (fMRI) to demonstrate differential sensitivity of the ventral and dorsal cortical streams of visual processing to images of objects and grasping hands, respectively.     

Geometry and the visual brain.

Receptive fields structure of neurons in primary visual cortex suggests that they process visual stimuli in the frequency domain, in a way similar to the frequency analysis performed in the auditory system. As a consequence, both psychophysicists and electrophysiologists have long probed the visual system using extended sine wave gratings that are well localized in the frequency domain but poorly defined in visual space. Meanwhile, how the brain processes the geometrical properties and the spatial and temporal relationships between stimulus parts has received less attention. Recent progress in visual neuroscience that uncovered long-range horizontal connections between cortical neurons and revealed the complex architecture of primary visual cortex and feedback connectivity led to new insights concerned with the processing of geometrical properties of visual stimuli in V1. This paper presents a short historical perspective of the emergence of new issues related to the cortical architecture and its functional consequences on the processing of geometrical properties.     

Auditory Attention Activates Peripheral Visual Cortex

Background

Recent neuroimaging studies have revealed that putatively unimodal regions of visual cortex can be activated during auditory tasks in sighted as well as in blind subjects. However, the task determinants and functional significance of auditory occipital activations (AOAs) remains unclear.

Methodology/Principal Findings

We examined AOAs in an intermodal selective attention task to distinguish whether they were stimulus-bound or recruited by higher-level cognitive operations associated with auditory attention. Cortical surface mapping showed that auditory occipital activations were localized to retinotopic visual cortex subserving the far peripheral visual field. AOAs depended strictly on the sustained engagement of auditory attention and were enhanced in more difficult listening conditions. In contrast, unattended sounds produced no AOAs regardless of their intensity, spatial location, or frequency.

Conclusions/Significance

Auditory attention, but not passive exposure to sounds, routinely activated peripheral regions of visual cortex when subjects attended to sound sources outside the visual field. Functional connections between auditory cortex and visual cortex subserving the peripheral visual field appear to underlie the generation of AOAs, which may reflect the priming of visual regions to process soon-to-appear objects associated with unseen sound sources.     

Overview of Sensory Systems of <Emphasis Type="Italic">Tarsius</Emphasis>

Tarsiers form the sister taxon to anthropoid primates, and their brains possess a mix of primitive and specialized features. We describe architectonically distinct subdivisions of the somatosensory, auditory, and visual systems for tarsiers, as well as nocturnal New World owl monkeys (Aotus) and strepsirhine galagos (Otolemur) for comparison. In general, the dorsal column nuclei, the ventroposterior nucleus, and primary somatosensory cortex are somewhat less distinctly differentiated in tarsiers, suggesting that the somatosensory system is less specialized for somatosensory processing. Although the inferior colliculus and the medial geniculate complex of the auditory system are architectonically similar across the 3 primates, the primary auditory cortex of tarsiers is more distinct, suggesting a greater role in auditory cortical processing. In the visual system, the differentiation of the superior colliculus is similar in all 3 primates, whereas the laminar pattern in the lateral geniculate nucleus and the subdivisions of the inferior pulvinar in tarsiers resemble those of anthropoid primates rather than strepsirhines, in agreement with the evidence that tarsiers form the sister clade for anthropoids. In addition, primary visual cortex has more distinct sublayers in tarsiers than other primates, attesting to its importance in this visual predator. Overall, tarsiers have well developed visual and auditory systems, and a less well developed somatosensory system, suggesting an enhanced reliance on the visual and auditory senses, rather than somatosensory sense.     

Sensory systems: auditory action streams?

The visual system is suggested to have two main processing streams, dorsal and ventral, the former being an 'action stream' concerned with motor responses, as opposed to perception. Two recent studies suggest the existence of a comparable mechanism in the auditory system.     

Interactive coding of visual spatial frequency and auditory amplitude-modulation rate

Spatial frequency is a fundamental visual feature coded in primary visual cortex, relevant for perceiving textures, objects, hierarchical structures, and scenes, as well as for directing attention and eye movements. Temporal amplitude-modulation (AM) rate is a fundamental auditory feature coded in primary auditory cortex, relevant for perceiving auditory objects, scenes, and speech. Spatial frequency and temporal AM rate are thus fundamental building blocks of visual and auditory perception. Recent results suggest that crossmodal interactions are commonplace across the primary sensory cortices and that some of the underlying neural associations develop through consistent multisensory experience such as audio-visually perceiving speech, gender, and objects. We demonstrate that people consistently and absolutely (rather than relatively) match specific auditory AM rates to specific visual spatial frequencies. We further demonstrate that this crossmodal mapping allows amplitude-modulated sounds to guide attention to and modulate awareness of specific visual spatial frequencies. Additional results show that the crossmodal association is approximately linear, based on physical spatial frequency, and generalizes to tactile pulses, suggesting that the association develops through multisensory experience during manual exploration of surfaces.     

Functional imaging reveals numerous fields in the monkey auditory cortex

Anatomical studies propose that the primate auditory cortex contains more fields than have actually been functionally confirmed or described. Spatially resolved functional magnetic resonance imaging (fMRI) with carefully designed acoustical stimulation could be ideally suited to extend our understanding of the processing within these fields. However, after numerous experiments in humans, many auditory fields remain poorly characterized. Imaging the macaque monkey is of particular interest as these species have a richer set of anatomical and neurophysiological data to clarify the source of the imaged activity. We functionally mapped the auditory cortex of behaving and of anesthetized macaque monkeys with high resolution fMRI. By optimizing our imaging and stimulation procedures, we obtained robust activity throughout auditory cortex using tonal and band-passed noise sounds. Then, by varying the frequency content of the sounds, spatially specific activity patterns were observed over this region. As a result, the activity patterns could be assigned to many auditory cortical fields, including those whose functional properties were previously undescribed. The results provide an extensive functional tessellation of the macaque auditory cortex and suggest that 11 fields contain neurons tuned for the frequency of sounds. This study provides functional support for a model where three fields in primary auditory cortex are surrounded by eight neighboring “belt” fields in non-primary auditory cortex. The findings can now guide neurophysiological recordings in the monkey to expand our understanding of the processing within these fields. Additionally, this work will improve fMRI investigations of the human auditory cortex.