Multisensory Oddity Detection as Bayesian Inference

A key goal for the perceptual system is to optimally combine information from all the senses that may be available in order to develop the most accurate and unified picture possible of the outside world. The contemporary theoretical framework of ideal observer maximum likelihood integration (MLI) has been highly successful in modelling how the human brain combines information from a variety of different sensory modalities. However, in various recent experiments involving multisensory stimuli of uncertain correspondence, MLI breaks down as a successful model of sensory combination. Within the paradigm of direct stimulus estimation, perceptual models which use Bayesian inference to resolve correspondence have recently been shown to generalize successfully to these cases where MLI fails. This approach has been known variously as model inference, causal inference or structure inference. In this paper, we examine causal uncertainty in another important class of multi-sensory perception paradigm – that of oddity detection and demonstrate how a Bayesian ideal observer also treats oddity detection as a structure inference problem. We validate this approach by showing that it provides an intuitive and quantitative explanation of an important pair of multi-sensory oddity detection experiments – involving cues across and within modalities – for which MLI previously failed dramatically, allowing a novel unifying treatment of within and cross modal multisensory perception. Our successful application of structure inference models to the new ‘oddity detection’ paradigm, and the resultant unified explanation of across and within modality cases provide further evidence to suggest that structure inference may be a commonly evolved principle for combining perceptual information in the brain.     

Multisensory integration and the body schema: close to hand and within reach

There has been a recent and dramatic growth of interest in the psychological and neural mechanisms of multisensory integration between different sensory modalities. Much of this recent research has focused specifically on how multisensory representations of body parts and of the 'peripersonal' space immediately around them, are constructed. Research has also focused on how this may lead to multisensorially determined perceptions of body parts, to action execution, and even to attributions of agency and self-ownership for the body parts in question. Converging evidence from animal and human studies suggests that the primate brain constructs various body-part-centred representations of space, based on the integration of visual, tactile and proprioceptive information. These representations can plastically change following active tool-use that extends reachable space and also modifies the representation of peripersonal space. These new results indicate that a modern cognitive neuroscience approach to the classical concept of the 'body schema' may now be within reach.     

Multisensory integration of colors and scents: insights from bees and flowers

Karl von Frisch’s studies of bees’ color vision and chemical senses opened a window into the perceptual world of a species other than our own. A century of subsequent research on bees’ visual and olfactory systems has developed along two productive but independent trajectories, leaving the questions of how and why bees use these two senses in concert largely unexplored. Given current interest in multimodal communication and recently discovered interplay between olfaction and vision in humans and Drosophila, understanding multisensory integration in bees is an opportunity to advance knowledge across fields. Using a classic ethological framework, we formulate proximate and ultimate perspectives on bees’ use of multisensory stimuli. We discuss interactions between scent and color in the context of bee cognition and perception, focusing on mechanistic and functional approaches, and we highlight opportunities to further explore the development and evolution of multisensory integration. We argue that although the visual and olfactory worlds of bees are perhaps the best-studied of any non-human species, research focusing on the interactions between these two sensory modalities is vitally needed.     

Multisensory Integration and Behavioral Plasticity in Sharks from Different Ecological Niches

The underwater sensory world and the sensory systems of aquatic animals have become better understood in recent decades, but typically have been studied one sense at a time. A comprehensive analysis of multisensory interactions during complex behavioral tasks has remained a subject of discussion without experimental evidence. We set out to generate a general model of multisensory information extraction by aquatic animals. For our model we chose to analyze the hierarchical, integrative, and sometimes alternate use of various sensory systems during the feeding sequence in three species of sharks that differ in sensory anatomy and behavioral ecology. By blocking senses in different combinations, we show that when some of their normal sensory cues were unavailable, sharks were often still capable of successfully detecting, tracking and capturing prey by switching to alternate sensory modalities. While there were significant species differences, odor was generally the first signal detected, leading to upstream swimming and wake tracking. Closer to the prey, as more sensory cues became available, the preferred sensory modalities varied among species, with vision, hydrodynamic imaging, electroreception, and touch being important for orienting to, striking at, and capturing the prey. Experimental deprivation of senses showed how sharks exploit the many signals that comprise their sensory world, each sense coming into play as they provide more accurate information during the behavioral sequence of hunting. The results may be applicable to aquatic hunting in general and, with appropriate modification, to other types of animal behavior.     

Behavioral and neuroplastic changes in the blind: evidence for functionally relevant cross-modal interactions.

The study of blind individuals provides insight into the brain re-organization and behavioral compensations that occur following sensory deprivation. While behavioral studies have yielded conflicting results in terms of performance levels within the remaining senses, deafferentation of visual cortical areas through peripheral blindness results in clear neuroplastic changes. Most striking is the activation of occipital cortex in response to auditory and tactile stimulation. Indeed, parts of the "unimodal" visual cortex are recruited by other sensory modalities to process sensory information in a functionally relevant manner. In addition, a larger area of the sensorimotor cortex is devoted to the representation of the reading finger in blind Braille readers. The "visual" function of the deafferented occipital cortex is also altered, where transcranial magnetic stimulation-induced phosphenes can be elicited in only 20% of blind subjects. The neural mechanisms underlying these changes remain elusive but recent data showing rapid cross-modal plasticity in blindfolded, sighted subjects argue against the establishment of new connections to explain cross-modal interactions in the blind. Rather, latent pathways that participate in multisensory percepts in sighted subjects might be unmasked and may be potentiated in the event of complete loss of visual input. These issues have important implications for the development of visual prosthesis aimed at restoring some degree of vision in the blind.     

Multisensory Causal Inference in the Brain

At any given moment, our brain processes multiple inputs from its different sensory modalities (vision, hearing, touch, etc.). In deciphering this array of sensory information, the brain has to solve two problems: (1) which of the inputs originate from the same object and should be integrated and (2) for the sensations originating from the same object, how best to integrate them. Recent behavioural studies suggest that the human brain solves these problems using optimal probabilistic inference, known as Bayesian causal inference. However, how and where the underlying computations are carried out in the brain have remained unknown. By combining neuroimaging-based decoding techniques and computational modelling of behavioural data, a new study now sheds light on how multisensory causal inference maps onto specific brain areas. The results suggest that the complexity of neural computations increases along the visual hierarchy and link specific components of the causal inference process with specific visual and parietal regions.     

Functional neuroimaging

Functional neuroimaging represents an area of brain imaging that has undergone tremendous advancements in the last decade. It is now possible to design experiments that elucidate the functional interplay between brain regions that give rise to specific human cognitive processes. Positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) form the core technologies that have allowed such studies. This article reviews the basis of these techniques, their strengths and limitations, the underlying neurophysiology, and the future of functional neuroimaging.     

An oscillatory neural network model that demonstrates the benefits of multisensory learning

Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.     

Integration of bimodal looming signals through neuronal coherence in the temporal lobe

The ability to integrate information across multiple sensory systems offers several behavioral advantages, from quicker reaction times and more accurate responses to better detection and more robust learning. At the neural level, multisensory integration requires large-scale interactions between different brain regions--the convergence of information from separate sensory modalities, represented by distinct neuronal populations. The interactions between these neuronal populations must be fast and flexible, so that behaviorally relevant signals belonging to the same object or event can be immediately integrated and integration of unrelated signals can be prevented. Looming signals are a particular class of signals that are behaviorally relevant for animals and that occur in both the auditory and visual domain. These signals indicate the rapid approach of objects and provide highly salient warning cues about impending impact. We show here that multisensory integration of auditory and visual looming signals may be mediated by functional interactions between auditory cortex and the superior temporal sulcus, two areas involved in integrating behaviorally relevant auditory-visual signals. Audiovisual looming signals elicited increased gamma-band coherence between these areas, relative to unimodal or receding-motion signals. This suggests that the neocortex uses fast, flexible intercortical interactions to mediate multisensory integration.     

Multisensory contributions to low-level, 'unisensory' processing

Neurobiologists have traditionally assumed that multisensory integration is a higher order process that occurs after sensory signals have undergone extensive processing through a hierarchy of unisensory subcortical and cortical regions. Recent findings, however, question this assumption. Studies in humans, nonhuman primates and other species demonstrate multisensory convergence in low level cortical structures that were generally believed to be unisensory in function. In addition to enriching current models of multisensory processing and perceptual functions, these new findings require a revision in our thinking about unisensory processing in low level cortical areas.     

Multisensory integration: methodological approaches and emerging principles in the human brain.

Understanding the conditions under which the brain integrates the different sensory streams and the mechanisms supporting this phenomenon is now a question at the forefront of neuroscience. In this paper, we discuss the opportunities for investigating these multisensory processes using modern imaging techniques, the nature of the information obtainable from each method and their benefits and limitations. Despite considerable variability in terms of paradigm design and analysis, some consistent findings are beginning to emerge. The detection of brain activity in human neuroimaging studies that resembles multisensory integration responses at the cellular level in other species, suggests similar crossmodal binding mechanisms may be operational in the human brain. These mechanisms appear to be distributed across distinct neuronal networks that vary depending on the nature of the shared information between different sensory cues. For example, differing extents of correspondence in time, space or content seem to reliably bias the involvement of different integrative networks which code for these cues. A combination of data obtained from haemodynamic and electromagnetic methods, which offer high spatial or temporal resolution respectively, are providing converging evidence of multisensory interactions at both "early" and "late" stages of processing--suggesting a cascade of synergistic processes operating in parallel at different levels of the cortex.     

Brain–computer interfaces for dissecting cognitive processes underlying sensorimotor control

Sensorimotor control engages cognitive processes such as prediction, learning, and multisensory integration. Understanding the neural mechanisms underlying these cognitive processes with arm reaching is challenging because we currently record only a fraction of the relevant neurons, the arm has nonlinear dynamics, and multiple modalities of sensory feedback contribute to control. A brain–computer interface (BCI) is a well-defined sensorimotor loop with key simplifying advantages that address each of these challenges, while engaging similar cognitive processes. As a result, BCI is becoming recognized as a powerful tool for basic scientific studies of sensorimotor control. Here, we describe the benefits of BCI for basic scientific inquiries and review recent BCI studies that have uncovered new insights into the neural mechanisms underlying sensorimotor control.     

Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex

BACKGROUND: Integrating information from the different senses markedly enhances the detection and identification of external stimuli. Compared with unimodal inputs, semantically and/or spatially congruent multisensory cues speed discrimination and improve reaction times. Discordant inputs have the opposite effect, reducing performance and slowing responses. These behavioural features of crossmodal processing appear to have parallels in the response properties of multisensory cells in the superior colliculi and cerebral cortex of non-human mammals. Although spatially concordant multisensory inputs can produce a dramatic, often multiplicative, increase in cellular activity, spatially disparate cues tend to induce a profound response depression. RESULTS: Using functional magnetic resonance imaging (fMRI), we investigated whether similar indices of crossmodal integration are detectable in human cerebral cortex, and for the synthesis of complex inputs relating to stimulus identity. Ten human subjects were exposed to varying epochs of semantically congruent and incongruent audio-visual speech and to each modality in isolation. Brain activations to matched and mismatched audio-visual inputs were contrasted with the combined response to both unimodal conditions. This strategy identified an area of heteromodal cortex in the left superior temporal sulcus that exhibited significant supra-additive response enhancement to matched audio-visual inputs and a corresponding sub-additive response to mismatched inputs. CONCLUSIONS: The data provide fMRI evidence of crossmodal binding by convergence in the human heteromodal cortex. They further suggest that response enhancement and depression may be a general property of multisensory integration operating at different levels of the neuroaxis and irrespective of the purpose for which sensory inputs are combined.     

<Emphasis Type="Italic">Umwelträume</Emphasis> and Multisensory Integration. Mirror Perspectives on the Subject–Object Dichotomy

This paper concerns epistemic developments in the field of sensory perception. I argue that Uexküll’s concept of the Umwelträume and certain principles of multisensory integration explain and describe in similar terms the manner in which different sensory modalities interact. Indeed, they both concern knowledge, describing in spatial terms how the mind makes itself up, makes up its objects, and how the objects, in turn, make up the mind. My intention is to set side by side these two trends of thought in order to mutually explain them. I suggest that there is some sort of dialectic at work between the Umwelträume and multisensory integration: in the first case interaction between the senses results in the creation of high definition subjects, whereas in the latter the interaction of the senses results in high definition objects.     

Bayesian multisensory integration and cross-modal spatial links.

Our perception of the word is the result of combining information between several senses, such as vision, audition and proprioception. These sensory modalities use widely different frames of reference to represent the properties and locations of object. Moreover, multisensory cues come with different degrees of reliability, and the reliability of a given cue can change in different contexts. The Bayesian framework--which we describe in this review--provides an optimal solution to deal with this issue of combining cues that are not equally reliable. However, this approach does not address the issue of frames of references. We show that this problem can be solved by creating cross-modal spatial links in basis function networks. Finally, we show how the basis function approach can be combined with the Bayesian framework to yield networks that can perform optimal multisensory combination. On the basis of this theory, we argue that multisensory integration is a dialogue between sensory modalities rather that the convergence of all sensory information onto a supra-modal area.     

Integration of touch and sound in auditory cortex

To form a coherent percept of the environment, our brain combines information from different senses. Such multisensory integration occurs in higher association cortices; but supposedly, it also occurs in early sensory areas. Confirming the latter hypothesis, we unequivocally demonstrate supra-additive integration of touch and sound stimulation at the second stage of the auditory cortex. Using high-resolution fMRI of the macaque monkey, we quantified the integration of auditory broad-band noise and tactile stimulation of hand and foot in anaesthetized animals. Integration was found posterior to and along the lateral side of the primary auditory cortex in the caudal auditory belt. Integration was stronger for temporally coincident stimuli and obeyed the principle of inverse effectiveness: greater enhancement for less effective stimuli. These findings demonstrates that multisensory integration occurs early and close to primary sensory areas and--because it occurs in anaesthetized animals--suggests that this integration is mediated by preattentive bottom-up mechanisms.     

Multisensory integration: space, time and superadditivity

The superior colliculus generates and controls eye and head movements based on signals from different senses. The latest research on this structure enhances our understanding of the mechanisms of multisensory integration in the brain.     

Multisensory integration of natural odors and sounds in the auditory cortex

Motherhood is associated with different forms of physiological alterations including transient hormonal changes and brain plasticity. The underlying impact of these changes on the emergence of maternal behaviors and sensory processing within the mother's brain are largely unknown. By using in?vivo cell-attached recordings in the primary auditory cortex of female mice, we discovered that exposure to pups' body odor reshapes neuronal responses to pure tones and natural auditory stimuli. This olfactory-auditory interaction appeared naturally in lactating mothers shortly after parturition and was long lasting. Naive virgins that had experience with the pups also showed an appearance of olfactory-auditory integration in A1, suggesting that multisensory integration may be experience dependent. Neurons from lactating mothers were more sensitive to sounds as compared to those from experienced mice, independent of the odor effects. These uni- and multisensory cortical changes may facilitate the detection and discrimination of pup distress calls and strengthen the bond between mothers and their neonates. VIDEO ABSTRACT:     

Molecular mechanisms linking diabetes mellitus and Alzheimer disease: beta-amyloid peptide, insulin signaling, and neuronal function

The incidence of Alzheimer disease (AD) and diabetes mellitus (DM) is increasing at an alarming rate and has become a major public health concern worldwide. Recent epidemiological studies have provided direct evidence that DM is a strong risk factor for AD; this finding is now attracting attention. However, the underlying mechanisms for this association remain largely unknown. Previous in vitro and in vivo studies reported that diabetic conditions could cause an increase in the beta-amyloid peptide (Aβ) levels, which exhibits neurotoxic properties and plays a causative role in AD. However, unexpectedly, recent clinicopathological studies have shown no evidence that the pathological hallmarks of AD, including amyloid plaque, were increased in the brains of diabetic patients, suggesting that DM could affect the pathogenesis of AD through mechanisms other than modulation of Aβ metabolism. One possible mechanism is the alteration in brain insulin signaling. It has been shown that insulin signaling is involved in a variety of neuronal functions, and that it also plays a significant role in the pathophysiology of AD. Thus, the modification of neuronal insulin signaling by diabetic conditions may contribute to AD progression. Another possible mechanism is cerebrovascular alteration, a common pathological change observed in both diseases. Accumulating evidence has suggested the importance of Aβ-induced cerebrovascular dysfunction in AD, and indicated that pathological interactions between the receptor for advanced glycation end products (RAGE) and Aβ peptides may play a role in this dysfunction. Our study has provided a further understanding of the potential underlying mechanisms linking DM and AD by establishing novel mouse models showing pathological manifestations of both diseases. The current review summarizes the results from recent studies on the pathological relationship between DM and AD while focusing on brain insulin signaling and cerebrovascular alteration. It also discusses the therapeutic potential of these findings and future treatment strategies for AD.     

Remembrance of odors past: human olfactory cortex in cross-modal recognition memory

Episodic memory is often imbued with multisensory richness, such that the recall of an event can be endowed with the sights, sounds, and smells of its prior occurrence. While hippocampus and related medial temporal structures are implicated in episodic memory retrieval, the participation of sensory-specific cortex in representing the qualities of an episode is less well established. We combined functional magnetic resonance imaging (fMRI) with a cross-modal paradigm, where objects were presented with odors during memory encoding. We then examined the effect of odor context on neural responses at retrieval when these same objects were presented alone. Primary olfactory (piriform) cortex, as well as anterior hippocampus, was activated during the successful retrieval of old (compared to new) objects. Our findings indicate that sensory features of the original engram are preserved in unimodal olfactory cortex. We suggest that reactivation of memory traces distributed across modality-specific brain areas underpins the sensory qualities of episodic memories.