Behavioral correlates of cortical semantic representations modeled by word vectors

The quantitative modeling of semantic representations in the brain plays a key role in understanding the neural basis of semantic processing. Previous studies have demonstrated that word vectors, which were originally developed for use in the field of natural language processing, provide a powerful tool for such quantitative modeling. However, whether semantic representations in the brain revealed by the word vector-based models actually capture our perception of semantic information remains unclear, as there has been no study explicitly examining the behavioral correlates of the modeled brain semantic representations. To address this issue, we compared the semantic structure of nouns and adjectives in the brain estimated from word vector-based brain models with that evaluated from human behavior. The brain models were constructed using voxelwise modeling to predict the functional magnetic resonance imaging (fMRI) response to natural movies from semantic contents in each movie scene through a word vector space. The semantic dissimilarity of brain word representations was then evaluated using the brain models. Meanwhile, data on human behavior reflecting the perception of semantic dissimilarity between words were collected in psychological experiments. We found a significant correlation between brain model- and behavior-derived semantic dissimilarities of words. This finding suggests that semantic representations in the brain modeled via word vectors appropriately capture our perception of word meanings.     

Vision and the representation of the surroundings in spatial memory

One of the paradoxes of vision is that the world as it appears to us and the image on the retina at any moment are not much like each other. The visual world seems to be extensive and continuous across time. However, the manner in which we sample the visual environment is neither extensive nor continuous. How does the brain reconcile these differences? Here, we consider existing evidence from both static and dynamic viewing paradigms together with the logical requirements of any representational scheme that would be able to support active behaviour. While static scene viewing paradigms favour extensive, but perhaps abstracted, memory representations, dynamic settings suggest sparser and task-selective representation. We suggest that in dynamic settings where movement within extended environments is required to complete a task, the combination of visual input, egocentric and allocentric representations work together to allow efficient behaviour. The egocentric model serves as a coding scheme in which actions can be planned, but also offers a potential means of providing the perceptual stability that we experience.     

The neural fate of consciously perceived and missed events in the attentional blink

Cognitive models of attention propose that visual perception is a product of two stages of visual processing: early operations permit rapid initial categorization of the visual world, while later attention-demanding capacity-limited stages are necessary for the conscious report of the stimuli. Here we used the attentional blink paradigm and fMRI to neurally distinguish these two stages of vision. Subjects detected a face target and a scene target presented rapidly among distractors at fixation. Although the second, scene target frequently went undetected by the subjects, it nonetheless activated regions of the medial temporal cortex involved in high-level scene representations, the parahippocampal place area (PPA). This PPA activation was amplified when the stimulus was consciously perceived. By contrast, the frontal cortex was activated only when scenes were successfully reported. These results suggest that medial temporal cortex permits rapid categorization of the visual input, while the frontal cortex is part of a capacity-limited attentional bottleneck to conscious report.     

Within- and Cross-Modal Distance Information Disambiguate Visual Size-Change Perception

Perception is fundamentally underconstrained because different combinations of object properties can generate the same sensory information. To disambiguate sensory information into estimates of scene properties, our brains incorporate prior knowledge and additional “auxiliary” (i.e., not directly relevant to desired scene property) sensory information to constrain perceptual interpretations. For example, knowing the distance to an object helps in perceiving its size. The literature contains few demonstrations of the use of prior knowledge and auxiliary information in combined visual and haptic disambiguation and almost no examination of haptic disambiguation of vision beyond “bistable” stimuli. Previous studies have reported humans integrate multiple unambiguous sensations to perceive single, continuous object properties, like size or position. Here we test whether humans use visual and haptic information, individually and jointly, to disambiguate size from distance. We presented participants with a ball moving in depth with a changing diameter. Because no unambiguous distance information is available under monocular viewing, participants rely on prior assumptions about the ball''s distance to disambiguate their -size percept. Presenting auxiliary binocular and/or haptic distance information augments participants'' prior distance assumptions and improves their size judgment accuracy—though binocular cues were trusted more than haptic. Our results suggest both visual and haptic distance information disambiguate size perception, and we interpret these results in the context of probabilistic perceptual reasoning.     

The parahippocampal place area: recognition, navigation, or encoding?

The parahippocampal place area (PPA) has been demonstrated to respond more strongly in fMRI to scenes depicting places than to other kinds of visual stimuli. Here, we test several hypotheses about the function of the PPA. We find that PPA activity (1) is not affected by the subjects' familiarity with the place depicted, (2) does not increase when subjects experience a sense of motion through the scene, and (3) is greater when viewing novel versus repeated scenes but not novel versus repeated faces. Thus, we find no evidence that the PPA is involved in matching perceptual information to stored representations in memory, in planning routes, or in monitoring locomotion through the local or distal environment but some evidence that it is involved in encoding new perceptual information about the appearance and layout of scenes.     

Visual, haptic and cross-modal recognition of objects and scenes.

In this article we review current literature on cross-modal recognition and present new findings from our studies on object and scene recognition. Specifically, we address the questions of what is the nature of the representation underlying each sensory system that facilitates convergence across the senses and how perception is modified by the interaction of the senses. In the first set of our experiments, the recognition of unfamiliar objects within and across the visual and haptic modalities was investigated under conditions of changes in orientation (0 degrees or 180 degrees ). An orientation change increased recognition errors within each modality but this effect was reduced across modalities. Our results suggest that cross-modal object representations of objects are mediated by surface-dependent representations. In a second series of experiments, we investigated how spatial information is integrated across modalities and viewpoint using scenes of familiar, 3D objects as stimuli. We found that scene recognition performance was less efficient when there was either a change in modality, or in orientation, between learning and test. Furthermore, haptic learning was selectively disrupted by a verbal interpolation task. Our findings are discussed with reference to separate spatial encoding of visual and haptic scenes. We conclude by discussing a number of constraints under which cross-modal integration is optimal for object recognition. These constraints include the nature of the task, and the amount of spatial and temporal congruency of information across the modalities.     

Haptic categorical perception of shape

Categorization and categorical perception have been extensively studied, mainly in vision and audition. In the haptic domain, our ability to categorize objects has also been demonstrated in earlier studies. Here we show for the first time that categorical perception also occurs in haptic shape perception. We generated a continuum of complex shapes by morphing between two volumetric objects. Using similarity ratings and multidimensional scaling we ensured that participants could haptically discriminate all objects equally. Next, we performed classification and discrimination tasks. After a short training with the two shape categories, both tasks revealed categorical perception effects. Training leads to between-category expansion resulting in higher discriminability of physical differences between pairs of stimuli straddling the category boundary. Thus, even brief training can alter haptic representations of shape. This suggests that the weights attached to various haptic shape features can be changed dynamically in response to top-down information about class membership.     

Auditory cortical contrast enhancing by global winner-take-all inhibitory interactions

Brains decompose the world into discrete objects of perception, thereby facing the problem of how to segregate and selectively address similar objects that are concurrently present in a scene. Theoretical models propose that this could be achieved by neuronal implementations of so-called winner-take-all algorithms where neuronal representations of objects or object features interact in a competitive manner. Here we present evidence for the existence of such a mechanism in an animal species. We present electrophysiological, neuropharmacological and neuroanatomical data which suggest a novel view of the role of GABA(A)-mediated inhibition in primary auditory cortex (AI), where intracortical GABA(A)-mediated inhibition operates on a global scale within a circular map of sound periodicity representation in AI, with functionally inhibitory projections of similar effect from any location throughout the whole map. These interactions could underlie the proposed competitive "winner-take-all" algorithm to support object segregation, e.g., segregation of different speakers in cocktail-party situations.     

The art of seeing and painting

The human urge to represent the three-dimensional world using two-dimensional pictorial representations dates back at least to Paleolithic times. Artists from ancient to modern times have struggled to understand how a few contours or color patches on a flat surface can induce mental representations of a three-dimensional scene. This article summarizes some of the recent breakthroughs in scientifically understanding how the brain sees that shed light on these struggles. These breakthroughs illustrate how various artists have intuitively understood paradoxical properties about how the brain sees, and have used that understanding to create great art. These paradoxical properties arise from how the brain forms the units of conscious visual perception; namely, representations of three-dimensional boundaries and surfaces. Boundaries and surfaces are computed in parallel cortical processing streams that obey computationally complementary properties. These streams interact at multiple levels to overcome their complementary weaknesses and to transform their complementary properties into consistent percepts. The article describes how properties of complementary consistency have guided the creation of many great works of art.     

Neural response to perception of volume in the lateral occipital complex

Projection of a 3D scene onto the 2D retina necessarily entails a loss of information, yet perceivers experience a world populated with volumetric objects. Using simultaneous behavioral and neural (fMRI) measures, we identify neural bases of volume perception. Neural activity in the lateral occipital cortex increased with presentation of 3D volumes relative to presentation of 2D shapes. Neural activity also modulated with perceived volume, independent of image information. When behavioral responses indicated that observers saw ambiguous images as 3D volumes, neural response increased; when behavioral data revealed a 2D interpretation, neural response waned. Crucially, the physical stimulus was identical under both interpretations; only the percept of volume can account for the increased neural activity.     

Distributed representation of perceptual categories in the auditory cortex

Categorical perception is a process by which a continuous stimulus space is partitioned to represent discrete sensory events. Early experience has been shown to shape categorical perception and enlarge cortical representations of experienced stimuli in the sensory cortex. The present study examines the hypothesis that enlargement in cortical stimulus representations is a mechanism of categorical perception. Perceptual discrimination and identification behaviors were analyzed in model auditory cortices that incorporated sound exposure-induced plasticity effects. The model auditory cortex with over-representations of specific stimuli exhibited categorical perception behaviors for those specific stimuli. These results indicate that enlarged stimulus representations in the sensory cortex may be a mechanism for categorical perceptual learning.     

Sound source perception in anuran amphibians

Sound source perception refers to the auditory system's ability to parse incoming sensory information into coherent representations of distinct sound sources in the environment. Such abilities are no doubt key to successful communication in many taxa, but we know little about their function in animal communication systems. For anuran amphibians (frogs and toads), social and reproductive behaviors depend on a listener's ability to hear and identify sound signals amid high levels of background noise in acoustically cluttered environments. Recent neuroethological studies are revealing how frogs parse these complex acoustic scenes to identify individual calls in noisy breeding choruses. Current evidence highlights some interesting similarities and differences in how the auditory systems of frogs and other vertebrates (most notably birds and mammals) perform auditory scene analysis.     

Modelling the Emergence and Dynamics of Perceptual Organisation in Auditory Streaming

Many sound sources can only be recognised from the pattern of sounds they emit, and not from the individual sound events that make up their emission sequences. Auditory scene analysis addresses the difficult task of interpreting the sound world in terms of an unknown number of discrete sound sources (causes) with possibly overlapping signals, and therefore of associating each event with the appropriate source. There are potentially many different ways in which incoming events can be assigned to different causes, which means that the auditory system has to choose between them. This problem has been studied for many years using the auditory streaming paradigm, and recently it has become apparent that instead of making one fixed perceptual decision, given sufficient time, auditory perception switches back and forth between the alternatives—a phenomenon known as perceptual bi- or multi-stability. We propose a new model of auditory scene analysis at the core of which is a process that seeks to discover predictable patterns in the ongoing sound sequence. Representations of predictable fragments are created on the fly, and are maintained, strengthened or weakened on the basis of their predictive success, and conflict with other representations. Auditory perceptual organisation emerges spontaneously from the nature of the competition between these representations. We present detailed comparisons between the model simulations and data from an auditory streaming experiment, and show that the model accounts for many important findings, including: the emergence of, and switching between, alternative organisations; the influence of stimulus parameters on perceptual dominance, switching rate and perceptual phase durations; and the build-up of auditory streaming. The principal contribution of the model is to show that a two-stage process of pattern discovery and competition between incompatible patterns can account for both the contents (perceptual organisations) and the dynamics of human perception in auditory streaming.     

The eyes of deep-sea fishes and the changing nature of visual scenes with depth

The visual scenes viewed by ocean animals change dramatically with depth. In the brighter epipelagic depths, daylight provides an extended field of illumination. In mesopelagic depths down to 1000 m the visual scene is semi-extended, with the downwelling daylight providing increasingly dim extended illumination with depth. In contrast, greater depths increase the prominence of point-source bioluminescent flashes. In bathypelagic depths (below 1000 m daylight no longer penetrates, and the visual scene consists exclusively of point-source bioluminescent flashes. In this paper, I show that the eyes of fishes match this change from extended to point-source illumination, becoming increasingly foveate and spatially acute with increasing depth. A sharp fovea is optimal for localizing point sources. Quite contrary to their reputation as 'degenerate' and 'regressed', I show here that the remarkably prominent foveae and relatively large pupils of bathypelagic fishes give them excellent perception and localization of bioluminescent flashes up to a few tens of metres distant. In a world with almost no food, where fishes are weak and must swim very slowly this range of detection (and interception) is energetically realistic, with distances greater than this physically beyond range. Larger and more sensitive eyes would give bathypelagic fishes little more than the useless ability to see flashes beyond reach.     

View from the top: hierarchies and reverse hierarchies in the visual system

We propose that explicit vision advances in reverse hierarchical direction, as shown for perceptual learning. Processing along the feedforward hierarchy of areas, leading to increasingly complex representations, is automatic and implicit, while conscious perception begins at the hierarchy's top, gradually returning downward as needed. Thus, our initial conscious percept--vision at a glance--matches a high-level, generalized, categorical scene interpretation, identifying "forest before trees." For later vision with scrutiny, reverse hierarchy routines focus attention to specific, active, low-level units, incorporating into conscious perception detailed information available there. Reverse Hierarchy Theory dissociates between early explicit perception and implicit low-level vision, explaining a variety of phenomena. Feature search "pop-out" is attributed to high areas, where large receptive fields underlie spread attention detecting categorical differences. Search for conjunctions or fine discriminations depends on reentry to low-level specific receptive fields using serial focused attention, consistent with recently reported primary visual cortex effects.     

Egomotion and relative depth map from optical flow

When an observer moves in a 3D world, optical flow fields are generated on his retina. We argue that such an observer can in principle compute the parameters of his egomotion, and following this, the relative depth map of the stationary environment solely from the instantaneous positional velocity fields (IPVF). Moreover, we argue that in the stationary world, this analysis can be done locally, and is not dependent on global properties of the optical flow under the imposed constraints (smoothness of the egomotion path, rigidity of objects, temporal continuity of perception). To investigate the method, and to analyze its performance, a computer model has been constructed which simulates an observer moving through a 3D world of stationary rectangular planes at different depths and orientations. The results suggest that the method offers a reasonable and computationally feasible means of extracting information about egomotion and surface layout from optical flows, under certain circumstances. We discuss some issues related to extending the analysis to the case of a rigid world of moving objects, and some issues related to the status of information extractable from optical flows with respect to other sources of information.     

Is object search mediated by object-based or image-based representations?

Recent research suggests that visually specific memory representations for previously fixated objects are maintained during scene perception. Here we investigate the degree of visual specificity by asking whether the memory representations are image-based or object-based. To that end we measured the effects of object orientation on the time to search for a familiar object from amongst a set of 7 familiar distractors arranged in a circular array. Search times were found to depend on the relative orientations of the target object and the probe object for both familiar and novel objects. This effect was found to be partly an image matching effect but there was also an advantage shown for the object's canonical view for familiar objects. Orientation effects were maintained even when the target object was specified as having unique or similar shape properties relative to the distractors. Participants' eye movements were monitored during two of the experiments. Eye movement patterns revealed selection for object shape and object orientation during the search process. Our findings provide evidence for object representations during search that are detailed and share image-based characteristics with more high-level characteristics from object memory.     

Extraction of Surface-Related Features in a Recurrent Model of V1-V2 Interactions

Background

Humans can effortlessly segment surfaces and objects from two-dimensional (2D) images that are projections of the 3D world. The projection from 3D to 2D leads partially to occlusions of surfaces depending on their position in depth and on viewpoint. One way for the human visual system to infer monocular depth cues could be to extract and interpret occlusions. It has been suggested that the perception of contour junctions, in particular T-junctions, may be used as cue for occlusion of opaque surfaces. Furthermore, X-junctions could be used to signal occlusion of transparent surfaces.

Methodology/Principal Findings

In this contribution, we propose a neural model that suggests how surface-related cues for occlusion can be extracted from a 2D luminance image. The approach is based on feedforward and feedback mechanisms found in visual cortical areas V1 and V2. In a first step, contours are completed over time by generating groupings of like-oriented contrasts. Few iterations of feedforward and feedback processing lead to a stable representation of completed contours and at the same time to a suppression of image noise. In a second step, contour junctions are localized and read out from the distributed representation of boundary groupings. Moreover, surface-related junctions are made explicit such that they are evaluated to interact as to generate surface-segmentations in static images. In addition, we compare our extracted junction signals with a standard computer vision approach for junction detection to demonstrate that our approach outperforms simple feedforward computation-based approaches.

Conclusions/Significance

A model is proposed that uses feedforward and feedback mechanisms to combine contextually relevant features in order to generate consistent boundary groupings of surfaces. Perceptually important junction configurations are robustly extracted from neural representations to signal cues for occlusion and transparency. Unlike previous proposals which treat localized junction configurations as 2D image features, we link them to mechanisms of apparent surface segregation. As a consequence, we demonstrate how junctions can change their perceptual representation depending on the scene context and the spatial configuration of boundary fragments.