Neural Representation of Ambiguous Visual Objects in the Inferior Temporal Cortex

Inferior temporal (IT) cortex as the final stage of the ventral visual pathway is involved in visual object recognition. In our everyday life we need to recognize visual objects that are degraded by noise. Psychophysical studies have shown that the accuracy and speed of the object recognition decreases as the amount of visual noise increases. However, the neural representation of ambiguous visual objects and the underlying neural mechanisms of such changes in the behavior are not known. Here, by recording the neuronal spiking activity of macaque monkeys’ IT we explored the relationship between stimulus ambiguity and the IT neural activity. We found smaller amplitude, later onset, earlier offset and shorter duration of the response as visual ambiguity increased. All of these modulations were gradual and correlated with the level of stimulus ambiguity. We found that while category selectivity of IT neurons decreased with noise, it was preserved for a large extent of visual ambiguity. This noise tolerance for category selectivity in IT was lost at 60% noise level. Interestingly, while the response of the IT neurons to visual stimuli at 60% noise level was significantly larger than their baseline activity and full (100%) noise, it was not category selective anymore. The latter finding shows a neural representation that signals the presence of visual stimulus without signaling what it is. In general these findings, in the context of a drift diffusion model, explain the neural mechanisms of perceptual accuracy and speed changes in the process of recognizing ambiguous objects.     

Deep Supervised,but Not Unsupervised,Models May Explain IT Cortical Representation

Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.     

Shape Similarity,Better than Semantic Membership,Accounts for the Structure of Visual Object Representations in a Population of Monkey Inferotemporal Neurons

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex.     

Feature-based attentional modulations in the absence of direct visual stimulation

When faced with a crowded visual scene, observers must selectively attend to behaviorally relevant objects to avoid sensory overload. Often this selection process is guided by prior knowledge of a target-defining feature (e.g., the color red when looking for an apple), which enhances the firing rate of visual neurons that are selective for the attended feature. Here, we used functional magnetic resonance imaging and a pattern classification algorithm to predict the attentional state of human observers as they monitored a visual feature (one of two directions of motion). We find that feature-specific attention effects spread across the visual field-even to regions of the scene that do not contain a stimulus. This spread of feature-based attention to empty regions of space may facilitate the perception of behaviorally relevant stimuli by increasing sensitivity to attended features at all locations in the visual field.     

Viewer-centered object representation in the human visual system revealed by viewpoint aftereffects

Are there neurons representing specific views of objects in the human visual system? A visual selective adaptation method was used to address this question. After visual adaptation to an object viewed either 15 or 30 degrees from one side, when the same object was subsequently presented near the frontal view, the perceived viewing directions were biased in a direction opposite to that of the adapted viewpoint. This aftereffect can be obtained with spatially nonoverlapping adapting and test stimuli, and it depends on the global representation of the adapting stimuli. Viewpoint aftereffects were found within, but not across, categories of objects tested (faces, cars, wire-like objects). The magnitude of this aftereffect depends on the angular difference between the adapting and test viewing angles and grows with increasing duration of adaptation. These results support the existence of object-selective neurons tuned to specific viewing angles in the human visual system.     

From thought to action: the parietal cortex as a bridge between perception, action, and cognition

The lateral intraparietal area (LIP) is a subdivision of the inferior parietal lobe that has been implicated in the guidance of spatial attention. In a variety of tasks, LIP provides a "salience representation" of the external world-a topographic visual representation that encodes the locations of salient or behaviorally relevant objects. Recent neurophysiological experiments show that this salience representation incorporates information about multiple behavioral variables-such as a specific motor response, reward, or category membership-associated with the task-relevant object. This integration occurs in a wide variety of tasks, including those requiring eye or limb movements or goal-directed or nontargeting operant responses. Thus, LIP acts as a multifaceted behavioral integrator that binds visuospatial, motor, and cognitive information into a topographically organized signal of behavioral salience. By specifying attentional priority as a synthesis of multiple task demands, LIP operates at the interface of perception, action, and cognition.     

Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance

A subcortical pathway through the superior colliculus and pulvinar to the amygdala is commonly assumed to mediate the non-conscious processing of affective visual stimuli. We review anatomical and physiological data that argue against the notion that such a pathway plays a prominent part in processing affective visual stimuli in humans. Instead, we propose that the primary role of the amygdala in visual processing, like that of the pulvinar, is to coordinate the function of cortical networks during evaluation of the biological significance of affective visual stimuli. Under this revised framework, the cortex has a more important role in emotion processing than is traditionally assumed.     

Size Constancy in Bat Biosonar? Perceptual Interaction of Object Aperture and Distance

Perception and encoding of object size is an important feature of sensory systems. In the visual system object size is encoded by the visual angle (visual aperture) on the retina, but the aperture depends on the distance of the object. As object distance is not unambiguously encoded in the visual system, higher computational mechanisms are needed. This phenomenon is termed “size constancy”. It is assumed to reflect an automatic re-scaling of visual aperture with perceived object distance. Recently, it was found that in echolocating bats, the ‘sonar aperture’, i.e., the range of angles from which sound is reflected from an object back to the bat, is unambiguously perceived and neurally encoded. Moreover, it is well known that object distance is accurately perceived and explicitly encoded in bat sonar. Here, we addressed size constancy in bat biosonar, recruiting virtual-object techniques. Bats of the species Phyllostomus discolor learned to discriminate two simple virtual objects that only differed in sonar aperture. Upon successful discrimination, test trials were randomly interspersed using virtual objects that differed in both aperture and distance. It was tested whether the bats spontaneously assigned absolute width information to these objects by combining distance and aperture. The results showed that while the isolated perceptual cues encoding object width, aperture, and distance were all perceptually well resolved by the bats, the animals did not assign absolute width information to the test objects. This lack of sonar size constancy may result from the bats relying on different modalities to extract size information at different distances. Alternatively, it is conceivable that familiarity with a behaviorally relevant, conspicuous object is required for sonar size constancy, as it has been argued for visual size constancy. Based on the current data, it appears that size constancy is not necessarily an essential feature of sonar perception in bats.     

Multifocal attention filters targets from distracters within and beyond primate MT neurons' receptive field boundaries

Visual attention has been classically described as a spotlight that enhances the processing of a behaviorally relevant object. However, in many situations, humans and animals must simultaneously attend to several relevant objects separated by distracters. To account for this ability, various models of attention have been proposed including splitting of the attentional spotlight into multiple foci, zooming of the spotlight over a region of space, and switching of the spotlight among objects. We investigated this controversial issue by recording neuronal activity in visual area MT of two macaques while they attended to two translating objects that circumvented a third distracter object located inside the neurons' receptive field. We found that when the attended objects passed through or nearby the receptive field, neuronal responses to the distracter were either decreased or remained unaltered. These results demonstrate that attention can split into multiple spotlights corresponding to relevant objects while filtering out interspersed distracters.     

Innate visual object recognition in vertebrates: some proposed pathways and mechanisms

Almost all vertebrates are capable of recognizing biologically relevant stimuli at or shortly after birth, and in some phylogenetically ancient species visual object recognition is exclusively innate. Extensive and detailed studies of the anuran visual system have resulted in the determination of the neural structures and pathways involved in innate prey and predator recognition in these species [Behav. Brain Sci. 10 (1987) 337; Comp. Biochem. Physiol. A 128 (2001) 417]. The structures involved include the optic tectum, pretectal nuclei and an area within the mesencephalic tegmentum. Here we investigate the structures and pathways involved in innate stimulus recognition in avian, rodent and primate species. We discuss innate stimulus preferences in maternal imprinting in chicks and argue that these preferences are due to innate visual recognition of conspecifics, entirely mediated by subtelencephalic structures. In rodent species, brainstem structures largely homologous to the components of the anuran subcortical visual system mediate innate visual object recognition. The primary components of the mammalian subcortical visual system are the superior colliculus, nucleus of the optic tract, anterior and posterior pretectal nuclei, nucleus of the posterior commissure, and an area within the mesopontine reticular formation that includes parts of the cuneiform, subcuneiform and pedunculopontine nuclei. We argue that in rodent species the innate sensory recognition systems function throughout ontogeny, acting in parallel with cortical sensory and recognition systems. In primates the structures involved in innate stimulus recognition are essentially the same as those in rodents, but overt innate recognition is only present in very early ontogeny, and after a transition period gives way to learned object recognition mediated by cortical structures. After the transition period, primate subcortical sensory systems still function to provide implicit innate stimulus recognition, and this recognition can still generate orienting, neuroendocrine and emotional responses to biologically relevant stimuli.     

Stimulus dependency of object-evoked responses in human visual cortex: an inverse problem for category specificity

Many studies have linked the processing of different object categories to specific event-related potentials (ERPs) such as the face-specific N170. Despite reports showing that object-related ERPs are influenced by visual stimulus features, there is consensus that these components primarily reflect categorical aspects of the stimuli. Here, we re-investigated this idea by systematically measuring the effects of visual feature manipulations on ERP responses elicited by both structure-from-motion (SFM)-defined and luminance-defined object stimuli. SFM objects elicited a novel component at 200-250 ms (N250) over parietal and posterior temporal sites. We found, however, that the N250 amplitude was unaffected by restructuring SFM stimuli into meaningless objects based on identical visual cues. This suggests that this N250 peak was not uniquely linked to categorical aspects of the objects, but is strongly determined by visual stimulus features. We provide strong support for this hypothesis by parametrically manipulating the depth range of both SFM- and luminance-defined object stimuli and showing that the N250 evoked by SFM stimuli as well as the well-known N170 to static faces were sensitive to this manipulation. Importantly, this effect could not be attributed to compromised object categorization in low depth stimuli, confirming a strong impact of visual stimulus features on object-related ERP signals. As ERP components linked with visual categorical object perception are likely determined by multiple stimulus features, this creates an interesting inverse problem when deriving specific perceptual processes from variations in ERP components.     

Visual exploration and object recognition by lattice deformation

Mechanisms of explicit object recognition are often difficult to investigate and require stimuli with controlled features whose expression can be manipulated in a precise quantitative fashion. Here, we developed a novel method (called "Dots"), for generating visual stimuli, which is based on the progressive deformation of a regular lattice of dots, driven by local contour information from images of objects. By applying progressively larger deformation to the lattice, the latter conveys progressively more information about the target object. Stimuli generated with the presented method enable a precise control of object-related information content while preserving low-level image statistics, globally, and affecting them only little, locally. We show that such stimuli are useful for investigating object recognition under a naturalistic setting--free visual exploration--enabling a clear dissociation between object detection and explicit recognition. Using the introduced stimuli, we show that top-down modulation induced by previous exposure to target objects can greatly influence perceptual decisions, lowering perceptual thresholds not only for object recognition but also for object detection (visual hysteresis). Visual hysteresis is target-specific, its expression and magnitude depending on the identity of individual objects. Relying on the particular features of dot stimuli and on eye-tracking measurements, we further demonstrate that top-down processes guide visual exploration, controlling how visual information is integrated by successive fixations. Prior knowledge about objects can guide saccades/fixations to sample locations that are supposed to be highly informative, even when the actual information is missing from those locations in the stimulus. The duration of individual fixations is modulated by the novelty and difficulty of the stimulus, likely reflecting cognitive demand.     

Gamma oscillations in human primary somatosensory cortex reflect pain perception

Successful behavior requires selection and preferred processing of relevant sensory information. The cortical representation of relevant sensory information has been related to neuronal oscillations in the gamma frequency band. Pain is of invariably high behavioral relevance and, thus, nociceptive stimuli receive preferred processing. Here, by using magnetoencephalography, we show that selective nociceptive stimuli induce gamma oscillations between 60 and 95 Hz in primary somatosensory cortex. Amplitudes of pain-induced gamma oscillations vary with objective stimulus intensity and subjective pain intensity. However, around pain threshold, perceived stimuli yielded stronger gamma oscillations than unperceived stimuli of equal stimulus intensity. These results show that pain induces gamma oscillations in primary somatosensory cortex that are particularly related to the subjective perception of pain. Our findings support the hypothesis that gamma oscillations are related to the internal representation of behaviorally relevant stimuli that should receive preferred processing.     

The inferior temporal cortex: Architecture, computation, and representation

Neurons in the inferior temporal cortex (IT), an area crucially involved in visual object recognition in monkeys, show the visual response properties and anatomical/chemical nature which are distinct from those in the cortical areas that feed visual inputs to the IT. Earlier physiological studies showed that IT neurons have large receptive fields covering the center and contralateral (often bilateral) visual fields, stimulus selectivity for images of complex objects or shapes, and translation invariance of the stimulus selectivity. Recent studies have revealed new aspects of their properties such as invariant selectivity for shapes despite drastic changes in various physical attributes of stimuli, latent excitatory inputs masked by stimulus-specific GABAergic inhibition, selectivity for binocular disparity and 3-dimensional surface structures, profound effects of learning on the stimulus selectivity, and columnar clustering of neurons with similarstimulus selectivity for shapes and other object features. Another line of research using histological techniques have revealed that pyramidal neurons in the IT are larger in the size of dendritic arbors, in the number of dendritic branches and spines, and in the size and distribution of horizontal axonal arbors than those in the earlier areas, allowing them to integrate a larger population of afferents and process more diverse inputs. The concentration of several neurochemicals including those related to synaptic transmission or plasticity changes systematically towards the IT along the occipitotemporal pathway. Many of the characteristics of IT neurons parallel or explain certain aspects of visual object perception, although the behavioral relevance has yet to be addressed experimentally.     

Integration of bimodal looming signals through neuronal coherence in the temporal lobe

The ability to integrate information across multiple sensory systems offers several behavioral advantages, from quicker reaction times and more accurate responses to better detection and more robust learning. At the neural level, multisensory integration requires large-scale interactions between different brain regions--the convergence of information from separate sensory modalities, represented by distinct neuronal populations. The interactions between these neuronal populations must be fast and flexible, so that behaviorally relevant signals belonging to the same object or event can be immediately integrated and integration of unrelated signals can be prevented. Looming signals are a particular class of signals that are behaviorally relevant for animals and that occur in both the auditory and visual domain. These signals indicate the rapid approach of objects and provide highly salient warning cues about impending impact. We show here that multisensory integration of auditory and visual looming signals may be mediated by functional interactions between auditory cortex and the superior temporal sulcus, two areas involved in integrating behaviorally relevant auditory-visual signals. Audiovisual looming signals elicited increased gamma-band coherence between these areas, relative to unimodal or receding-motion signals. This suggests that the neocortex uses fast, flexible intercortical interactions to mediate multisensory integration.     

Medial axis shape coding in macaque inferotemporal cortex

The basic, still unanswered question about visual object representation is this: what specific information is encoded by neural signals? Theorists have long predicted that neurons would encode medial axis or skeletal object shape, yet recent studies reveal instead neural coding of boundary or surface shape. Here, we addressed this theoretical/experimental disconnect, using adaptive shape sampling to demonstrate explicit coding of medial axis shape in high-level object cortex (macaque monkey inferotemporal cortex or IT). Our metric shape analyses revealed a coding continuum, along which most neurons represent a configuration of both medial axis and surface components. Thus, IT response functions embody a rich basis set for simultaneously representing skeletal and external shape of complex objects. This would be especially useful for representing biological shapes, which are often characterized by both complex, articulated skeletal structure and specific surface features.     

Extracting the Behaviorally Relevant Stimulus: Unique Neural Representation of Farnesol,a Component of the Recruitment Pheromone of Bombus terrestris

To trigger innate behavior, sensory neural networks are pre-tuned to extract biologically relevant stimuli. Many male-female or insect-plant interactions depend on this phenomenon. Especially communication among individuals within social groups depends on innate behaviors. One example is the efficient recruitment of nest mates by successful bumblebee foragers. Returning foragers release a recruitment pheromone in the nest while they perform a ‘dance’ behavior to activate unemployed nest mates. A major component of this pheromone is the sesquiterpenoid farnesol. How farnesol is processed and perceived by the olfactory system, has not yet been identified. It is much likely that processing farnesol involves an innate mechanism for the extraction of relevant information to trigger a fast and reliable behavioral response. To test this hypothesis, we used population response analyses of 100 antennal lobe (AL) neurons recorded in alive bumblebee workers under repeated stimulation with four behaviorally different, but chemically related odorants (geraniol, citronellol, citronellal and farnesol). The analysis identified a unique neural representation of the recruitment pheromone component compared to the other odorants that are predominantly emitted by flowers. The farnesol induced population activity in the AL allowed a reliable separation of farnesol from all other chemically related odor stimuli we tested. We conclude that the farnesol induced population activity may reflect a predetermined representation within the AL-neural network allowing efficient and fast extraction of a behaviorally relevant stimulus. Furthermore, the results show that population response analyses of multiple single AL-units may provide a powerful tool to identify distinct representations of behaviorally relevant odors.     

Object discrimination by pigeons: effects of object color and shape

Can nonhuman animals attend to visual stimuli as whole, coherent objects? We investigated this question by adapting for use with pigeons a task in which human participants must report whether two visual attributes belong to the same object (one-object trial) or to different objects (two-object trial). We trained pigeons to discriminate a pair of differently colored shapes that had two targets either on a single object or on two different objects. Each target equally often appeared on the one-object and two-object stimuli; therefore, a specific target location could not serve as a discriminative cue. The pigeons learned to report whether the two target dots were located on a single object or on two different objects; follow-up tests demonstrated that this ability was not entirely based on memorization of the dot patterns and locations. Additional tests disclosed predominate stimulus control by the color, but not by the shape of the two objects. These findings suggest that human psychophysical methods are readily applicable to the study of object discrimination by nonhuman animals.     

Neurons in area V4 of the macaque translate attended visual features into behaviorally relevant categories

Neural processing at most stages of the primate visual system is modulated by selective attention, such that behaviorally relevant information is emphasized at the expenses of irrelevant, potentially distracting information. The form of attention best understood at the cellular level is when stimuli at a given location in the visual field must be selected (space-based attention). In contrast, fewer single-unit recording studies have so far explored the cellular mechanisms of attention operating on individual stimulus features, specifically when one feature (e.g., color) of an object must guide behavioral responses while a second feature (e.g., shape) of the same object is potentially interfering and therefore must be ignored. Here we show that activity of neurons in macaque area V4 can underlie the selection of elemental object features and their "translation" into a categorical format that can directly contribute to the control of the animal's behavior.     

The Distinct Role of the Amygdala,Superior Colliculus and Pulvinar in Processing of Central and Peripheral Snakes

Introduction

Visual processing of ecologically relevant stimuli involves a central bias for stimuli demanding detailed processing (e.g., faces), whereas peripheral object processing is based on coarse identification. Fast detection of animal shapes holding a significant phylogenetic value, such as snakes, may benefit from peripheral vision. The amygdala together with the pulvinar and the superior colliculus are implicated in an ongoing debate regarding their role in automatic and deliberate spatial processing of threat signals.

Methods

Here we tested twenty healthy participants in an fMRI task, and investigated the role of spatial demands (the main effect of central vs. peripheral vision) in the processing of fear-relevant ecological features. We controlled for stimulus dependence using true or false snakes; snake shapes or snake faces and for task constraints (implicit or explicit). The main idea justifying this double task is that amygdala and superior colliculus are involved in both automatic and controlled processes. Moreover the explicit/implicit instruction in the task with respect to emotion is not necessarily equivalent to explicit vs. implicit in the sense of endogenous vs. exogenous attention, or controlled vs. automatic processes.

Results

We found that stimulus-driven processing led to increased amygdala responses specifically to true snake shapes presented in the centre or in the peripheral left hemifield (right hemisphere). Importantly, the superior colliculus showed significantly biased and explicit central responses to snake-related stimuli. Moreover, the pulvinar, which also contains foveal representations, also showed strong central responses, extending the results of a recent single cell pulvinar study in monkeys. Similar hemispheric specialization was found across structures: increased amygdala responses occurred to true snake shapes presented to the right hemisphere, with this pattern being closely followed by the superior colliculus and the pulvinar.

Conclusion

These results show that subcortical structures containing foveal representations such as the amygdala, pulvinar and superior colliculus play distinct roles in the central and peripheral processing of snake shapes. Our findings suggest multiple phylogenetic fingerprints in the responses of subcortical structures to fear-relevant stimuli.