Three-dimensional shape representation in monkey cortex

Using fMRI in anesthetized monkeys, this study investigates how the primate visual system constructs representations of three-dimensional (3D) shape from a variety of cues. Computer-generated 3D objects defined by shading, random dots, texture elements, or silhouettes were presented either statically or dynamically (rotating). Results suggest that 3D shape representations are highly localized, although widely distributed, in occipital, temporal, parietal, and frontal cortices and may involve common brain regions regardless of shape cue. This distributed network of areas cuts across both "what" and "where" processing streams, reflecting multiple uses for 3D shape representation in perception, recognition, and action.     

Towards optimal views of proteins

MOTIVATION: Graphical representations of proteins in online databases generally give default views orthogonal to the PDB file coordinate system. These views are often uninformative in terms of protein structure and/or function. Here we discuss the development of a simple automatic algorithm to provide a 'good' view of a protein domain with respect to its structural features. RESULTS: We used dimension reduction with the preservation of topology (using Kohonen's self organising map) to map 3D carbon alpha coordinates into 2D. The original protein structure was then rotated to the view which corresponded most closely to the 2D mapping. This procedure, which we call OVOP, was evaluated in a public blind trial on the web against random views and a 'flattest' view. The OVOP views were consistently rated 'better' than the other views by our volunteers. AVAILABILITY: The source code is available from the OVOP homepage: http://www.sbc.su.se/~oscar/ovop.     

Distinct Visual Working Memory Systems for View-Dependent and View-Invariant Representation

Background

How do people sustain a visual representation of the environment? Currently, many researchers argue that a single visual working memory system sustains non-spatial object information such as colors and shapes. However, previous studies tested visual working memory for two-dimensional objects only. In consequence, the nature of visual working memory for three-dimensional (3D) object representation remains unknown.

Methodology/Principal Findings

Here, I show that when sustaining information about 3D objects, visual working memory clearly divides into two separate, specialized memory systems, rather than one system, as was previously thought. One memory system gradually accumulates sensory information, forming an increasingly precise view-dependent representation of the scene over the course of several seconds. A second memory system sustains view-invariant representations of 3D objects. The view-dependent memory system has a storage capacity of 3–4 representations and the view-invariant memory system has a storage capacity of 1–2 representations. These systems can operate independently from one another and do not compete for working memory storage resources.

Conclusions/Significance

These results provide evidence that visual working memory sustains object information in two separate, specialized memory systems. One memory system sustains view-dependent representations of the scene, akin to the view-specific representations that guide place recognition during navigation in humans, rodents and insects. The second memory system sustains view-invariant representations of 3D objects, akin to the object-based representations that underlie object cognition.     

How Lateral Connections and Spiking Dynamics May Separate Multiple Objects Moving Together

Over successive stages, the ventral visual system of the primate brain develops neurons that respond selectively to particular objects or faces with translation, size and view invariance. The powerful neural representations found in Inferotemporal cortex form a remarkably rapid and robust basis for object recognition which belies the difficulties faced by the system when learning in natural visual environments. A central issue in understanding the process of biological object recognition is how these neurons learn to form separate representations of objects from complex visual scenes composed of multiple objects. We show how a one-layer competitive network comprised of ‘spiking’ neurons is able to learn separate transformation-invariant representations (exemplified by one-dimensional translations) of visual objects that are always seen together moving in lock-step, but separated in space. This is achieved by combining ‘Mexican hat’ functional lateral connectivity with cell firing-rate adaptation to temporally segment input representations of competing stimuli through anti-phase oscillations (perceptual cycles). These spiking dynamics are quickly and reliably generated, enabling selective modification of the feed-forward connections to neurons in the next layer through Spike-Time-Dependent Plasticity (STDP), resulting in separate translation-invariant representations of each stimulus. Variations in key properties of the model are investigated with respect to the network’s ability to develop appropriate input representations and subsequently output representations through STDP. Contrary to earlier rate-coded models of this learning process, this work shows how spiking neural networks may learn about more than one stimulus together without suffering from the ‘superposition catastrophe’. We take these results to suggest that spiking dynamics are key to understanding biological visual object recognition.     

A unified model of spatial and episodic memory

Medial temporal lobe structures including the hippocampus are implicated by separate investigations in both episodic memory and spatial function. We show that a single recurrent attractor network can store both the discrete memories that characterize episodic memory and the continuous representations that characterize physical space. Combining both types of representation in a single network is actually necessary if objects and where they are located in space must be stored. We thus show that episodic memory and spatial theories of medial temporal lobe function can be combined in a unified model.     

Size-Sensitive Perceptual Representations Underlie Visual and Haptic Object Recognition

A variety of similarities between visual and haptic object recognition suggests that the two modalities may share common representations. However, it is unclear whether such common representations preserve low-level perceptual features or whether transfer between vision and haptics is mediated by high-level, abstract representations. Two experiments used a sequential shape-matching task to examine the effects of size changes on unimodal and crossmodal visual and haptic object recognition. Participants felt or saw 3D plastic models of familiar objects. The two objects presented on a trial were either the same size or different sizes and were the same shape or different but similar shapes. Participants were told to ignore size changes and to match on shape alone. In Experiment 1, size changes on same-shape trials impaired performance similarly for both visual-to-visual and haptic-to-haptic shape matching. In Experiment 2, size changes impaired performance on both visual-to-haptic and haptic-to-visual shape matching and there was no interaction between the cost of size changes and direction of transfer. Together the unimodal and crossmodal matching results suggest that the same, size-specific perceptual representations underlie both visual and haptic object recognition, and indicate that crossmodal memory for objects must be at least partly based on common perceptual representations.     

Using views of Systems Biology Cloud: application for model building

A large and growing network (“cloud”) of interlinked terms and records of items of Systems Biology knowledge is available from the web. These items include pathways, reactions, substances, literature references, organisms, and anatomy, all described in different data sets. Here, we discuss how the knowledge from the cloud can be molded into representations (views) useful for data visualization and modeling. We discuss methods to create and use various views relevant for visualization, modeling, and model annotations, while hiding irrelevant details without unacceptable loss or distortion. We show that views are compatible with understanding substances and processes as sets of microscopic compounds and events respectively, which allows the representation of specializations and generalizations as subsets and supersets respectively. We explain how these methods can be implemented based on the bridging ontology Systems Biological Pathway Exchange (SBPAX) in the Systems Biology Linker (SyBiL) we have developed.     

Invariant visual object recognition: a model, with lighting invariance.

How are invariant representations of objects formed in the visual cortex? We describe a neurophysiological and computational approach which focusses on a feature hierarchy model in which invariant representations can be built by self-organizing learning based on the statistics of the visual input. The model can use temporal continuity in an associative synaptic learning rule with a short term memory trace, and/or it can use spatial continuity in Continuous Transformation learning. The model of visual processing in the ventral cortical stream can build representations of objects that are invariant with respect to translation, view, size, and in this paper we show also lighting. The model has been extended to provide an account of invariant representations in the dorsal visual system of the global motion produced by objects such as looming, rotation, and object-based movement. The model has been extended to incorporate top-down feedback connections to model the control of attention by biased competition in for example spatial and object search tasks. The model has also been extended to account for how the visual system can select single objects in complex visual scenes, and how multiple objects can be represented in a scene.     

Towards understanding of the cortical network underlying associative memory

Declarative knowledge and experiences are represented in the association cortex and are recalled by reactivation of the neural representation. Electrophysiological experiments have revealed that associations between semantically linked visual objects are formed in neural representations in the temporal and limbic cortices. Memory traces are created by the reorganization of neural circuits. These regions are reactivated during retrieval and contribute to the contents of a memory. Two different types of retrieval signals are suggested as follows: automatic and active. One flows backward from the medial temporal lobe during the automatic retrieval process, whereas the other is conveyed as a top-down signal from the prefrontal cortex to the temporal cortex during the active retrieval process. By sending the top-down signal, the prefrontal cortex manipulates and organizes to-be-remembered information, devises strategies for retrieval and monitors the outcome. To further understand the neural mechanism of memory, the following two complementary views are needed: how the multiple cortical areas in the brain-wide network interact to orchestrate cognitive functions and how the properties of single neurons and their synaptic connections with neighbouring neurons combine to form local circuits and to exhibit the function of each cortical area. We will discuss some new methodological innovations that tackle these challenges.     

Tangible interfaces for structural molecular biology

The evolving technology of computer autofabrication makes it possible to produce physical models for complex biological molecules and assemblies. Augmented reality has recently developed as a computer interface technology that enables the mixing of real-world objects and computer-generated graphics. We report an application that demonstrates the use of autofabricated tangible models and augmented reality for research and communication in molecular biology. We have extended our molecular modeling environment, PMV, to support the fabrication of a wide variety of physical molecular models, and have adapted an augmented reality system to allow virtual 3D representations to be overlaid onto the tangible molecular models. Users can easily change the overlaid information, switching between different representations of the molecule, displays of molecular properties, or dynamic information. The physical models provide a powerful, intuitive interface for manipulating the computer models, streamlining the interface between human intent, the physical model, and the computational activity.     

Visual, haptic and cross-modal recognition of objects and scenes.

In this article we review current literature on cross-modal recognition and present new findings from our studies on object and scene recognition. Specifically, we address the questions of what is the nature of the representation underlying each sensory system that facilitates convergence across the senses and how perception is modified by the interaction of the senses. In the first set of our experiments, the recognition of unfamiliar objects within and across the visual and haptic modalities was investigated under conditions of changes in orientation (0 degrees or 180 degrees ). An orientation change increased recognition errors within each modality but this effect was reduced across modalities. Our results suggest that cross-modal object representations of objects are mediated by surface-dependent representations. In a second series of experiments, we investigated how spatial information is integrated across modalities and viewpoint using scenes of familiar, 3D objects as stimuli. We found that scene recognition performance was less efficient when there was either a change in modality, or in orientation, between learning and test. Furthermore, haptic learning was selectively disrupted by a verbal interpolation task. Our findings are discussed with reference to separate spatial encoding of visual and haptic scenes. We conclude by discussing a number of constraints under which cross-modal integration is optimal for object recognition. These constraints include the nature of the task, and the amount of spatial and temporal congruency of information across the modalities.     

Eccentricity bias as an organizing principle for human high-order object areas

We have recently proposed a center-periphery organization based on resolution needs, in which objects engaging in recognition processes requiring central-vision (e.g., face-related) are associated with center-biased representations, while objects requiring large-scale feature integration (e.g., buildings) are associated with periphery-biased representations. Here we tested this hypothesis by comparing the center-periphery organization with activations to five object categories: faces, buildings, tools, letter strings, and words. We found that faces, letter strings, and words were mapped preferentially within the center-biased representation. Faces showed a hemispheric lateralization opposite to that of letter strings and words. In contrast, buildings were mapped mainly to the periphery-biased representation, while tools activated both central and peripheral representations. The results are compatible with the notion that center-periphery organization allows the optimal allocation of cortical magnification to the specific requirements of various recognition processes.     

Deep Supervised,but Not Unsupervised,Models May Explain IT Cortical Representation

Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.     

Shaping of object representations in the human medial temporal lobe based on temporal regularities

Regularities are gradually represented in cortex after extensive experience [1], and yet they can influence behavior after minimal exposure [2, 3]. What kind of representations support such rapid statistical learning? The medial temporal lobe (MTL) can represent information from even a single experience [4], making it a good candidate system for assisting in initial learning about regularities. We combined anatomical segmentation of the MTL, high-resolution fMRI, and multivariate pattern analysis to identify representations of objects in cortical and hippocampal areas of human MTL, assessing how these representations were shaped by exposure to regularities. Subjects viewed a continuous visual stream containing hidden temporal relationships-pairs of objects that reliably appeared nearby in time. We compared the pattern of blood oxygen level-dependent activity evoked by each object before and after this exposure, and found that perirhinal cortex, parahippocampal cortex, subiculum, CA1, and CA2/CA3/dentate gyrus (CA2/3/DG) encoded regularities by increasing the representational similarity of their constituent objects. Most regions exhibited bidirectional associative shaping, whereas CA2/3/DG represented regularities in a forward-looking predictive manner. These findings suggest that object representations in MTL come to mirror the temporal structure of the environment, supporting rapid and incidental statistical learning.     

3D reconstruction of dental specimens from 2D histological images and microCT-scans

Direct comparison of experimental and theoretical results in biomechanical studies requires a careful reconstruction of specimen surfaces to achieve a satisfactory congruence for validation. In this paper a semi-automatic approach is described to reconstruct triangular boundary representations from images originating from, either histological sections or microCT-, CT- or MRI-data, respectively. In a user-guided first step, planar 2D contours were extracted for every material of interest, using image segmentation techniques. In a second step, standard 2D triangulation algorithms were used to derive high quality mesh representations of the underlying surfaces. This was accomplished by converting the 2D meshes into 3D meshes by a novel lifting procedure. The meshes can be imported as is into finite element programme packages such as Marc/Mentat or COSMOS/M. Accuracy and feasibility of the algorithm is demonstrated by reconstructing several specimens as examples and comparing simulated results with available measurements performed on the original objects.     

Learning invariant object recognition in the visual system with continuous transformations

The cerebral cortex utilizes spatiotemporal continuity in the world to help build invariant representations. In vision, these might be representations of objects. The temporal continuity typical of objects has been used in an associative learning rule with a short-term memory trace to help build invariant object representations. In this paper, we show that spatial continuity can also provide a basis for helping a system to self-organize invariant representations. We introduce a new learning paradigm “continuous transformation learning” which operates by mapping spatially similar input patterns to the same postsynaptic neurons in a competitive learning system. As the inputs move through the space of possible continuous transforms (e.g. translation, rotation, etc.), the active synapses are modified onto the set of postsynaptic neurons. Because other transforms of the same stimulus overlap with previously learned exemplars, a common set of postsynaptic neurons is activated by the new transforms, and learning of the new active inputs onto the same postsynaptic neurons is facilitated. We demonstrate that a hierarchical model of cortical processing in the ventral visual system can be trained with continuous transform learning, and highlight differences in the learning of invariant representations to those achieved by trace learning.     

Unifying Inference of Meso-Scale Structures in Networks

Networks are among the most prevalent formal representations in scientific studies, employed to depict interactions between objects such as molecules, neuronal clusters, or social groups. Studies performed at meso-scale that involve grouping of objects based on their distinctive interaction patterns form one of the main lines of investigation in network science. In a social network, for instance, meso-scale structures can correspond to isolated social groupings or groups of individuals that serve as a communication core. Currently, the research on different meso-scale structures such as community and core-periphery structures has been conducted via independent approaches, which precludes the possibility of an algorithmic design that can handle multiple meso-scale structures and deciding which structure explains the observed data better. In this study, we propose a unified formulation for the algorithmic detection and analysis of different meso-scale structures. This facilitates the investigation of hybrid structures that capture the interplay between multiple meso-scale structures and statistical comparison of competing structures, all of which have been hitherto unavailable. We demonstrate the applicability of the methodology in analyzing the human brain network, by determining the dominant organizational structure (communities) of the brain, as well as its auxiliary characteristics (core-periphery).     

Recognizing novel three-dimensional objects by summing signals from parts and views

Visually recognizing objects at different orientations and distances has been assumed to depend either on extracting from the retinal image a viewpoint-invariant, typically three-dimensional (3D) structure, such as object parts, or on mentally transforming two-dimensional (2D) views. To test how these processes might interact with each other, an experiment was performed in which observers discriminated images of novel, computer-generated, 3D objects, differing by rotations in 3D space and in the number of parts (in principle, a viewpoint-invariant, 'non-accidental' property) or in the curvature, length or angle of join of their parts (in principle, each a viewpoint-dependent, metric property), such that the discriminatory cue varied along a common physical scale. Although differences in the number of parts were more readily discriminated than differences in metric properties, they showed almost exactly the same orientation dependence. Overall, visual performance proved remarkably lawful: for both long (2 s) and short (100 ms) display durations, it could be summarized by a simple, compact equation with one term representing generalized viewpoint-invariant parts-based processing of 3D object structure, including metric structure, and another term representing structure-invariant processing of 2D views. Object discriminability was determined by summing signals from these two independent processes.     

Dynamic shape synthesis in posterior inferotemporal cortex

How does the brain synthesize low-level neural signals for simple shape parts into coherent representations of complete objects? Here, we present evidence for a dynamic process of object part integration in macaque posterior inferotemporal cortex (IT). Immediately after stimulus onset, neural responses carried information about individual object parts (simple contour fragments) only. Subsequently, information about specific multipart configurations emerged, building gradually over the course of approximately 60 ms, producing a sparser and more explicit representation of object shape. We show that this gradual transformation can be explained by a recurrent network process that effectively compares parts signals across neurons to generate inferences about multipart shape configurations.