A neural network model of semantic memory linking feature-based object representation and words

Recent theories in cognitive neuroscience suggest that semantic memory is a distributed process, which involves many cortical areas and is based on a multimodal representation of objects. The aim of this work is to extend a previous model of object representation to realize a semantic memory, in which sensory-motor representations of objects are linked with words. The model assumes that each object is described as a collection of features, coded in different cortical areas via a topological organization. Features in different objects are segmented via γ-band synchronization of neural oscillators. The feature areas are further connected with a lexical area, devoted to the representation of words. Synapses among the feature areas, and among the lexical area and the feature areas are trained via a time-dependent Hebbian rule, during a period in which individual objects are presented together with the corresponding words. Simulation results demonstrate that, during the retrieval phase, the network can deal with the simultaneous presence of objects (from sensory-motor inputs) and words (from acoustic inputs), can correctly associate objects with words and segment objects even in the presence of incomplete information. Moreover, the network can realize some semantic links among words representing objects with shared features. These results support the idea that semantic memory can be described as an integrated process, whose content is retrieved by the co-activation of different multimodal regions. In perspective, extended versions of this model may be used to test conceptual theories, and to provide a quantitative assessment of existing data (for instance concerning patients with neural deficits).     

A feature-based neurocomputational model of semantic memory

According with a featural organization of semantic memory, this work is aimed at investigating, through an attractor network, the role of different kinds of features in the representation of concepts, both in normal and neurodegenerative conditions. We implemented new synaptic learning rules in order to take into account the role of partially shared features and of distinctive features with different saliency. The model includes semantic and lexical layers, coding, respectively for object features and word-forms. Connections among nodes are strongly asymmetrical. To account for the feature saliency, asymmetrical synapses were created using Hebbian rules of potentiation and depotentiation, setting different pre-synaptic and post-synaptic thresholds. A variable post-synaptic threshold, which automatically changed to reflect the feature frequency in different concepts (i.e., how many concepts share a feature), was used to account for partially shared features. The trained network solved naming tasks and word recognition tasks very well, exploiting the different role of salient versus marginal features in concept identification. In the case of damage, superordinate concepts were preserved better than the subordinate ones. Interestingly, the degradation of salient features, but not of marginal ones, prevented object identification. The model suggests that Hebbian rules, with adjustable post-synaptic thresholds, can provide a reliable semantic representation of objects exploiting the statistics of input features.     

Binding and segmentation of multiple objects through neural oscillators inhibited by contour information

 Temporal correlation of neuronal activity has been suggested as a criterion for multiple object recognition. In this work, a two-dimensional network of simplified Wilson-Cowan oscillators is used to manage the binding and segmentation problem of a visual scene according to the connectedness Gestalt criterion. Binding is achieved via original coupling terms that link excitatory units to both excitatory and inhibitory units of adjacent neurons. These local coupling terms are time independent, i.e., they do not require Hebbian learning during the simulations. Segmentation is realized by a two-layer processing of the visual image. The first layer extracts all object contours from the image by means of “retinal cells” with an “on-center” receptive field. Information on contour is used to selectively inhibit Wilson-Cowan oscillators in the second layer, thus realizing a strong separation among neurons in different objects. Accidental synchronism between oscillations in different objects is prevented with the use of a global inhibitor, i.e., a global neuron that computes the overall activity in the Wilson-Cowan network and sends back an inhibitory signal. Simulations performed in a 50×50 neural grid with 21 different visual scenes (containing up to eight objects + background) with random initial conditions demonstrate that the network can correctly segment objects in almost 100% of cases using a single set of parameters, i.e., without the need to adjust parameters from one visual scene to the next. The network is robust with reference to dynamical noise superimposed on oscillatory neurons. Moreover, the network can segment both black objects on white background and vice versa and is able to deal with the problem of “fragmentation.” The main limitation of the network is its sensitivity to static noise superimposed on the objects. Overcoming this problem requires implementation of more robust mechanisms for contour enhancement in the first layer in agreement with mechanisms actually realized in the visual cortex. Received: 25 October 2001 / Accepted: 26 February 2003 / Published online: 20 May 2003 Correspondence to: Mauro Ursino (e-mail: mursino@deis.unibo.it, Tel.: +39-051-2093008, Fax: +39-051-2093073)     

Coding the presence of visual objects in a recurrent neural network of visual cortex

Before we can recognize a visual object, our visual system has to segregate it from its background. This requires a fast mechanism for establishing the presence and location of objects independently of their identity. Recently, border-ownership neurons were recorded in monkey visual cortex which might be involved in this task [Zhou, H., Friedmann, H., von der Heydt, R., 2000. Coding of border ownership in monkey visual cortex. J. Neurosci. 20 (17), 6594-6611]. In order to explain the basic mechanisms required for fast coding of object presence, we have developed a neural network model of visual cortex consisting of three stages. Feed-forward and lateral connections support coding of Gestalt properties, including similarity, good continuation, and convexity. Neurons of the highest area respond to the presence of an object and encode its position, invariant of its form. Feedback connections to the lowest area facilitate orientation detectors activated by contours belonging to potential objects, and thus generate the experimentally observed border-ownership property. This feedback control acts fast and significantly improves the figure-ground segregation required for the consecutive task of object recognition.     

To Perceive or Not Perceive: The Role of Gamma-band Activity in Signaling Object Percepts

The relation of gamma-band synchrony to holistic perception in which concerns the effects of sensory processing, high level perceptual gestalt formation, motor planning and response is still controversial. To provide a more direct link to emergent perceptual states we have used holistic EEG/ERP paradigms where the moment of perceptual “discovery” of a global pattern was variable. Using a rapid visual presentation of short-lived Mooney objects we found an increase of gamma-band activity locked to perceptual events. Additional experiments using dynamic Mooney stimuli showed that gamma activity increases well before the report of an emergent holistic percept. To confirm these findings in a data driven manner we have further used a support vector machine classification approach to distinguish between perceptual vs. non perceptual states, based on time-frequency features. Sensitivity, specificity and accuracy were all above 95%. Modulations in the 30–75 Hz range were larger for perception states. Interestingly, phase synchrony was larger for perception states for high frequency bands. By focusing on global gestalt mechanisms instead of local processing we conclude that gamma-band activity and synchrony provide a signature of holistic perceptual states of variable onset, which are separable from sensory and motor processing.     

Ant clustering with locally weighted ant perception and diversified memory

Ant clustering algorithms are a robust and flexible tool for clustering data that have produced some promising results. This paper introduces two improvements that can be incorporated into any ant clustering algorithm: kernel function similarity weights and a similarity memory model replacement scheme. A kernel function weights objects within an ant’s neighborhood according to the object distance and provides an alternate interpretation of the similarity of objects in an ant’s neighborhood. Ants can hill-climb the kernel gradients as they look for a suitable place to drop a carried object. The similarity memory model equips ants with a small memory consisting of a sampling of the current clustering space. We test several kernel functions and memory replacement schemes on the Iris, Wisconsin Breast Cancer, and Lincoln Lab network intrusion datasets. Compared to a basic ant clustering algorithm, we show that kernel functions and the similarity memory model increase clustering speed and cluster quality, especially for datasets with an unbalanced class distribution, such as network intrusion.     

Cortical mechanisms for trans-saccadic memory and integration of multiple object features

Constructing an internal representation of the world from successive visual fixations, i.e. separated by saccadic eye movements, is known as trans-saccadic perception. Research on trans-saccadic perception (TSP) has been traditionally aimed at resolving the problems of memory capacity and visual integration across saccades. In this paper, we review this literature on TSP with a focus on research showing that egocentric measures of the saccadic eye movement can be used to integrate simple object features across saccades, and that the memory capacity for items retained across saccades, like visual working memory, is restricted to about three to four items. We also review recent transcranial magnetic stimulation experiments which suggest that the right parietal eye field and frontal eye fields play a key functional role in spatial updating of objects in TSP. We conclude by speculating on possible cortical mechanisms for governing egocentric spatial updating of multiple objects in TSP.     

Eccentricity bias as an organizing principle for human high-order object areas

We have recently proposed a center-periphery organization based on resolution needs, in which objects engaging in recognition processes requiring central-vision (e.g., face-related) are associated with center-biased representations, while objects requiring large-scale feature integration (e.g., buildings) are associated with periphery-biased representations. Here we tested this hypothesis by comparing the center-periphery organization with activations to five object categories: faces, buildings, tools, letter strings, and words. We found that faces, letter strings, and words were mapped preferentially within the center-biased representation. Faces showed a hemispheric lateralization opposite to that of letter strings and words. In contrast, buildings were mapped mainly to the periphery-biased representation, while tools activated both central and peripheral representations. The results are compatible with the notion that center-periphery organization allows the optimal allocation of cortical magnification to the specific requirements of various recognition processes.     

The Time Course of Activation of Object Shape and Shape+Colour Representations during Memory Retrieval

Little is known about the timing of activating memory for objects and their associated perceptual properties, such as colour, and yet this is important for theories of human cognition. We investigated the time course associated with early cognitive processes related to the activation of object shape and object shape+colour representations respectively, during memory retrieval as assessed by repetition priming in an event-related potential (ERP) study. The main findings were as follows: (1) we identified a unique early modulation of mean ERP amplitude during the N1 that was associated with the activation of object shape independently of colour; (2) we also found a subsequent early P2 modulation of mean amplitude over the same electrode clusters associated with the activation of object shape+colour representations; (3) these findings were apparent across both familiar (i.e., correctly coloured – yellow banana) and novel (i.e., incorrectly coloured - blue strawberry) objects; and (4) neither of the modulations of mean ERP amplitude were evident during the P3. Together the findings delineate the timing of object shape and colour memory systems and support the notion that perceptual representations of object shape mediate the retrieval of temporary shape+colour representations for familiar and novel objects.     

Shape Similarity,Better than Semantic Membership,Accounts for the Structure of Visual Object Representations in a Population of Monkey Inferotemporal Neurons

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex.     

Representing occluded objects in the human infant brain

One of the most striking phenomena in cognitive development has been the apparent failure of infants to show 'object permanence' in manual reaching tasks although they show evidence for representing hidden objects in studies measuring looking times. We report a neural correlate of object permanence in six-month-old infants: a burst of gamma-band EEG activity over the temporal lobe that occurs during an occlusion event and when an object is expected to appear from behind an occluder. We interpret this burst as being related to the infants' mental representation of the occluded object.     

A real-world size organization of object responses in occipitotemporal cortex

While there are selective regions of occipitotemporal cortex that respond to faces, letters, and bodies, the large-scale neural organization of most object categories remains unknown. Here, we find that object representations can be differentiated along the ventral temporal cortex by their real-world size. In a functional neuroimaging experiment, observers were shown pictures of big and small real-world objects (e.g., table, bathtub; paperclip, cup), presented at the same retinal size. We observed a consistent medial-to-lateral organization of big and small object preferences in the ventral temporal cortex, mirrored along the lateral surface. Regions in the lateral-occipital, inferotemporal, and parahippocampal cortices showed strong peaks of differential real-world size selectivity and maintained these preferences over changes in retinal size and in mental imagery. These data demonstrate that the real-world size of objects can provide insight into the spatial topography of object representation.     

Directed cortical information flow during human object recognition: analyzing induced EEG gamma-band responses in brain's source space

The increase of induced gamma-band responses (iGBRs; oscillations >30 Hz) elicited by familiar (meaningful) objects is well established in electroencephalogram (EEG) research. This frequency-specific change at distinct locations is thought to indicate the dynamic formation of local neuronal assemblies during the activation of cortical object representations. As analytically power increase is just a property of a single location, phase-synchrony was introduced to investigate the formation of large-scale networks between spatially distant brain sites. However, classical phase-synchrony reveals symmetric, pair-wise correlations and is not suited to uncover the directionality of interactions. Here, we investigated the neural mechanism of visual object processing by means of directional coupling analysis going beyond recording sites, but rather assessing the directionality of oscillatory interactions between brain areas directly. This study is the first to identify the directionality of oscillatory brain interactions in source space during human object recognition and suggests that familiar, but not unfamiliar, objects engage widespread reciprocal information flow. Directionality of cortical information-flow was calculated based upon an established Granger-Causality coupling-measure (partial-directed coherence; PDC) using autoregressive modeling. To enable comparison with previous coupling studies lacking directional information, phase-locking analysis was applied, using wavelet-based signal decompositions. Both, autoregressive modeling and wavelet analysis, revealed an augmentation of iGBRs during the presentation of familiar objects relative to unfamiliar controls, which was localized to inferior-temporal, superior-parietal and frontal brain areas by means of distributed source reconstruction. The multivariate analysis of PDC evaluated each possible direction of brain interaction and revealed widespread reciprocal information-transfer during familiar object processing. In contrast, unfamiliar objects entailed a sparse number of only unidirectional connections converging to parietal areas. Considering the directionality of brain interactions, the current results might indicate that successful activation of object representations is realized through reciprocal (feed-forward and feed-backward) information-transfer of oscillatory connections between distant, functionally specific brain areas.     

Feature-Specific Encoding Flexibility in Visual Working Memory

The current study examined selective encoding in visual working memory by systematically investigating interference from task-irrelevant features. The stimuli were objects defined by three features (color, shape, and location), and during a delay period, any of the features could switch between two objects. Additionally, single- and whole-probe trials were randomized within experimental blocks to investigate effects of memory retrieval. A series of relevant-feature switch detection tasks, where one feature was task-irrelevant, showed that interference from the task-irrelevant feature was only observed in the color-shape task, suggesting that color and shape information could be successfully filtered out, but location information could not, even when location was a task-irrelevant feature. Therefore, although location information is added to object representations independent of task demands in a relatively automatic manner, other features (e.g., color, shape) can be flexibly added to object representations.     

Associatieve visuele agnosie. De minder zichtbare gevolgen van een herseninfarct

After a cerebral infarction, some patients acutely demonstrate contralateral hemiplegia, or aphasia. Those are the obvious symptoms of a cerebral infarction. However, less visible but burdensome consequences may go unnoticed without closer investigation. The importance of a thorough clinical examination is exemplified by a single case study of a 72-year-old, right-handed male. Two years before he had suffered from an ischemic stroke in the territory of the left posterior cerebral artery, with right homonymous hemianopia and global alexia (i.e., impairment in letter recognition and profound impairment of reading) without agraphia. Naming was impaired on visual presentation (20%-39% correct), but improved significantly after tactile presentation (87% correct) or verbal definition (89%). Pre-semantic visual processing was normal (correct matching of different views of the same object), as was his access to structural knowledge from vision (he reliably distinguished real objects from non-objects). On a colour decision task he reliably indicated which of two items was coloured correctly. Though he was unable to mime how visually presented objects were used, he more reliably matched pictures of objects with pictures of a mime artist gesturing the use of the object. He obtained normal scores on word definition (WAIS-III), synonym judgment and word-picture matching tasks with perceptual and semantic distractors. He however failed when he had to match physically dissimilar specimens of the same object or when he had to decide which two of five objects were related associatively (Pyramids and Palm Trees Test). The patient thus showed a striking contrast in his intact ability to access knowledge of object shape or colour from vision and impaired functional and associative knowledge. As a result, he could not access a complete semantic representation, required for activating phonological representations to name visually presented objects. The pattern of impairments and preserved abilities is considered to be a specific difficulty to access a full semantic representation from an intact structural representation of visually presented objects, i.e., a form of visual object agnosia.     

Neuromodulatory Control of Neocortical Microcircuits with Activity-Dependent Short-Term Synaptic Depression

A biophysical model of a neocortical microcircuit system is formulated and employed in studies of neuromodulatory control of dynamics and function. The model is based on recent observations of reciprocal connections between pyramidal cells and inhibitory interneurons and incorporates a new type of activity-dependent short-term depression of synaptic couplings recently observed. The model neurons are of a low-dimensional type also accounting for neuronal adaptation, i.e. the coupling between neuronal activity and excitability, which can be regulated by various neuromodulators in the brain. The results obtained demonstrate a capacity for neuromodulatory control of dynamical mode linked to functional mode. The functional aspects considered refer to the observed resolution of multiple objects in working memory as well as the binding of different features for the perception of an object. The effects of neuromodulators displayed by the model are in accordance with many observations on neuromodulatory influence on cognitive functions and brain disorders.     

Bats' avoidance of real and virtual objects: implications for the sonar coding of object size

Fast movement in complex environments requires the controlled evasion of obstacles. Sonar-based obstacle evasion involves analysing the acoustic features of object-echoes (e.g., echo amplitude) that correlate with this object's physical features (e.g., object size). Here, we investigated sonar-based obstacle evasion in bats emerging in groups from their day roost. Using video-recordings, we first show that the bats evaded a small real object (ultrasonic loudspeaker) despite the familiar flight situation. Secondly, we studied the sonar coding of object size by adding a larger virtual object. The virtual object echo was generated by real-time convolution of the bats’ calls with the acoustic impulse response of a large spherical disc and played from the loudspeaker. Contrary to the real object, the virtual object did not elicit evasive flight, despite the spectro-temporal similarity of real and virtual object echoes. Yet, their spatial echo features differ: virtual object echoes lack the spread of angles of incidence from which the echoes of large objects arrive at a bat's ears (sonar aperture). We hypothesise that this mismatch of spectro-temporal and spatial echo features caused the lack of virtual object evasion and suggest that the sonar aperture of object echoscapes contributes to the sonar coding of object size.     

Distinct Visual Working Memory Systems for View-Dependent and View-Invariant Representation

Background

How do people sustain a visual representation of the environment? Currently, many researchers argue that a single visual working memory system sustains non-spatial object information such as colors and shapes. However, previous studies tested visual working memory for two-dimensional objects only. In consequence, the nature of visual working memory for three-dimensional (3D) object representation remains unknown.

Methodology/Principal Findings

Here, I show that when sustaining information about 3D objects, visual working memory clearly divides into two separate, specialized memory systems, rather than one system, as was previously thought. One memory system gradually accumulates sensory information, forming an increasingly precise view-dependent representation of the scene over the course of several seconds. A second memory system sustains view-invariant representations of 3D objects. The view-dependent memory system has a storage capacity of 3–4 representations and the view-invariant memory system has a storage capacity of 1–2 representations. These systems can operate independently from one another and do not compete for working memory storage resources.

Conclusions/Significance

These results provide evidence that visual working memory sustains object information in two separate, specialized memory systems. One memory system sustains view-dependent representations of the scene, akin to the view-specific representations that guide place recognition during navigation in humans, rodents and insects. The second memory system sustains view-invariant representations of 3D objects, akin to the object-based representations that underlie object cognition.     

Interaction between Attention and Bottom-Up Saliency Mediates the Representation of Foreground and Background in an Auditory Scene

The mechanism by which a complex auditory scene is parsed into coherent objects depends on poorly understood interactions between task-driven and stimulus-driven attentional processes. We illuminate these interactions in a simultaneous behavioral–neurophysiological study in which we manipulate participants' attention to different features of an auditory scene (with a regular target embedded in an irregular background). Our experimental results reveal that attention to the target, rather than to the background, correlates with a sustained (steady-state) increase in the measured neural target representation over the entire stimulus sequence, beyond auditory attention's well-known transient effects on onset responses. This enhancement, in both power and phase coherence, occurs exclusively at the frequency of the target rhythm, and is only revealed when contrasting two attentional states that direct participants' focus to different features of the acoustic stimulus. The enhancement originates in auditory cortex and covaries with both behavioral task and the bottom-up saliency of the target. Furthermore, the target's perceptual detectability improves over time, correlating strongly, within participants, with the target representation's neural buildup. These results have substantial implications for models of foreground/background organization, supporting a role of neuronal temporal synchrony in mediating auditory object formation.