Distinct Visual Working Memory Systems for View-Dependent and View-Invariant Representation

Background

How do people sustain a visual representation of the environment? Currently, many researchers argue that a single visual working memory system sustains non-spatial object information such as colors and shapes. However, previous studies tested visual working memory for two-dimensional objects only. In consequence, the nature of visual working memory for three-dimensional (3D) object representation remains unknown.

Methodology/Principal Findings

Here, I show that when sustaining information about 3D objects, visual working memory clearly divides into two separate, specialized memory systems, rather than one system, as was previously thought. One memory system gradually accumulates sensory information, forming an increasingly precise view-dependent representation of the scene over the course of several seconds. A second memory system sustains view-invariant representations of 3D objects. The view-dependent memory system has a storage capacity of 3–4 representations and the view-invariant memory system has a storage capacity of 1–2 representations. These systems can operate independently from one another and do not compete for working memory storage resources.

Conclusions/Significance

These results provide evidence that visual working memory sustains object information in two separate, specialized memory systems. One memory system sustains view-dependent representations of the scene, akin to the view-specific representations that guide place recognition during navigation in humans, rodents and insects. The second memory system sustains view-invariant representations of 3D objects, akin to the object-based representations that underlie object cognition.     

Categorical judgments do not modify sensory representations in working memory

Categorical judgments can systematically bias the perceptual interpretation of stimulus features. However, it remained unclear whether categorical judgments directly modify working memory representations or, alternatively, generate these biases via an inference process down-stream from working memory. To address this question we ran two novel psychophysical experiments in which human subjects had to reverse their categorical judgments about a stimulus feature, if incorrect, before providing an estimate of the feature. If categorical judgments indeed directly altered sensory representations in working memory, subjects’ estimates should reflect some aspects of their initial (incorrect) categorical judgment in those trials. We found no traces of the initial categorical judgment. Rather, subjects seemed to be able to flexibly switch their categorical judgment if needed and use the correct corresponding categorical prior to properly perform feature inference. A cross-validated model comparison also revealed that feedback may lead to selective memory recall such that only memory samples that are consistent with the categorical judgment are accepted for the inference process. Our results suggest that categorical judgments do not modify sensory information in working memory but rather act as top-down expectations in the subsequent sensory recall and inference process.     

Bistable Perception Modeled as Competing Stochastic Integrations at Two Levels

We propose a novel explanation for bistable perception, namely, the collective dynamics of multiple neural populations that are individually meta-stable. Distributed representations of sensory input and of perceptual state build gradually through noise-driven transitions in these populations, until the competition between alternative representations is resolved by a threshold mechanism. The perpetual repetition of this collective race to threshold renders perception bistable. This collective dynamics – which is largely uncoupled from the time-scales that govern individual populations or neurons – explains many hitherto puzzling observations about bistable perception: the wide range of mean alternation rates exhibited by bistable phenomena, the consistent variability of successive dominance periods, and the stabilizing effect of past perceptual states. It also predicts a number of previously unsuspected relationships between observable quantities characterizing bistable perception. We conclude that bistable perception reflects the collective nature of neural decision making rather than properties of individual populations or neurons.     

To Perceive or Not Perceive: The Role of Gamma-band Activity in Signaling Object Percepts

The relation of gamma-band synchrony to holistic perception in which concerns the effects of sensory processing, high level perceptual gestalt formation, motor planning and response is still controversial. To provide a more direct link to emergent perceptual states we have used holistic EEG/ERP paradigms where the moment of perceptual “discovery” of a global pattern was variable. Using a rapid visual presentation of short-lived Mooney objects we found an increase of gamma-band activity locked to perceptual events. Additional experiments using dynamic Mooney stimuli showed that gamma activity increases well before the report of an emergent holistic percept. To confirm these findings in a data driven manner we have further used a support vector machine classification approach to distinguish between perceptual vs. non perceptual states, based on time-frequency features. Sensitivity, specificity and accuracy were all above 95%. Modulations in the 30–75 Hz range were larger for perception states. Interestingly, phase synchrony was larger for perception states for high frequency bands. By focusing on global gestalt mechanisms instead of local processing we conclude that gamma-band activity and synchrony provide a signature of holistic perceptual states of variable onset, which are separable from sensory and motor processing.     

The Time Course of Activation of Object Shape and Shape+Colour Representations during Memory Retrieval

Little is known about the timing of activating memory for objects and their associated perceptual properties, such as colour, and yet this is important for theories of human cognition. We investigated the time course associated with early cognitive processes related to the activation of object shape and object shape+colour representations respectively, during memory retrieval as assessed by repetition priming in an event-related potential (ERP) study. The main findings were as follows: (1) we identified a unique early modulation of mean ERP amplitude during the N1 that was associated with the activation of object shape independently of colour; (2) we also found a subsequent early P2 modulation of mean amplitude over the same electrode clusters associated with the activation of object shape+colour representations; (3) these findings were apparent across both familiar (i.e., correctly coloured – yellow banana) and novel (i.e., incorrectly coloured - blue strawberry) objects; and (4) neither of the modulations of mean ERP amplitude were evident during the P3. Together the findings delineate the timing of object shape and colour memory systems and support the notion that perceptual representations of object shape mediate the retrieval of temporary shape+colour representations for familiar and novel objects.     

The body beyond the body: expectation of a sensory event is enough to induce ownership over a fake hand

More than 100 papers have been published on the rubber hand illusion since its discovery 14 years ago. The illusion has been proposed as a demonstration that the body is distinguished from other objects by its participation in specific forms of intermodal perceptual correlation. Here, we radically challenge this view by claiming that perceptual correlation is not necessary to produce the experience of this body as mine. Each of 15 participants was seated with his/her right arm resting upon a table just below another smaller table. Thus, the real hand was hidden from the participant''s view and a life-sized rubber model of a right hand was placed on the small table in front of the participant. The participant observed the experimenter''s hand while approaching—without touching—the rubber hand. Phenomenology of the illusion was measured by means of skin conductance response and questionnaire. Both measures indicated that participants experienced the illusion that the experimenter''s hand was about to touch their hidden hand rather than the rubber hand, as if the latter replaced their own hand. This did not occur when the rubber hand was rotated by 180° or replaced by a piece of wood. This illusion indicates that our brain does not build a sense of self in a merely reactive way, via perceptual correlations; rather it generates predictions on what may or may not belong to itself.     

Hippocampus Is Place of Interaction between Unconscious and Conscious Memories

Recent evidence suggests that humans can form and later retrieve new semantic relations unconsciously by way of hippocampus—the key structure also recruited for conscious relational (episodic) memory. If the hippocampus subserves both conscious and unconscious relational encoding/retrieval, one would expect the hippocampus to be place of unconscious-conscious interactions during memory retrieval. We tested this hypothesis in an fMRI experiment probing the interaction between the unconscious and conscious retrieval of face-associated information. For the establishment of unconscious relational memories, we presented subliminal (masked) combinations of unfamiliar faces and written occupations (“actor” or “politician”). At test, we presented the former subliminal faces, but now supraliminally, as cues for the reactivation of the unconsciously associated occupations. We hypothesized that unconscious reactivation of the associated occupation—actor or politician—would facilitate or inhibit the subsequent conscious retrieval of a celebrity’s occupation, which was also actor or politician. Depending on whether the reactivated unconscious occupation was congruent or incongruent to the celebrity’s occupation, we expected either quicker or delayed conscious retrieval process. Conscious retrieval was quicker in the congruent relative to a neutral baseline condition but not delayed in the incongruent condition. fMRI data collected during subliminal face-occupation encoding confirmed previous evidence that the hippocampus was interacting with neocortical storage sites of semantic knowledge to support relational encoding. fMRI data collected at test revealed that the facilitated conscious retrieval was paralleled by deactivations in the hippocampus and neocortical storage sites of semantic knowledge. We assume that the unconscious reactivation has pre-activated overlapping relational representations in the hippocampus reducing the neural effort for conscious retrieval. This finding supports the notion of synergistic interactions between conscious and unconscious relational memories in a common, cohesive hippocampal-neocortical memory space.     

Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex

Whether cognitive representations are better conceived as language-based, symbolic representations or perceptually related, analog representations is a subject of debate. If cognitive processes parallel perceptual processes, then fundamental psychophysical laws should hold for each. To test this, we analyzed both behavioral and neuronal representations of numerosity in the prefrontal cortex of rhesus monkeys. The data were best described by a nonlinearly compressed scaling of numerical information, as postulated by the Weber-Fechner law or Stevens' law for psychophysical/sensory magnitudes. This nonlinear compression was observed on the neural level during the acquisition phase of the task and maintained through the memory phase with no further compression. These results suggest that certain cognitive and perceptual/sensory representations share the same fundamental mechanisms and neural coding schemes.     

Multisensory Oddity Detection as Bayesian Inference

A key goal for the perceptual system is to optimally combine information from all the senses that may be available in order to develop the most accurate and unified picture possible of the outside world. The contemporary theoretical framework of ideal observer maximum likelihood integration (MLI) has been highly successful in modelling how the human brain combines information from a variety of different sensory modalities. However, in various recent experiments involving multisensory stimuli of uncertain correspondence, MLI breaks down as a successful model of sensory combination. Within the paradigm of direct stimulus estimation, perceptual models which use Bayesian inference to resolve correspondence have recently been shown to generalize successfully to these cases where MLI fails. This approach has been known variously as model inference, causal inference or structure inference. In this paper, we examine causal uncertainty in another important class of multi-sensory perception paradigm – that of oddity detection and demonstrate how a Bayesian ideal observer also treats oddity detection as a structure inference problem. We validate this approach by showing that it provides an intuitive and quantitative explanation of an important pair of multi-sensory oddity detection experiments – involving cues across and within modalities – for which MLI previously failed dramatically, allowing a novel unifying treatment of within and cross modal multisensory perception. Our successful application of structure inference models to the new ‘oddity detection’ paradigm, and the resultant unified explanation of across and within modality cases provide further evidence to suggest that structure inference may be a commonly evolved principle for combining perceptual information in the brain.     

The relationships between working memory and long-term memory

Recent studies have led to the proposal that working memory operates not as a gateway between sensory input and long-term memory but as a workspace. The core of argument is that access to acquired knowledge and prior learning occurs before information becomes available to working memory. This proposition is a way to accomodate Baddeley's multiple component working memory model and the view that considers that working memory is nothing other than temporary activations of representations and procedures in long-term memory. However, this ‘workspace’ conception of working memory raises the question of the relationships between the central executive system and long-term memory.     

A Hierarchy of Time-Scales and the Brain

In this paper, we suggest that cortical anatomy recapitulates the temporal hierarchy that is inherent in the dynamics of environmental states. Many aspects of brain function can be understood in terms of a hierarchy of temporal scales at which representations of the environment evolve. The lowest level of this hierarchy corresponds to fast fluctuations associated with sensory processing, whereas the highest levels encode slow contextual changes in the environment, under which faster representations unfold. First, we describe a mathematical model that exploits the temporal structure of fast sensory input to track the slower trajectories of their underlying causes. This model of sensory encoding or perceptual inference establishes a proof of concept that slowly changing neuronal states can encode the paths or trajectories of faster sensory states. We then review empirical evidence that suggests that a temporal hierarchy is recapitulated in the macroscopic organization of the cortex. This anatomic-temporal hierarchy provides a comprehensive framework for understanding cortical function: the specific time-scale that engages a cortical area can be inferred by its location along a rostro-caudal gradient, which reflects the anatomical distance from primary sensory areas. This is most evident in the prefrontal cortex, where complex functions can be explained as operations on representations of the environment that change slowly. The framework provides predictions about, and principled constraints on, cortical structure–function relationships, which can be tested by manipulating the time-scales of sensory input.     

The role of prefrontal cortex in working memory: examining the contents of consciousness.

Working memory enables us to hold in our ''mind''s eye'' the contents of our conscious awareness, even in the absence of sensory input, by maintaining an active representation of information for a brief period of time. In this review we consider the functional organization of the prefrontal cortex and its role in this cognitive process. First, we present evidence from brain-imaging studies that prefrontal cortex shows sustained activity during the delay period of visual working memory tasks, indicating that this cortex maintains on-line representations of stimuli after they are removed from view. We then present evidence for domain specificity within frontal cortex based on the type of information, with object working memory mediated by more ventral frontal regions and spatial working memory mediated by more dorsal frontal regions. We also propose that a second dimension for domain specificity within prefrontal cortex might exist for object working memory on the basis of the type of representation, with analytic representations maintained preferentially in the left hemisphere and image-based representations maintained preferentially in the right hemisphere. Furthermore, we discuss the possibility that there are prefrontal areas brought into play during the monitoring and manipulation of information in working memory in addition to those engaged during the maintenance of this information. Finally, we consider the relationship of prefrontal areas important for working memory, both to posterior visual processing areas and to prefrontal areas associated with long-term memory.     

From Sensory Signals to Modality-Independent Conceptual Representations: A Probabilistic Language of Thought Approach

People learn modality-independent, conceptual representations from modality-specific sensory signals. Here, we hypothesize that any system that accomplishes this feat will include three components: a representational language for characterizing modality-independent representations, a set of sensory-specific forward models for mapping from modality-independent representations to sensory signals, and an inference algorithm for inverting forward models—that is, an algorithm for using sensory signals to infer modality-independent representations. To evaluate this hypothesis, we instantiate it in the form of a computational model that learns object shape representations from visual and/or haptic signals. The model uses a probabilistic grammar to characterize modality-independent representations of object shape, uses a computer graphics toolkit and a human hand simulator to map from object representations to visual and haptic features, respectively, and uses a Bayesian inference algorithm to infer modality-independent object representations from visual and/or haptic signals. Simulation results show that the model infers identical object representations when an object is viewed, grasped, or both. That is, the model’s percepts are modality invariant. We also report the results of an experiment in which different subjects rated the similarity of pairs of objects in different sensory conditions, and show that the model provides a very accurate account of subjects’ ratings. Conceptually, this research significantly contributes to our understanding of modality invariance, an important type of perceptual constancy, by demonstrating how modality-independent representations can be acquired and used. Methodologically, it provides an important contribution to cognitive modeling, particularly an emerging probabilistic language-of-thought approach, by showing how symbolic and statistical approaches can be combined in order to understand aspects of human perception.     

The Composite Task Reveals Stronger Holistic Processing in Children than Adults for Child Faces

Background

While own-age faces have been reported to be better recognized than other-age faces, the underlying cause of this phenomenon remains unclear. One potential cause is holistic face processing, a special kind of perceptual and cognitive processing reserved for perceiving upright faces. Previous studies have indeed found that adults show stronger holistic processing when looking at adult faces compared to child faces, but whether a similar own-age bias exists in children remains to be shown.

Methodology/Principal Findings

Here we used the composite face task – a standard test of holistic face processing – to investigate if, for child faces, holistic processing is stronger for children than adults. Results showed child participants (8–13 years) had a larger composite effect than adult participants (22–65 years).

Conclusions/Significance

Our finding suggests that differences in strength of holistic processing may underlie the own-age bias on recognition memory. We discuss the origin of own-age biases in terms of relative experience, face-space tuning, and social categorization.     

Vibrotactile Sensory Substitution Elicits Feeling of Ownership of an Alien Hand

Tactile feedback plays a key role in the attribution of a limb to the self and in the motor control of grasping and manipulation. However, due to technological limits, current prosthetic hands do not provide amputees with cutaneous touch feedback. Recent findings showed that amputees can be tricked into experiencing an alien rubber hand as part of their own body, by applying synchronous touches to the stump which is out of view, and to the rubber hand in full view. It was suggested that similar effects could be achieved by using a prosthesis with touch sensors that provides synchronous cutaneous feedback through an array of tactile stimulators on the stump. Such a prosthesis holds the potential to be easily incorporated within one’s body scheme, because it would reproduce the perceptual illusion in everyday usage. We propose to use sensory substitution – specifically vibrotactile – to address this issue, as current haptic technology is still too bulky and inefficient. In this basic study we addressed the fundamental question of whether visuo-tactile modality mismatch promotes self-attribution of a limb, and to what extent compared to a modality-matched paradigm, on normally-limbed subjects. We manipulated visuo-tactile stimulations, comprising combinations of modality matched, modality mismatched, synchronous and asynchronous stimulations, in a set of experiments fashioned after the Rubber Hand Illusion. Modality mismatched stimulation was provided using a keypad-controlled vibrotactile display. Results from three independent measures of embodiment (questionnaires, pointing tests and skin conductance responses) indicate that vibrotactile sensory substitution can be used to induce self-attribution of a rubber hand when synchronous but modality-conflicting visuo-tactile stimulation is delivered to the biological finger pads and to the equivalent rubber hand phalanges.     

Effects of Acute Cortisol Administration on Perceptual Priming of Trauma-Related Material

Intrusive memories are a hallmark symptom of posttraumatic stress disorder (PTSD). They reflect excessive and uncontrolled retrieval of the traumatic memory. Acute elevations of cortisol are known to impair the retrieval of already stored memory information. Thus, continuous cortisol administration might help in reducing intrusive memories in PTSD. Strong perceptual priming for neutral stimuli associated with a “traumatic” context has been shown to be one important learning mechanism that leads to intrusive memories. However, the memory modulating effects of cortisol have only been shown for explicit declarative memory processes. Thus, in our double blind, placebo controlled study we aimed to investigate whether cortisol influences perceptual priming of neutral stimuli that appeared in a “traumatic” context. Two groups of healthy volunteers (N = 160) watched either neutral or “traumatic” picture stories on a computer screen. Neutral objects were presented in between the pictures. Memory for these neutral objects was tested after 24 hours with a perceptual priming task and an explicit memory task. Prior to memory testing half of the participants in each group received 25 mg of cortisol, the other half received placebo. In the placebo group participants in the “traumatic” stories condition showed more perceptual priming for the neutral objects than participants in the neutral stories condition, indicating a strong perceptual priming effect for neutral stimuli presented in a “traumatic” context. In the cortisol group this effect was not present: Participants in the neutral stories and participants in the “traumatic” stories condition in the cortisol group showed comparable priming effects for the neutral objects. Our findings show that cortisol inhibits perceptual priming for neutral stimuli that appeared in a “traumatic” context. These findings indicate that cortisol influences PTSD-relevant memory processes and thus further support the idea that administration of cortisol might be an effective treatment strategy in reducing intrusive reexperiencing.     

Priority-based transformations of stimulus representation in visual working memory

How does the brain prioritize among the contents of working memory (WM) to appropriately guide behavior? Previous work, employing inverted encoding modeling (IEM) of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) datasets, has shown that unprioritized memory items (UMI) are actively represented in the brain, but in a “flipped”, or opposite, format compared to prioritized memory items (PMI). To acquire independent evidence for such a priority-based representational transformation, and to explore underlying mechanisms, we trained recurrent neural networks (RNNs) with a long short-term memory (LSTM) architecture to perform a 2-back WM task. Visualization of LSTM hidden layer activity using Principal Component Analysis (PCA) confirmed that stimulus representations undergo a representational transformation–consistent with a flip—while transitioning from the functional status of UMI to PMI. Demixed (d)PCA of the same data identified two representational trajectories, one each within a UMI subspace and a PMI subspace, both undergoing a reversal of stimulus coding axes. dPCA of data from an EEG dataset also provided evidence for priority-based transformations of the representational code, albeit with some differences. This type of transformation could allow for retention of unprioritized information in WM while preventing it from interfering with concurrent behavior. The results from this initial exploration suggest that the algorithmic details of how this transformation is carried out by RNNs, versus by the human brain, may differ.     

Context sensitivity in action decreases along the autism spectrum: a predictive processing perspective

Recent predictive processing accounts of perception and action point towards a key challenge for the nervous system in dynamically optimizing the balance between incoming sensory information and existing expectations regarding the state of the environment. Here, we report differences in the influence of the preceding sensory context on motor function, varying with respect to both clinical and subclinical features of autism spectrum disorder (ASD). Reach-to-grasp movements were recorded subsequent to an inactive period in which illusory ownership of a prosthetic limb was induced. We analysed the sub-components of reach trajectories derived using a minimum-jerk fitting procedure. Non-clinical adults low in autistic features showed disrupted movement execution following the illusion compared to a control condition. By contrast, individuals higher in autistic features (both those with ASD and non-clinical individuals high in autistic traits) showed reduced sensitivity to the presence of the illusion in their reaching movements while still exhibiting the typical perceptual effects of the illusion. Clinical individuals were distinct from non-clinical individuals scoring high in autistic features, however, in the early stages of movement. These results suggest that the influence of high-level representations of the environment differs between individuals, contributing to clinical and subclinical differences in motor performance that manifest in a contextual manner. As high-level representations of context help to explain fluctuations in sensory input over relatively longer time scales, more circumscribed sensitivity to prior or contextual information in autistic sensory processing could contribute more generally to reduced social comprehension, sensory impairments and a stronger desire for predictability and routine.     

Mental Summation of Temporal Duration within and across Senses

Perceiving, memorizing, and estimating temporal durations are key cognitive functions in everyday life. In this study, a duration summation paradigm was used to examine whether summation of temporal durations introduces an underestimation or overestimation bias, and whether this bias is common to visual and auditory modalities. Two within- or across-modality stimuli were presented sequentially for variable durations. Participants were asked to reproduce the sum of the two durations (0.6–1.1 s). We found that the sum of two durations was overestimated regardless of stimulus modalities. A subsequent control experiment indicated that the overestimation bias arose from the summation process, not perceptual or memory processes. Furthermore, we observed strong positive correlations between the overestimation bias for different sensory modalities within participants. These results suggest that the sum of two durations is overestimated, and that supra-modal processes may be responsible for this overestimation bias.     

Cortical signatures of auditory object binding in children with autism spectrum disorder are anomalous in concordance with behavior and diagnosis

Organizing sensory information into coherent perceptual objects is fundamental to everyday perception and communication. In the visual domain, indirect evidence from cortical responses suggests that children with autism spectrum disorder (ASD) have anomalous figure–ground segregation. While auditory processing abnormalities are common in ASD, especially in environments with multiple sound sources, to date, the question of scene segregation in ASD has not been directly investigated in audition. Using magnetoencephalography, we measured cortical responses to unattended (passively experienced) auditory stimuli while parametrically manipulating the degree of temporal coherence that facilitates auditory figure–ground segregation. Results from 21 children with ASD (aged 7–17 years) and 26 age- and IQ-matched typically developing children provide evidence that children with ASD show anomalous growth of cortical neural responses with increasing temporal coherence of the auditory figure. The documented neurophysiological abnormalities did not depend on age, and were reflected both in the response evoked by changes in temporal coherence of the auditory scene and in the associated induced gamma rhythms. Furthermore, the individual neural measures were predictive of diagnosis (83% accuracy) and also correlated with behavioral measures of ASD severity and auditory processing abnormalities. These findings offer new insight into the neural mechanisms underlying auditory perceptual deficits and sensory overload in ASD, and suggest that temporal-coherence-based auditory scene analysis and suprathreshold processing of coherent auditory objects may be atypical in ASD.

To navigate everyday environments, the auditory system must analyze the temporal coherence of sound elements scattered across different frequencies to organize them into discrete perceptual objects. This neuroimaging study reveals that this process is altered in autism, potentially explaining the often-experienced sensory overload.