Representational drift: Emerging theories for continual learning and experimental future directions

Recent work has revealed that the neural activity patterns correlated with sensation, cognition, and action often are not stable and instead undergo large scale changes over days and weeks—a phenomenon called representational drift. Here, we highlight recent observations of drift, how drift is unlikely to be explained by experimental confounds, and how the brain can likely compensate for drift to allow stable computation. We propose that drift might have important roles in neural computation to allow continual learning, both for separating and relating memories that occur at distinct times. Finally, we present an outlook on future experimental directions that are needed to further characterize drift and to test emerging theories for drift's role in computation.     

Motor learning with unstable neural representations

It is often assumed that learning takes place by changing an otherwise stable neural representation. To test this assumption, we studied changes in the directional tuning of primate motor cortical neurons during reaching movements performed in familiar and novel environments. During the familiar task, tuning curves exhibited slow random drift. During learning of the novel task, random drift was accompanied by systematic shifts of tuning curves. Our analysis suggests that motor learning is based on a surprisingly unstable neural representation. To explain these results, we propose that motor cortex is a redundant neural network, i.e., any single behavior can be realized by multiple configurations of synaptic strengths. We further hypothesize that synaptic modifications underlying learning contain a random component, which causes wandering among synaptic configurations with equivalent behaviors but different neural representations. We use a simple model to explore the implications of these assumptions.     

Learning shapes spontaneous activity itinerating over memorized states

Learning is a process that helps create neural dynamical systems so that an appropriate output pattern is generated for a given input. Often, such a memory is considered to be included in one of the attractors in neural dynamical systems, depending on the initial neural state specified by an input. Neither neural activities observed in the absence of inputs nor changes caused in the neural activity when an input is provided were studied extensively in the past. However, recent experimental studies have reported existence of structured spontaneous neural activity and its changes when an input is provided. With this background, we propose that memory recall occurs when the spontaneous neural activity changes to an appropriate output activity upon the application of an input, and this phenomenon is known as bifurcation in the dynamical systems theory. We introduce a reinforcement-learning-based layered neural network model with two synaptic time scales; in this network, I/O relations are successively memorized when the difference between the time scales is appropriate. After the learning process is complete, the neural dynamics are shaped so that it changes appropriately with each input. As the number of memorized patterns is increased, the generated spontaneous neural activity after learning shows itineration over the previously learned output patterns. This theoretical finding also shows remarkable agreement with recent experimental reports, where spontaneous neural activity in the visual cortex without stimuli itinerate over evoked patterns by previously applied signals. Our results suggest that itinerant spontaneous activity can be a natural outcome of successive learning of several patterns, and it facilitates bifurcation of the network when an input is provided.     

Complex Population Response of Dorsal Putamen Neurons Predicts the Ability to Learn

Day-to-day variability in performance is a common experience. We investigated its neural correlate by studying learning behavior of monkeys in a two-alternative forced choice task, the two-armed bandit task. We found substantial session-to-session variability in the monkeys’ learning behavior. Recording the activity of single dorsal putamen neurons we uncovered a dual function of this structure. It has been previously shown that a population of neurons in the DLP exhibits firing activity sensitive to the reward value of chosen actions. Here, we identify putative medium spiny neurons in the dorsal putamen that are cue-selective and whose activity builds up with learning. Remarkably we show that session-to-session changes in the size of this population and in the intensity with which this population encodes cue-selectivity is correlated with session-to-session changes in the ability to learn the task. Moreover, at the population level, dorsal putamen activity in the very beginning of the session is correlated with the performance at the end of the session, thus predicting whether the monkey will have a "good" or "bad" learning day. These results provide important insights on the neural basis of inter-temporal performance variability.     

Estimating the hidden learning representations

Successful adaptation relies on the ability to learn the consequence of our actions in different environments. However, understanding the neural bases of this ability still represents one of the great challenges of system neuroscience. In fact, the neuronal plasticity changes occurring during learning cannot be fully controlled experimentally and their evolution is hidden. Our approach is to provide hypotheses about the structure and dynamics of the hidden plasticity changes using behavioral learning theory. In fact, behavioral models of animal learning provide testable predictions about the hidden learning representations by formalizing their relation with the observables of the experiment (stimuli, actions and outcomes). Thus, we can understand whether and how the predicted learning processes are represented at the neural level by estimating their evolution and correlating them with neural data. Here, we present a bayesian model approach to estimate the evolution of the internal learning representations from the observations of the experiment (state estimation), and to identify the set of models' parameters (parameter estimation) and the class of behavioral model (model selection) that are most likely to have generated a given sequence of actions and outcomes. More precisely, we use Sequential Monte Carlo methods for state estimation and the maximum likelihood principle (MLP) for model selection and parameter estimation. We show that the method recovers simulated trajectories of learning sessions on a single-trial basis and provides predictions about the activity of different categories of neurons that should participate in the learning process. By correlating the estimated evolutions of the learning variables, we will be able to test the validity of different models of instrumental learning and possibly identify the neural bases of learning.     

Neural Representation of Ambiguous Visual Objects in the Inferior Temporal Cortex

Inferior temporal (IT) cortex as the final stage of the ventral visual pathway is involved in visual object recognition. In our everyday life we need to recognize visual objects that are degraded by noise. Psychophysical studies have shown that the accuracy and speed of the object recognition decreases as the amount of visual noise increases. However, the neural representation of ambiguous visual objects and the underlying neural mechanisms of such changes in the behavior are not known. Here, by recording the neuronal spiking activity of macaque monkeys’ IT we explored the relationship between stimulus ambiguity and the IT neural activity. We found smaller amplitude, later onset, earlier offset and shorter duration of the response as visual ambiguity increased. All of these modulations were gradual and correlated with the level of stimulus ambiguity. We found that while category selectivity of IT neurons decreased with noise, it was preserved for a large extent of visual ambiguity. This noise tolerance for category selectivity in IT was lost at 60% noise level. Interestingly, while the response of the IT neurons to visual stimuli at 60% noise level was significantly larger than their baseline activity and full (100%) noise, it was not category selective anymore. The latter finding shows a neural representation that signals the presence of visual stimulus without signaling what it is. In general these findings, in the context of a drift diffusion model, explain the neural mechanisms of perceptual accuracy and speed changes in the process of recognizing ambiguous objects.     

Sequence learning recodes cortical representations instead of strengthening initial ones

We contrast two computational models of sequence learning. The associative learner posits that learning proceeds by strengthening existing association weights. Alternatively, recoding posits that learning creates new and more efficient representations of the learned sequences. Importantly, both models propose that humans act as optimal learners but capture different statistics of the stimuli in their internal model. Furthermore, these models make dissociable predictions as to how learning changes the neural representation of sequences. We tested these predictions by using fMRI to extract neural activity patterns from the dorsal visual processing stream during a sequence recall task. We observed that only the recoding account can explain the similarity of neural activity patterns, suggesting that participants recode the learned sequences using chunks. We show that associative learning can theoretically store only very limited number of overlapping sequences, such as common in ecological working memory tasks, and hence an efficient learner should recode initial sequence representations.     

Neural network features distinguish chemosensory stimuli in Caenorhabditis elegans

Nervous systems extract and process information from the environment to alter animal behavior and physiology. Despite progress in understanding how different stimuli are represented by changes in neuronal activity, less is known about how they affect broader neural network properties. We developed a framework for using graph-theoretic features of neural network activity to predict ecologically relevant stimulus properties, in particular stimulus identity. We used the transparent nematode, Caenorhabditis elegans, with its small nervous system to define neural network features associated with various chemosensory stimuli. We first immobilized animals using a microfluidic device and exposed their noses to chemical stimuli while monitoring changes in neural activity of more than 50 neurons in the head region. We found that graph-theoretic features, which capture patterns of interactions between neurons, are modulated by stimulus identity. Further, we show that a simple machine learning classifier trained using graph-theoretic features alone, or in combination with neural activity features, can accurately predict salt stimulus. Moreover, by focusing on putative causal interactions between neurons, the graph-theoretic features were almost twice as predictive as the neural activity features. These results reveal that stimulus identity modulates the broad, network-level organization of the nervous system, and that graph theory can be used to characterize these changes.     

A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem—action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.     

Dynamic Changes in Single Unit Activity and Gamma Oscillations in a Thalamocortical Circuit during Rapid Instrumental Learning

The medial prefrontal cortex (mPFC) and mediodorsal thalamus (MD) together form a thalamocortical circuit that has been implicated in the learning and production of goal-directed actions. In this study we measured neural activity in both regions simultaneously, as rats learned to press a lever to earn food rewards. In both MD and mPFC, instrumental learning was accompanied by dramatic changes in the firing patterns of the neurons, in particular the rapid emergence of single-unit neural activity reflecting the completion of the action and reward delivery. In addition, we observed distinct patterns of changes in the oscillatory LFP response in MD and mPFC. With learning, there was a significant increase in theta band oscillations (6–10 Hz) in the MD, but not in the mPFC. By contrast, gamma band oscillations (40–55 Hz) increased in the mPFC, but not in the MD. Coherence between these two regions also changed with learning: gamma coherence in relation to reward delivery increased, whereas theta coherence did not. Together these results suggest that, as rats learned the instrumental contingency between action and outcome, the emergence of task related neural activity is accompanied by enhanced functional interaction between MD and mPFC in response to the reward feedback.     

Network, cellular and molecular mechanisms of plasticity in simple nervous systems

In the present study we will try to single out several principles of the nervous system functioning essential for describing mechanisms of learning and memory basing on our own experimental investigation of cellular mechanisms of memory in the nervous system of gastropod molluscs and literature data: main changes in functioning due to learning occur in effectivity of synaptic inputs and in the intrinsic properties of postsynaptic neurons; due to learning some synaptic inputs of neurons selectively change its effectivity due to pre- and postsynaptic changes, but the induction of plasticity always starts in postsynapse, maintaining of long-term memory in postsynapse is also shown; reinforcement is not related to activity of the neural chain receptor-sensory neuron-interneuron-motoneuron-effector; reinforcement is mediated via activity of modulatory neurons, and in some cases can be exerted by a single neuron; activity of modulatory neurons is necessary for development of plastic modifications of behavior (including associative), but is not needed for recall of conditioned responses. At the same time, the modulatory neurons (in fact they constitute a neural reinforcement system) are necessary for recall of context associative memory; changes due to learning occur at least in two independent loci in the nervous system. A possibility for erasure of memory with participation of nitroxide is experimentally and theoretically based.     

The search for a hippocampal engram

Understanding the molecular and cellular changes that underlie memory, the engram, requires the identification, isolation and manipulation of the neurons involved. This presents a major difficulty for complex forms of memory, for example hippocampus-dependent declarative memory, where the participating neurons are likely to be sparse, anatomically distributed and unique to each individual brain and learning event. In this paper, I discuss several new approaches to this problem. In vivo calcium imaging techniques provide a means of assessing the activity patterns of large numbers of neurons over long periods of time with precise anatomical identification. This provides important insight into how the brain represents complex information and how this is altered with learning. The development of techniques for the genetic modification of neural ensembles based on their natural, sensory-evoked, activity along with optogenetics allows direct tests of the coding function of these ensembles. These approaches provide a new methodological framework in which to examine the mechanisms of complex forms of learning at the level of the neurons involved in a specific memory.     

Exploring Memory Representations with Activity-Based Genetics

The brain is thought to represent specific memories through the activity of sparse and distributed neural ensembles. In this review, we examine the use of immediate early genes (IEGs), genes that are induced by neural activity, to specifically identify and genetically modify neurons activated naturally by environmental experience. Recent studies using this approach have identified cellular and molecular changes specific to neurons activated during learning relative to their inactive neighbors. By using opto- and chemogenetic regulators of neural activity, the neurons naturally recruited during learning can be artificially reactivated to directly test their role in coding external information. In contextual fear conditioning, artificial reactivation of learning-induced neural ensembles in the hippocampus or neocortex can substitute for the context itself. That is, artificial stimulation of these neurons can apparently cause the animals to “think” they are in the context. This represents a powerful approach to testing the principles by which the brain codes for the external world and how these circuits are modified with learning.A central feature of nervous systems is that, to function properly, specific neurons must become active in response to specific stimuli. The nature of this selective activation and its modification with experience is the focus of much neuroscience research, ranging from studies of sensory processing in experimental animals to disorders of thought such as schizophrenia in humans. The central dogma of neuroscience is that perceptions, memories, thoughts, and higher mental functions arise from the pattern and timing of the activity in neural ensembles in specific parts of the brain at specific points in time. Until quite recently, the investigation of these “circuit”-based questions has primarily been limited to observational techniques, such as single unit recording, functional magnetic resonance imagery (fMRI), and calcium imaging, to document the patterns of neural activity evoked by sensory experience or even complex psychological contingencies in human fMRI studies. These techniques have been enormously successful and created a framework for understanding information processing in the brain. For example, recordings in the visual system have indicated that, in the primary visual cortex, neurons are tuned to the orientation of linear stimuli (Hubel and Wiesel 1962). In contrast, neurons in higher brain areas can respond to discrete items. The most striking example of this specificity comes from in vivo recording in the human medial temporal lobe in which single units have been identified that respond to photos of the actress Halle Berry as well as her written name (Quiroga et al. 2005). This highly selective tuning of neural activity is suggestive of function, but how can this be directly tested? What would be the effect of stimulating just this rare population of neurons, a memory of the actress, a sensory illusion of her image? How does this type of specific firing arise? Do these neurons differ from their nonresponsive neighbors in terms of biochemistry, cell biology, or connectivity? Do they undergo molecular alterations when new information is learned about this individual and are these changes required for the learning? These types of questions have recently become accessible to study in mice through the use of activity-based genetic manipulation, in which neurons that are activated by a specific sensory stimulus can be altered to express any gene of experimental interest. These studies and approaches will be the focus of this work.     

Tracking Plasticity: Effects of Long-Term Rehearsal in Expert Dancers Encoding Music to Movement

Our knowledge of neural plasticity suggests that neural networks show adaptation to environmental and intrinsic change. In particular, studies investigating the neuroplastic changes associated with learning and practicing motor tasks have shown that practicing such tasks results in an increase in neural activation in several specific brain regions. However, studies comparing experts and non-experts suggest that experts employ less neuronal activation than non-experts when performing a familiar motor task. Here, we aimed to determine the long-term changes in neural networks associated with learning a new dance in professional ballet dancers over 34 weeks. Subjects visualized dance movements to music while undergoing fMRI scanning at four time points over 34-weeks. Results demonstrated that initial learning and performance at seven weeks led to increases in activation in cortical regions during visualization compared to the first week. However, at 34 weeks, the cortical networks showed reduced activation compared to week seven. Specifically, motor learning and performance over the 34 weeks showed the typical inverted-U-shaped function of learning. Further, our result demonstrate that learning of a motor sequence of dance movements to music in the real world can be visualized by expert dancers using fMRI and capture highly significant modeled fits of the brain network variance of BOLD signals from early learning to expert level performance.     

The role of synaptic ion channels in synaptic plasticity

The nervous system receives a large amount of information about the environment through elaborate sensory routes. Processing and integration of these wide-ranging inputs often results in long-term behavioural alterations as a result of past experiences. These relatively permanent changes in behaviour are manifestations of the capacity of the nervous system for learning and memory. At the cellular level, synaptic plasticity is one of the mechanisms underlying this process. Repeated neural activity generates physiological changes in the nervous system that ultimately modulate neuronal communication through synaptic transmission. Recent studies implicate both presynaptic and postsynaptic ion channels in the process of synapse strength modulation. Here, we review the role of synaptic ion channels in learning and memory, and discuss the implications and significance of these findings towards deciphering the molecular biology of learning and memory.     

Training feedforward neural networks with dynamic particle swarm optimisation

Particle swarm optimisation has been successfully applied to train feedforward neural networks in static environments. Many real-world problems to which neural networks are applied are dynamic in the sense that the underlying data distribution changes over time. In the context of classification problems, this leads to concept drift where decision boundaries may change over time. This article investigates the applicability of dynamic particle swarm optimisation algorithms as neural network training algorithms under the presence of concept drift.     

Making evolution stick: using sticky notes to teach the mechanisms of evolutionary change

Evolution and its mechanisms of action are concepts that unite all aspects of biology, but remain some of the most difficult for students to understand. To address this challenge, we designed a hands-on activity that introduces fundamental mechanisms of evolutionary change: natural selection, genetic drift, and gene flow. In small groups, students use a population of sticky notes to reveal the consequences of each mechanism on phenotype frequency. In a follow-up homework assignment, students then explore how changes in phenotype frequency reflect changes in allele frequency in the population. This activity is suitable for anyone learning the basics of evolution, from high-school through the undergraduate level. We have provided detailed instructions, in-class worksheets, follow-up homework, and extensions that allow the activity to be simplified or made more complex as needed. In our own classrooms, we have observed that the concrete and collaborative nature of this activity enables students to deepen their understanding of the mechanisms through which evolution occurs. We have designed this study such that, in completing this activity, we hope to offer students the opportunity to confront potential misconceptions about evolution and gain a solid foundation for future explorations in the discipline.     

Embedding Responses in Spontaneous Neural Activity Shaped through Sequential Learning

Recent experimental measurements have demonstrated that spontaneous neural activity in the absence of explicit external stimuli has remarkable spatiotemporal structure. This spontaneous activity has also been shown to play a key role in the response to external stimuli. To better understand this role, we proposed a viewpoint, “memories-as-bifurcations,” that differs from the traditional “memories-as-attractors” viewpoint. Memory recall from the memories-as-bifurcations viewpoint occurs when the spontaneous neural activity is changed to an appropriate output activity upon application of an input, known as a bifurcation in dynamical systems theory, wherein the input modifies the flow structure of the neural dynamics. Learning, then, is a process that helps create neural dynamical systems such that a target output pattern is generated as an attractor upon a given input. Based on this novel viewpoint, we introduce in this paper an associative memory model with a sequential learning process. Using a simple Hebbian-type learning, the model is able to memorize a large number of input/output mappings. The neural dynamics shaped through the learning exhibit different bifurcations to make the requested targets stable upon an increase in the input, and the neural activity in the absence of input shows chaotic dynamics with occasional approaches to the memorized target patterns. These results suggest that these dynamics facilitate the bifurcations to each target attractor upon application of the corresponding input, which thus increases the capacity for learning. This theoretical finding about the behavior of the spontaneous neural activity is consistent with recent experimental observations in which the neural activity without stimuli wanders among patterns evoked by previously applied signals. In addition, the neural networks shaped by learning properly reflect the correlations of input and target-output patterns in a similar manner to those designed in our previous study.     

Specificity of sensorimotor learning and the neural code: Neuronal representations in the primary motor cortex

Human studies show that the learning of a new sensorimotor mapping that requires adaptation to directional errors is local and generalizes poorly to untrained directions. We trained monkeys to learn new visuomotor rotations for only one target in space and recorded neuronal activity in the primary motor cortex before, during and after learning. Similar to humans, the monkeys showed poor transfer of learning to other directions, as observed by behavioral aftereffects for untrained directions. To test for internal representations underlying these changes, we compared two features of neuronal activity before and after learning: changes in firing rates and changes in information content. Specific elevations of firing rate were only observed in a subpopulation of cells in the motor cortex with directional properties corresponding to the locally learned rotation; namely cells only showed plasticity if their preferred direction was near the training one. We applied measures from information theory to probe for learning-related changes in the neuronal code. Single cells conveyed more information about the direction of movement and this specific improvement in encoding was correlated with an increase in the slope of the neurons' tuning curve. Further, the improved information after learning enabled a more accurate reconstruction of movement direction from neuronal populations. Our findings suggest a neural mechanism for the confined generalization of a newly acquired internal model by showing a tight relationship between the locality of learning and the properties of neurons. They also provide direct evidence for improvement in the neural code as a result of learning.     

Falling towards forgetfulness: synaptic decay prevents spontaneous recovery of memory

Long after a new language has been learned and forgotten, relearning a few words seems to trigger the recall of other words. This "free-lunch learning" (FLL) effect has been demonstrated both in humans and in neural network models. Specifically, previous work proved that linear networks that learn a set of associations, then partially forget them all, and finally relearn some of the associations, show improved performance on the remaining (i.e., nonrelearned) associations. Here, we prove that relearning forgotten associations decreases performance on nonrelearned associations; an effect we call negative free-lunch learning. The difference between free-lunch learning and the negative free-lunch learning presented here is due to the particular method used to induce forgetting. Specifically, if forgetting is induced by isotropic drifting of weight vectors (i.e., by adding isotropic noise), then free-lunch learning is observed. However, as proved here, if forgetting is induced by weight values that simply decay or fall towards zero, then negative free-lunch learning is observed. From a biological perspective, and assuming that nervous systems are analogous to the networks used here, this suggests that evolution may have selected physiological mechanisms that involve forgetting using a form of synaptic drift rather than synaptic decay, because synaptic drift, but not synaptic decay, yields free-lunch learning.