A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem—action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.     

Pre- and postsynaptically expressed spike-timing-dependent plasticity contribute differentially to neuronal learning

A plethora of experimental studies have shown that long-term synaptic plasticity can be expressed pre- or postsynaptically depending on a range of factors such as developmental stage, synapse type, and activity patterns. The functional consequences of this diversity are not clear, although it is understood that whereas postsynaptic expression of plasticity predominantly affects synaptic response amplitude, presynaptic expression alters both synaptic response amplitude and short-term dynamics. In most models of neuronal learning, long-term synaptic plasticity is implemented as changes in connective weights. The consideration of long-term plasticity as a fixed change in amplitude corresponds more closely to post- than to presynaptic expression, which means theoretical outcomes based on this choice of implementation may have a postsynaptic bias. To explore the functional implications of the diversity of expression of long-term synaptic plasticity, we adapted a model of long-term plasticity, more specifically spike-timing-dependent plasticity (STDP), such that it was expressed either independently pre- or postsynaptically, or in a mixture of both ways. We compared pair-based standard STDP models and a biologically tuned triplet STDP model, and investigated the outcomes in a minimal setting, using two different learning schemes: in the first, inputs were triggered at different latencies, and in the second a subset of inputs were temporally correlated. We found that presynaptic changes adjusted the speed of learning, while postsynaptic expression was more efficient at regulating spike timing and frequency. When combining both expression loci, postsynaptic changes amplified the response range, while presynaptic plasticity allowed control over postsynaptic firing rates, potentially providing a form of activity homeostasis. Our findings highlight how the seemingly innocuous choice of implementing synaptic plasticity by single weight modification may unwittingly introduce a postsynaptic bias in modelling outcomes. We conclude that pre- and postsynaptically expressed plasticity are not interchangeable, but enable complimentary functions.     

Hebbian learning from higher-order correlations requires crosstalk minimization

Activity-dependent synaptic plasticity should be extremely connection specific, though experiments have shown it is not, and biophysics suggests it cannot be. Extreme specificity (near-zero “crosstalk”) might be essential for unsupervised learning from higher-order correlations, especially when a neuron has many inputs. It is well known that a normalized nonlinear Hebbian rule can learn “unmixing” weights from inputs generated by linearly combining independently fluctuating nonGaussian sources using an orthogonal mixing matrix. We previously reported that even if the matrix is only approximately orthogonal, a nonlinear-specific Hebbian rule can usually learn almost correct unmixing weights (Cox and Adams in Front Comput Neurosci 3: doi:10.3389/neuro.10.011.2009 2009). We also reported simulations that showed that as crosstalk increases from zero, the learned weight vector first moves slightly away from the crosstalk-free direction and then, at a sharp threshold level of inspecificity, jumps to a completely incorrect direction. Here, we report further numerical experiments that show that above this threshold, residual learning is driven instead almost entirely by second-order input correlations, as occurs using purely Gaussian sources or a linear rule, and any amount of crosstalk. Thus, in this “ICA” model learning from higher-order correlations, required for unmixing, requires high specificity. We compare our results with a recent mathematical analysis of the effect of crosstalk for exactly orthogonal mixing, which revealed that a second, even lower, threshold, exists below which successful learning is impossible unless weights happen to start close to the correct direction. Our simulations show that this also holds when the mixing is not exactly orthogonal. These results suggest that if the brain uses simple Hebbian learning, it must operate with extraordinarily accurate synaptic plasticity to ensure powerful high-dimensional learning. Synaptic crowding would preclude this when inputs are numerous, and we propose that the neocortex might be distinguished by special circuitry that promotes extreme specificity for high-dimensional nonlinear learning.     

Modeling neural mechanisms of vertebrate habituation: Locus specificity and pattern discrimination

A critical problem in neurobiology is to explain how the central nervous system coordinates pattern discrimination and locus specificity in learning. This problem is investigated in anuran amphibians who demonstrate both locus specificity and pattern discrimination in visual habituation. A neural mechanism is proposed whereby neural circuitry for pattern discrimination is shared by a spatial memory system. Such learning processes are argued to occur in the medial pallium (MP), the anuran's homolog of mammalian hippocampus. Necessary mapping from the shared network to spatial memory is set up by a mechanism that forms topographical connections, with desired orientation determined by activity gradient in presynaptic and postsynaptic layers. The model of MP is tested on both locus and stimulus specific habituation, which involve short-term as well as long-term synaptic plasticity. Successful modeling yields a set of predictions concerning MP organization and learning properties.     

Impairment of hippocampal long-term depression and defective spatial learning and memory in p35 mice

Cdk5 (cyclin-dependent kinase 5) activity is dependent upon association with one of two neuron-specific activators, p35 or p39. Genetic deletion of Cdk5 causes perinatal lethality with severe defects in corticogenesis and neuronal positioning. p35(-/-) mice are viable with milder histological abnormalities. Although substantial evidence implicates Cdk5 in synaptic plasticity, its role in learning and memory has not been evaluated using mutant mouse models. We report here that p35(-/-) mice have deficiencies in spatial learning and memory. Close examination of hippocampal circuitry revealed subtle histological defects in CA1 pyramidal cells. Furthermore, p35(-/-) mice exhibit impaired long-term depression and depotentiation of long-term potentiation in the Schaeffer collateral CA1 pathway. Moreover, the Cdk5-dependent phosphorylation state of protein phosphatase inhibitor-1 was increased in 4-week-old mice due to increased levels of p39, which co-localized with inhibitor-1 and Cdk5 in the cytoplasm. These results demonstrate that p35-dependent Cdk5 activity is important to learning and synaptic plasticity. Deletion of p35 may shift the substrate specificity of Cdk5 due to compensatory expression of p39.     

Learning of Precise Spike Times with Homeostatic Membrane Potential Dependent Synaptic Plasticity

Precise spatio-temporal patterns of neuronal action potentials underly e.g. sensory representations and control of muscle activities. However, it is not known how the synaptic efficacies in the neuronal networks of the brain adapt such that they can reliably generate spikes at specific points in time. Existing activity-dependent plasticity rules like Spike-Timing-Dependent Plasticity are agnostic to the goal of learning spike times. On the other hand, the existing formal and supervised learning algorithms perform a temporally precise comparison of projected activity with the target, but there is no known biologically plausible implementation of this comparison. Here, we propose a simple and local unsupervised synaptic plasticity mechanism that is derived from the requirement of a balanced membrane potential. Since the relevant signal for synaptic change is the postsynaptic voltage rather than spike times, we call the plasticity rule Membrane Potential Dependent Plasticity (MPDP). Combining our plasticity mechanism with spike after-hyperpolarization causes a sensitivity of synaptic change to pre- and postsynaptic spike times which can reproduce Hebbian spike timing dependent plasticity for inhibitory synapses as was found in experiments. In addition, the sensitivity of MPDP to the time course of the voltage when generating a spike allows MPDP to distinguish between weak (spurious) and strong (teacher) spikes, which therefore provides a neuronal basis for the comparison of actual and target activity. For spatio-temporal input spike patterns our conceptually simple plasticity rule achieves a surprisingly high storage capacity for spike associations. The sensitivity of the MPDP to the subthreshold membrane potential during training allows robust memory retrieval after learning even in the presence of activity corrupted by noise. We propose that MPDP represents a biophysically plausible mechanism to learn temporal target activity patterns.     

Interference in ballistic motor learning: specificity and role of sensory error signals

Humans are capable of learning numerous motor skills, but newly acquired skills may be abolished by subsequent learning. Here we ask what factors determine whether interference occurs in motor learning. We speculated that interference requires competing processes of synaptic plasticity in overlapping circuits and predicted specificity. To test this, subjects learned a ballistic motor task. Interference was observed following subsequent learning of an accuracy-tracking task, but only if the competing task involved the same muscles and movement direction. Interference was not observed from a non-learning task suggesting that interference requires competing learning. Subsequent learning of the competing task 4 h after initial learning did not cause interference suggesting disruption of early motor memory consolidation as one possible mechanism underlying interference. Repeated transcranial magnetic stimulation (rTMS) of corticospinal motor output at intensities below movement threshold did not cause interference, whereas suprathreshold rTMS evoking motor responses and (re)afferent activation did. Finally, the experiments revealed that suprathreshold repetitive electrical stimulation of the agonist (but not antagonist) peripheral nerve caused interference. The present study is, to our knowledge, the first to demonstrate that peripheral nerve stimulation may cause interference. The finding underscores the importance of sensory feedback as error signals in motor learning. We conclude that interference requires competing plasticity in overlapping circuits. Interference is remarkably specific for circuits involved in a specific movement and it may relate to sensory error signals.     

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.     

A model for complex sequence learning and reproduction in neural populations

Temporal patterns of activity which repeat above chance level in the brains of vertebrates and in the mammalian neocortex have been reported experimentally. This temporal structure is thought to subserve functions such as movement, speech, and generation of rhythms. Several studies aim to explain how particular sequences of activity are learned, stored, and reproduced. The learning of sequences is usually conceived as the creation of an excitation pathway within a homogeneous neuronal population, but models embodying the autonomous function of such a learning mechanism are fraught with concerns about stability, robustness, and biological plausibility. We present two related computational models capable of learning and reproducing sequences which come from external stimuli. Both models assume that there exist populations of densely interconnected excitatory neurons, and that plasticity can occur at the population level. The first model uses temporally asymmetric Hebbian plasticity to create excitation pathways between populations in response to activation from an external source. The transition of the activity from one population to the next is permitted by the interplay of excitatory and inhibitory populations, which results in oscillatory behavior that seems to agree with experimental findings in the mammalian neocortex. The second model contains two layers, each one like the network used in the first model, with unidirectional excitatory connections from the first to the second layer experiencing Hebbian plasticity. Input sequences presented in the second layer become associated with the ongoing first layer activity, so that this activity can later elicit the the presented sequence in the absence of input. We explore the dynamics of these models, and discuss their potential implications, particularly to working memory, oscillations, and rhythm generation.     

HDAC2:阿尔茨海默病治疗靶点

表观调节机制在阿尔茨海默病的发生、发展过程中起着重要作用。乙酰化组蛋白和乙酰化非组蛋白在基因表达与信号转导过程中具有重要的调控作用。组蛋白去乙酰化酶抑制剂可以改善AD患者突触可塑性与学习记忆能力。HDAC2在控制神经元形成中起关键作用。HDAC2参与海马区域记忆形成相关蛋白表达,对学习和记忆的形成具有负调节作用,影响神经突触可塑性和数量。目前应用的HDAC抑制剂为广谱药物缺乏特异性,分析HDAC2作用机制有利于研究出针对疾病的靶点药物。     

Learning flexible sensori-motor mappings in a complex network

Given the complex structure of the brain, how can synaptic plasticity explain the learning and forgetting of associations when these are continuously changing? We address this question by studying different reinforcement learning rules in a multilayer network in order to reproduce monkey behavior in a visuomotor association task. Our model can only reproduce the learning performance of the monkey if the synaptic modifications depend on the pre- and postsynaptic activity, and if the intrinsic level of stochasticity is low. This favored learning rule is based on reward modulated Hebbian synaptic plasticity and shows the interesting feature that the learning performance does not substantially degrade when adding layers to the network, even for a complex problem.     

Soft-bound Synaptic Plasticity Increases Storage Capacity

Accurate models of synaptic plasticity are essential to understand the adaptive properties of the nervous system and for realistic models of learning and memory. Experiments have shown that synaptic plasticity depends not only on pre- and post-synaptic activity patterns, but also on the strength of the connection itself. Namely, weaker synapses are more easily strengthened than already strong ones. This so called soft-bound plasticity automatically constrains the synaptic strengths. It is known that this has important consequences for the dynamics of plasticity and the synaptic weight distribution, but its impact on information storage is unknown. In this modeling study we introduce an information theoretic framework to analyse memory storage in an online learning setting. We show that soft-bound plasticity increases a variety of performance criteria by about 18% over hard-bound plasticity, and likely maximizes the storage capacity of synapses.     

Representational drift as a window into neural and behavioural plasticity

Large-scale recordings of neural activity over days and weeks have revealed that neural representations of familiar tasks, precepts and actions continually evolve without obvious changes in behaviour. We hypothesise that this steady drift in neural activity and accompanying physiological changes is due in part to the continuous application of a learning rule at the cellular and population level. Explicit predictions of this drift can be found in neural network models that use iterative learning to optimise weights. Drift therefore provides a measurable signal that can reveal systems–level properties of biological plasticity mechanisms, such as their precision and effective learning rates.     

Presynaptic plasticity and associative learning are impaired in a Drosophila presenilin null mutant

Alzheimer's disease is a neurodegenerative disorder characterized by progressive memory and cognitive decline that is associated with changes in synaptic plasticity and neuronal cell loss. Recent evidence suggests that some of these defects may be due to a loss of normal presenilin activity. Here, we have examined the effect of loss of Drosophila presenilin (psn) function on synaptic plasticity and learning. Basal transmitter release was elevated in psn mutants while both paired pulse synaptic plasticity and post-tetanic potentiation were impaired. These defects in synaptic strength and plasticity were not due to developmental defects in NMJ morphology. We also found that psn null terminals take up significantly less FM 4-64 than control terminals when loaded with high frequency stimulation, suggesting a defect in synaptic vesicle availability or mobilization. To determine whether these reductions in synaptic plasticity had any impact on learning, we tested the larvae for defects in associative learning. Using both olfactory and visual learning assays, we found that associative learning is impaired in psn mutants compared with controls. Both the learning and synaptic defects could be rescued by expression of a full length psn transgene suggesting the defects are specifically due to a loss of psn function. Taken together, these results provide the first evidence of learning and synaptic defects in a Drosophila psn mutant and strongly suggest a presynaptic role for presenilin in normal neuronal function.     

A Mathematical Model of the Cerebellar-Olivary System II: Motor Adaptation Through Systematic Disruption of Climbing Fiber Equilibrium

The implications for motor learning of the model developed in the previous article are analyzed using idealized Pavlovian eyelid conditioning trials, a simple example of cerebellar motor learning. Results suggest that changes in grPkj synapses produced by a training trial disrupt equilibrium and lead to subsequent changes in the opposite direction that restore equilibrium. We show that these opposing phases would make the net plasticity at each grPkj synapse proportional to the change in its activity during the training trial, as influenced by a factor that precludes plasticity when changes in activity are inconsistent. This yields an expression for the component of granule cell activity that supports learning, the across-trials consistency vector, the square of which determines the expected rate of learning. These results suggest that the equilibrium maintained by the cerebellar-olivary system must be disrupted in a specific and systematic manner to promote cerebellar-mediated motor learning.     

Protein kinase D promotes plasticity-induced F-actin stabilization in dendritic spines and regulates memory formation

Actin turnover in dendritic spines influences spine development, morphology, and plasticity, with functional consequences on learning and memory formation. In nonneuronal cells, protein kinase D (PKD) has an important role in stabilizing F-actin via multiple molecular pathways. Using in vitro models of neuronal plasticity, such as glycine-induced chemical long-term potentiation (LTP), known to evoke synaptic plasticity, or long-term depolarization block by KCl, leading to homeostatic morphological changes, we show that actin stabilization needed for the enlargement of dendritic spines is dependent on PKD activity. Consequently, impaired PKD functions attenuate activity-dependent changes in hippocampal dendritic spines, including LTP formation, cause morphological alterations in vivo, and have deleterious consequences on spatial memory formation. We thus provide compelling evidence that PKD controls synaptic plasticity and learning by regulating actin stability in dendritic spines.     

Learning-Induced Synchronization and Plasticity of a Developing Neural Network

Learning-induced synchronization of a neural network at various developing stages is studied by computer simulations using a pulse-coupled neural network model in which the neuronal activity is simulated by a one-dimensional map. Two types of Hebbian plasticity rules are investigated and their differences are compared. For both models, our simulations show a logarithmic increase in the synchronous firing frequency of the network with the culturing time of the neural network. This result is consistent with recent experimental observations. To investigate how to control the synchronization behavior of a neural network after learning, we compare the occurrence of synchronization for four networks with different designed patterns under the influence of an external signal. The effect of such a signal on the network activity highly depends on the number of connections between neurons. We discuss the synaptic plasticity and enhancement effects for a random network after learning at various developing stages.     

Robustness of learning that is based on covariance-driven synaptic plasticity

It is widely believed that learning is due, at least in part, to long-lasting modifications of the strengths of synapses in the brain. Theoretical studies have shown that a family of synaptic plasticity rules, in which synaptic changes are driven by covariance, is particularly useful for many forms of learning, including associative memory, gradient estimation, and operant conditioning. Covariance-based plasticity is inherently sensitive. Even a slight mistuning of the parameters of a covariance-based plasticity rule is likely to result in substantial changes in synaptic efficacies. Therefore, the biological relevance of covariance-based plasticity models is questionable. Here, we study the effects of mistuning parameters of the plasticity rule in a decision making model in which synaptic plasticity is driven by the covariance of reward and neural activity. An exact covariance plasticity rule yields Herrnstein's matching law. We show that although the effect of slight mistuning of the plasticity rule on the synaptic efficacies is large, the behavioral effect is small. Thus, matching behavior is robust to mistuning of the parameters of the covariance-based plasticity rule. Furthermore, the mistuned covariance rule results in undermatching, which is consistent with experimentally observed behavior. These results substantiate the hypothesis that approximate covariance-based synaptic plasticity underlies operant conditioning. However, we show that the mistuning of the mean subtraction makes behavior sensitive to the mistuning of the properties of the decision making network. Thus, there is a tradeoff between the robustness of matching behavior to changes in the plasticity rule and its robustness to changes in the properties of the decision making network.