Spatio-temporal credit assignment in neuronal population learning

In learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.     

Sequential Delivery of Synaptic GluA1- and GluA4-containing AMPA Receptors (AMPARs) by SAP97 Anchored Protein Complexes in Classical Conditioning

Multiple signaling pathways are involved in AMPAR trafficking to synapses during synaptic plasticity and learning. The mechanisms for how these pathways are coordinated in parallel but maintain their functional specificity involves subcellular compartmentalization of kinase function by scaffolding proteins, but how this is accomplished is not well understood. Here, we focused on characterizing the molecular machinery that functions in the sequential synaptic delivery of GluA1- and GluA4-containing AMPARs using an in vitro model of eyeblink classical conditioning. We show that conditioning induces the interaction of selective protein complexes with the key structural protein SAP97, which tightly regulates the synaptic delivery of GluA1 and GluA4 AMPAR subunits. The results demonstrate that in the early stages of conditioning the initial activation of PKA stimulates the formation of a SAP97-AKAP/PKA-GluA1 protein complex leading to synaptic delivery of GluA1-containing AMPARs through a SAP97-PSD95 interaction. This is followed shortly thereafter by generation of a SAP97-KSR1/PKC-GluA4 complex for GluA4 AMPAR subunit delivery again through a SAP97-PSD95 interaction. These data suggest that SAP97 forms the molecular backbone of a protein scaffold critical for delivery of AMPARs to the PSD during conditioning. Together, the findings reveal a cooperative interaction of multiple scaffolding proteins for appropriately timed delivery of subunit-specific AMPARs to synapses and support a sequential two-stage model of AMPAR synaptic delivery during classical conditioning.     

Computational Consequences of Temporally Asymmetric Learning Rules: I. Differential Hebbian Learning

Temporally asymetric learning rules governing plastic changes in synaptic efficacy have recently been identified in physiological studies. In these rules, the exact timing of pre- and postsynaptic spikes is critical to the induced change of synaptic efficacy. The temporal learning rules treated in this article are approximately antisymmetric; the synaptic efficacy is enhanced if the postsynaptic spike follows the presynaptic spike by a few milliseconds, but the efficacy is depressed if the postsynaptic spike precedes the presynaptic spike. The learning dynamics of this rule are studied using a stochastic model neuron receiving a set of serially delayed inputs. The average change of synaptic efficacy due to the temporally antisymmetric learning rule is shown to yield differential Hebbian learning. These results are demonstrated with both mathematical analyses and computer simulations, and connections with theories of classical conditioning are discussed.     

Molecular mechanisms underlying emotional learning and memory in the lateral amygdala

Fear conditioning is a valuable behavioral paradigm for studying the neural basis of emotional learning and memory. The lateral nucleus of the amygdala (LA) is a crucial site of neural changes that occur during fear conditioning. Pharmacological manipulations of the LA, strategically timed with respect to training and testing, have shed light on the molecular events that mediate the acquisition of fear associations and the formation and maintenance of long-term memories of those associations. Similar mechanisms have been found to underlie long-term potentiation (LTP) in LA, an artificial means of inducing synaptic plasticity and a physiological model of learning and memory. Thus, LTP-like changes in synaptic plasticity may underlie fear conditioning. Given that the neural circuit underlying fear conditioning has been implicated in emotional disorders in humans, the molecular mechanisms of fear conditioning are potential targets for psychotherapeutic drug development.     

Role of delayed nonsynaptic neuronal plasticity in long-term associative memory

BACKGROUND: It is now well established that persistent nonsynaptic neuronal plasticity occurs after learning and, like synaptic plasticity, it can be the substrate for long-term memory. What still remains unclear, though, is how nonsynaptic plasticity contributes to the altered neural network properties on which memory depends. Understanding how nonsynaptic plasticity is translated into modified network and behavioral output therefore represents an important objective of current learning and memory research. RESULTS: By using behavioral single-trial classical conditioning together with electrophysiological analysis and calcium imaging, we have explored the cellular mechanisms by which experience-induced nonsynaptic electrical changes in a neuronal soma remote from the synaptic region are translated into synaptic and circuit level effects. We show that after single-trial food-reward conditioning in the snail Lymnaea stagnalis, identified modulatory neurons that are extrinsic to the feeding network become persistently depolarized between 16 and 24 hr after training. This is delayed with respect to early memory formation but concomitant with the establishment and duration of long-term memory. The persistent nonsynaptic change is extrinsic to and maintained independently of synaptic effects occurring within the network directly responsible for the generation of feeding. Artificial membrane potential manipulation and calcium-imaging experiments suggest a novel mechanism whereby the somal depolarization of an extrinsic neuron recruits command-like intrinsic neurons of the circuit underlying the learned behavior. CONCLUSIONS: We show that nonsynaptic plasticity in an extrinsic modulatory neuron encodes information that enables the expression of long-term associative memory, and we describe how this information can be translated into modified network and behavioral output.     

Activity-dependent regulation of receptive field properties of cat area 17 by supervised Hebbian learning.

Most algorithms currently used to model synaptic plasticity in self-organizing cortical networks suppose that the change in synaptic efficacy is governed by the same structuring factor, i.e., the temporal correlation of activity between pre- and postsynaptic neurons. Functional predictions generated by such algorithms have been tested electrophysiologically in the visual cortex of anesthetized and paralyzed cats. Supervised learning procedures were applied at the cellular level to change receptive field (RF) properties during the time of recording of an individual functionally identified cell. The protocols were devised as cellular analogs of the plasticity of RF properties, which is normally expressed during a critical period of postnatal development. We summarize here evidence demonstrating that changes in covariance between afferent input and postsynaptic response imposed during extracellular and intracellular conditioning can acutely induce selective long-lasting up- and down-regulations of visual responses. The functional properties that could be modified in 40% of cells submitted to differential pairing protocols include ocular dominance, orientation selectivity and orientation preference, interocular orientation disparity, and the relative dominance of ON and OFF responses. Since changes in RF properties can be induced in the adult as well, our findings also suggest that similar activity-dependent processes may occur during development and during active phases of learning under the supervision of behavioral attention or contextual signals. Such potential for plasticity in primary visual cortical neurons suggests the existence of a hidden connectivity expressing a wider functional competence than the one revealed at the spiking level. In particular, in the spatial domain the sensory synaptic integration field is larger than the classical discharge field. It can be shaped by supervised learning and its subthreshold extent can be unmasked by the pharmacological blockade of intracortical inhibition.     

Persistent sodium current is a nonsynaptic substrate for long-term associative memory

Although synaptic plasticity is widely regarded as the primary mechanism of memory [1], forms of nonsynaptic plasticity, such as increased somal or dendritic excitability or membrane potential depolarization, also have been implicated in learning in both vertebrate and invertebrate experimental systems [2], [3], [4], [5], [6] and [7]. Compared to synaptic plasticity, however, there is much less information available on the mechanisms of specific types of nonsynaptic plasticity involved in well-defined examples of behavioral memory. Recently, we have shown that learning-induced somal depolarization of an identified modulatory cell type (the cerebral giant cells, CGCs) of the snail Lymnaea stagnalis encodes information that enables the expression of long-term associative memory [8]. The Lymnaea CGCs therefore provide a highly suitable experimental system for investigating the ionic mechanisms of nonsynaptic plasticity that can be linked to behavioral learning. Based on a combined behavioral, electrophysiological, immunohistochemical, and computer simulation approach, here we show that an increase of a persistent sodium current of this neuron underlies its delayed and persistent depolarization after behavioral single-trial classical conditioning. Our findings provide new insights into how learning-induced membrane level changes are translated into a form of long-lasting neuronal plasticity already known to contribute to maintained adaptive modifications at the network and behavioral level [8].     

Robustness of learning that is based on covariance-driven synaptic plasticity

It is widely believed that learning is due, at least in part, to long-lasting modifications of the strengths of synapses in the brain. Theoretical studies have shown that a family of synaptic plasticity rules, in which synaptic changes are driven by covariance, is particularly useful for many forms of learning, including associative memory, gradient estimation, and operant conditioning. Covariance-based plasticity is inherently sensitive. Even a slight mistuning of the parameters of a covariance-based plasticity rule is likely to result in substantial changes in synaptic efficacies. Therefore, the biological relevance of covariance-based plasticity models is questionable. Here, we study the effects of mistuning parameters of the plasticity rule in a decision making model in which synaptic plasticity is driven by the covariance of reward and neural activity. An exact covariance plasticity rule yields Herrnstein's matching law. We show that although the effect of slight mistuning of the plasticity rule on the synaptic efficacies is large, the behavioral effect is small. Thus, matching behavior is robust to mistuning of the parameters of the covariance-based plasticity rule. Furthermore, the mistuned covariance rule results in undermatching, which is consistent with experimentally observed behavior. These results substantiate the hypothesis that approximate covariance-based synaptic plasticity underlies operant conditioning. However, we show that the mistuning of the mean subtraction makes behavior sensitive to the mistuning of the properties of the decision making network. Thus, there is a tradeoff between the robustness of matching behavior to changes in the plasticity rule and its robustness to changes in the properties of the decision making network.     

Time and tide in cerebellar memory formation

The notion that the olivocerebellar system is crucial for motor learning is well established. In recent years, it has become evident that there can be many forms of both synaptic and non-synaptic plasticity within this system and that each might have a different role in developing and maintaining motor learning across a wide range of tasks. There are several possible molecular and cellular mechanisms that could underlie adaptation of the vestibulo-ocular reflex and eyeblink conditioning. Although causal relationships between particular cellular processes and individual components of a learned behaviour have not been demonstrated unequivocally, an overall picture is emerging that the different types and sites of cellular plasticity relate importantly to the stage of learning and/or its temporal specifics.     

Molecular mechanisms of fear learning and memory

Pavlovian fear conditioning is a particularly useful behavioral paradigm for exploring the molecular mechanisms of learning and memory because a well-defined response to a specific environmental stimulus is produced through associative learning processes. Synaptic plasticity in the lateral nucleus of the amygdala (LA) underlies this form of associative learning. Here, we summarize the molecular mechanisms that contribute to this synaptic plasticity in the context of auditory fear conditioning, the form of fear conditioning best understood at the molecular level. We discuss the neurotransmitter systems and signaling cascades that contribute to three phases of auditory fear conditioning: acquisition, consolidation, and reconsolidation. These studies suggest that multiple intracellular signaling pathways, including those triggered by activation of Hebbian processes and neuromodulatory receptors, interact to produce neural plasticity in the LA and behavioral fear conditioning. Collectively, this body of research illustrates the power of fear conditioning as a model system for characterizing the mechanisms of learning and memory in mammals and potentially for understanding fear-related disorders, such as PTSD and phobias.     

Fear Extinction as a Model for Synaptic Plasticity in Major Depressive Disorder

Background

The neuroplasticity hypothesis of major depressive disorder proposes that a dysfunction of synaptic plasticity represents a basic pathomechanism of the disorder. Animal models of depression indicate enhanced plasticity in a ventral emotional network, comprising the amygdala. Here, we investigated fear extinction learning as a non-invasive probe for amygdala-dependent synaptic plasticity in patients with major depressive disorder and healthy controls.

Methods

Differential fear conditioning was measured in 37 inpatients with severe unipolar depression (International Classification of Diseases, 10^th revision, criteria) and 40 healthy controls. The eye-blink startle response, a subcortical output signal that is modulated by local synaptic plasticity in the amygdala in fear acquisition and extinction learning, was recorded as the primary outcome parameter.

Results

After robust and similar fear acquisition in both groups, patients with major depressive disorder showed significantly enhanced fear extinction learning in comparison to healthy controls, as indicated by startle responses to conditioned stimuli. The strength of extinction learning was positively correlated with the total illness duration.

Conclusions

The finding of enhanced fear extinction learning in major depressive disorder is consistent with the concept that the disorder is characterized by enhanced synaptic plasticity in the amygdala and the ventral emotional network. Clinically, the observation emphasizes the potential of successful extinction learning, the basis of exposure therapy, in anxiety-related disorders despite the frequent comorbidity of major depressive disorder.     

SRC3 acetylates calmodulin in the mouse brain to regulate synaptic plasticity and fear learning

Protein acetylation is a reversible posttranslational modification, which is regulated by lysine acetyltransferase (KAT) and lysine deacetyltransferase (KDAC). Although protein acetylation has been shown to regulate synaptic plasticity, this was mainly for histone protein acetylation. The function and regulation of nonhistone protein acetylation in synaptic plasticity and learning remain largely unknown. Calmodulin (CaM), a ubiquitous Ca²⁺ sensor, plays critical roles in synaptic plasticity such as long-term potentiation (LTP). During LTP induction, activation of NMDA receptor triggers Ca²⁺ influx, and the Ca²⁺ binds with CaM and activates calcium/calmodulin-dependent protein kinase IIα (CaMKIIα). In our previous study, we demonstrated that acetylation of CaM was important for synaptic plasticity and fear learning in mice. However, the KAT responsible for CaM acetylation is currently unknown. Here, following an HEK293 cell-based screen of candidate KATs, steroid receptor coactivator 3 (SRC3) is identified as the most active KAT for CaM. We further demonstrate that SRC3 interacts with and acetylates CaM in a Ca²⁺ and NMDA receptor-dependent manner. We also show that pharmacological inhibition or genetic downregulation of SRC3 impairs CaM acetylation, synaptic plasticity, and contextual fear learning in mice. Moreover, the effects of SRC3 inhibition on synaptic plasticity and fear learning could be rescued by 3KQ-CaM, a mutant form of CaM, which mimics acetylation. Together, these observations demonstrate that SRC3 acetylates CaM and regulates synaptic plasticity and learning in mice.     

Coexistence of Reward and Unsupervised Learning During the Operant Conditioning of Neural Firing Rates

A fundamental goal of neuroscience is to understand how cognitive processes, such as operant conditioning, are performed by the brain. Typical and well studied examples of operant conditioning, in which the firing rates of individual cortical neurons in monkeys are increased using rewards, provide an opportunity for insight into this. Studies of reward-modulated spike-timing-dependent plasticity (RSTDP), and of other models such as R-max, have reproduced this learning behavior, but they have assumed that no unsupervised learning is present (i.e., no learning occurs without, or independent of, rewards). We show that these models cannot elicit firing rate reinforcement while exhibiting both reward learning and ongoing, stable unsupervised learning. To fix this issue, we propose a new RSTDP model of synaptic plasticity based upon the observed effects that dopamine has on long-term potentiation and depression (LTP and LTD). We show, both analytically and through simulations, that our new model can exhibit unsupervised learning and lead to firing rate reinforcement. This requires that the strengthening of LTP by the reward signal is greater than the strengthening of LTD and that the reinforced neuron exhibits irregular firing. We show the robustness of our findings to spike-timing correlations, to the synaptic weight dependence that is assumed, and to changes in the mean reward. We also consider our model in the differential reinforcement of two nearby neurons. Our model aligns more strongly with experimental studies than previous models and makes testable predictions for future experiments.     

Brain plasticity and ion channels

It is generally believed that spatio-temporal configurations of distributed activity in the brain contribute to the coding of neuronal information and that synaptic contacts between nerve cells could play a central role in the formation of privileged pathways of activity. Synaptic plasticity is not the only mode of regulation of information processing in the brain and persistent regulations of ionic conductances in some specialized neuronal areas such as the dendrites, the cell body and the axon could also modulate, in the short- and the long-term, the propagation of information in the brain. Persistent changes in intrinsic excitability have been reported in several brain areas in which activity is modified during a classical conditioning. The role of synaptic activity seems to be determinant in the induction but the learning rules and the underlying mechanisms remain to be defined. This review discusses the role of neuronal activity in the induction of intrinsic plasticity in cortical, hippocampal and cerebellar neurons. Activation and inactivation properties of ionic channels in the axon determine the short-term dynamics of axonal propagation and synaptic transmission. Activation of glutamate receptors initiates a long-term modification in neuronal excitability that may represent the substrate for the mnesic engram and for the stabilization of the epileptic state. Similarly to synaptic plasticity, long-lasting intrinsic plasticity appears to be reversible and to express a certain level of input or cellular specificity. These non-synaptic forms of plasticity affect the signal propagation in the axon, the dendrites and the soma. They not only share common learning rules and induction pathways with the better known synaptic plasticity such as NMDA receptor-dependent LTP and LTD but also contribute in synergy with these synaptic changes to the formation of a coherent mnesic engram.     

Oligomeric Amyloid-β Inhibits the Proteolytic Conversion of Brain-derived Neurotrophic Factor (BDNF), AMPA Receptor Trafficking,and Classical Conditioning

Amyloid-β (Aβ) peptide is thought to have a significant role in the progressive memory loss observed in patients with Alzheimer disease and inhibits synaptic plasticity in animal models of learning. We previously demonstrated that brain-derived neurotrophic factor (BDNF) is critical for synaptic AMPA receptor delivery in an in vitro model of eyeblink classical conditioning. Here, we report that acquisition of conditioned responses was significantly attenuated by bath application of oligomeric (200 nm), but not fibrillar, Aβ peptide. Western blotting revealed that BDNF protein expression during conditioning is significantly reduced by treatment with oligomeric Aβ, as were phosphorylation levels of cAMP-response element-binding protein (CREB), Ca²⁺/calmodulin-dependent protein kinase II (CaMKII), Ca²⁺/calmodulin-dependent protein kinase IV (CaMKIV), and ERK. However, levels of PKA and PKCζ/λ were unaffected, as was PDK-1. Protein localization studies using confocal imaging indicate that oligomeric Aβ, but not fibrillar or scrambled forms, suppresses colocalization of GluR1 and GluR4 AMPA receptor subunits with synaptophysin, indicating that trafficking of these subunits to synapses during the conditioning procedure is blocked. In contrast, coapplication of BDNF with oligomeric Aβ significantly reversed these findings. Interestingly, a tolloid-like metalloproteinase in turtle, tTLLs (turtle tolloid-like protein), which normally processes the precursor proBDNF into mature BDNF, was found to degrade oligomeric Aβ into small fragments. These data suggest that an Aβ-induced reduction in BDNF, perhaps due to interference in the proteolytic conversion of proBDNF to BDNF, results in inhibition of synaptic AMPA receptor delivery and suppression of the acquisition of conditioning.     

Interplay between Short- and Long-Term Plasticity in Cell-Assembly Formation

Various hippocampal and neocortical synapses of mammalian brain show both short-term plasticity and long-term plasticity, which are considered to underlie learning and memory by the brain. According to Hebb’s postulate, synaptic plasticity encodes memory traces of past experiences into cell assemblies in cortical circuits. However, it remains unclear how the various forms of long-term and short-term synaptic plasticity cooperatively create and reorganize such cell assemblies. Here, we investigate the mechanism in which the three forms of synaptic plasticity known in cortical circuits, i.e., spike-timing-dependent plasticity (STDP), short-term depression (STD) and homeostatic plasticity, cooperatively generate, retain and reorganize cell assemblies in a recurrent neuronal network model. We show that multiple cell assemblies generated by external stimuli can survive noisy spontaneous network activity for an adequate range of the strength of STD. Furthermore, our model predicts that a symmetric temporal window of STDP, such as observed in dopaminergic modulations on hippocampal neurons, is crucial for the retention and integration of multiple cell assemblies. These results may have implications for the understanding of cortical memory processes.     

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.     

Altered synaptic plasticity and behavioral abnormalities in CNGA3‐deficient mice

The role of the cyclic nucleotide‐gated (CNG) channel CNGA3 is well established in cone photoreceptors and guanylyl cyclase‐D‐expressing olfactory neurons. To assess a potential function of CNGA3 in the mouse amygdala and hippocampus, we examined synaptic plasticity and performed a comparative analysis of spatial learning, fear conditioning and step‐down avoidance in wild‐type mice and CNGA3 null mutants (CNGA3^?/?). CNGA3^?/? mice showed normal basal synaptic transmission in the amygdala and the hippocampus. However, cornu Ammonis (CA1) hippocampal long‐term potentiation (LTP) induced by a strong tetanus was significantly enhanced in CNGA3^?/? mice as compared with their wild‐type littermates. Unlike in the hippocampus, LTP was not significantly altered in the amygdala of CNGA3^?/? mice. Enhanced hippocampal LTP did not coincide with changes in hippocampus‐dependent learning, as both wild‐type and mutant mice showed a similar performance in water maze tasks and contextual fear conditioning, except for a trend toward higher step‐down latencies in a passive avoidance task. In contrast, CNGA3^?/? mice showed markedly reduced freezing to the conditioned tone in the amygdala‐dependent cued fear conditioning task. In conclusion, our study adds a new entry on the list of physiological functions of the CNGA3 channel. Despite the dissociation between physiological and behavioral parameters, our data describe a so far unrecognized role of CNGA3 in modulation of hippocampal plasticity and amydgala‐dependent fear memory.