Coexistence of Reward and Unsupervised Learning During the Operant Conditioning of Neural Firing Rates

A fundamental goal of neuroscience is to understand how cognitive processes, such as operant conditioning, are performed by the brain. Typical and well studied examples of operant conditioning, in which the firing rates of individual cortical neurons in monkeys are increased using rewards, provide an opportunity for insight into this. Studies of reward-modulated spike-timing-dependent plasticity (RSTDP), and of other models such as R-max, have reproduced this learning behavior, but they have assumed that no unsupervised learning is present (i.e., no learning occurs without, or independent of, rewards). We show that these models cannot elicit firing rate reinforcement while exhibiting both reward learning and ongoing, stable unsupervised learning. To fix this issue, we propose a new RSTDP model of synaptic plasticity based upon the observed effects that dopamine has on long-term potentiation and depression (LTP and LTD). We show, both analytically and through simulations, that our new model can exhibit unsupervised learning and lead to firing rate reinforcement. This requires that the strengthening of LTP by the reward signal is greater than the strengthening of LTD and that the reinforced neuron exhibits irregular firing. We show the robustness of our findings to spike-timing correlations, to the synaptic weight dependence that is assumed, and to changes in the mean reward. We also consider our model in the differential reinforcement of two nearby neurons. Our model aligns more strongly with experimental studies than previous models and makes testable predictions for future experiments.     

Spatio-temporal credit assignment in neuronal population learning

In learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.     

Balancing homeostasis and learning in neural circuits

Neural circuits are remarkably adaptable, providing animals with the ability to modify their behavior on the basis of experience. At the same time, they are extremely robust and maintain stability despite the changes associated with adaptation. This combination of adaptability and stability is difficult to achieve, and it provides a strong constraint on any models of plasticity in neural circuits. New evidence suggests that the effect of action potential timing on synaptic plasticity may be an important element in reconciling homeostasis with adaptability. In particular, spike-timing dependent plasticity can act as both an adaptive and a homeostatic mechanism, controlling overall firing rates and distributions of synaptic efficacies while making neurons selective for certain aspects of their inputs. It can also cause networks that initially represent the present state of a stimulus to predict its future state on the basis of experience, a theoretical result supported by experimental data in behaving rats.     

Decision making in recurrent neuronal circuits

Decision making has recently emerged as a central theme in neurophysiological studies of cognition, and experimental and computational work has led to the proposal of a cortical circuit mechanism of elemental decision computations. This mechanism depends on slow recurrent synaptic excitation balanced by fast feedback inhibition, which not only instantiates attractor states for forming categorical choices but also long transients for gradually accumulating evidence in favor of or against alternative options. Such a circuit endowed with reward-dependent synaptic plasticity is able to produce adaptive choice behavior. While decision threshold is a core concept for reaction time tasks, it can be dissociated from a general decision rule. Moreover, perceptual decisions and value-based economic choices are described within a unified framework in which probabilistic choices result from irregular neuronal activity as well as iterative interactions of a decision maker with an uncertain environment or other unpredictable decision makers in a social group.     

Self-Organization of Microcircuits in Networks of Spiking Neurons with Plastic Synapses

The synaptic connectivity of cortical networks features an overrepresentation of certain wiring motifs compared to simple random-network models. This structure is shaped, in part, by synaptic plasticity that promotes or suppresses connections between neurons depending on their joint spiking activity. Frequently, theoretical studies focus on how feedforward inputs drive plasticity to create this network structure. We study the complementary scenario of self-organized structure in a recurrent network, with spike timing-dependent plasticity driven by spontaneous dynamics. We develop a self-consistent theory for the evolution of network structure by combining fast spiking covariance with a slow evolution of synaptic weights. Through a finite-size expansion of network dynamics we obtain a low-dimensional set of nonlinear differential equations for the evolution of two-synapse connectivity motifs. With this theory in hand, we explore how the form of the plasticity rule drives the evolution of microcircuits in cortical networks. When potentiation and depression are in approximate balance, synaptic dynamics depend on weighted divergent, convergent, and chain motifs. For additive, Hebbian STDP these motif interactions create instabilities in synaptic dynamics that either promote or suppress the initial network structure. Our work provides a consistent theoretical framework for studying how spiking activity in recurrent networks interacts with synaptic plasticity to determine network structure.     

A decision-making model based on a spiking neural circuit and synaptic plasticity

To adapt to the environment and survive, most animals can control their behaviors by making decisions. The process of decision-making and responding according to cues in the environment is stable, sustainable, and learnable. Understanding how behaviors are regulated by neural circuits and the encoding and decoding mechanisms from stimuli to responses are important goals in neuroscience. From results observed in Drosophila experiments, the underlying decision-making process is discussed, and a neural circuit that implements a two-choice decision-making model is proposed to explain and reproduce the observations. Compared with previous two-choice decision making models, our model uses synaptic plasticity to explain changes in decision output given the same environment. Moreover, biological meanings of parameters of our decision-making model are discussed. In this paper, we explain at the micro-level (i.e., neurons and synapses) how observable decision-making behavior at the macro-level is acquired and achieved.     

Progressive plasticity of auditory cortex during appetitive operant conditioning

In stimulus-response-outcome learning, different regions in the cortico-basal ganglia network are progressively involved according to the stage of learning. However, the involvement of sensory cortex remains ellusive even though massive cortical projections to the striatum imply its significant role in this learning. Here we show that the global tonotopic representation in the auditory cortex changed progressively depending on the stage of training in auditory operant conditioning. At the early stage, tone-responsive areas mainly in the core cortex expanded, while both the core and belt cortices shrank at the late stage as behavior became conditioned. Taken together with previous findings, this progressive global plasticity from the core to belt cortices suggests differentiated roles in these areas: the core cortex serves as a filter to better identify auditory objects for hierarchical computation within the belt cortex, while the belt stores auditory objects and affects decision making through direct projections to limbic system and higher association cortex. Thus, the progressive plasticity in the present study reflects a shift from identification to storage of a behaviorally relevant auditory object, which is potentially associated with a habitual behavior.     

Plasticity in the Rat Prefrontal Cortex: Linking Gene Expression and an Operant Learning with a Computational Theory

The plasticity in the medial Prefrontal Cortex (mPFC) of rodents or lateral prefrontal cortex in non human primates (lPFC), plays a key role neural circuits involved in learning and memory. Several genes, like brain-derived neurotrophic factor (BDNF), cAMP response element binding (CREB), Synapsin I, Calcium/calmodulin-dependent protein kinase II (CamKII), activity-regulated cytoskeleton-associated protein (Arc), c-jun and c-fos have been related to plasticity processes. We analysed differential expression of related plasticity genes and immediate early genes in the mPFC of rats during learning an operant conditioning task. Incompletely and completely trained animals were studied because of the distinct events predicted by our computational model at different learning stages. During learning an operant conditioning task, we measured changes in the mRNA levels by Real-Time RT-PCR during learning; expression of these markers associated to plasticity was incremented while learning and such increments began to decline when the task was learned. The plasticity changes in the lPFC during learning predicted by the model matched up with those of the representative gene BDNF. Herein, we showed for the first time that plasticity in the mPFC in rats during learning of an operant conditioning is higher while learning than when the task is learned, using an integrative approach of a computational model and gene expression.     

Stability of the memory of eye position in a recurrent network of conductance-based model neurons

Studies of the neural correlates of short-term memory in a wide variety of brain areas have found that transient inputs can cause persistent changes in rates of action potential firing, through a mechanism that remains unknown. In a premotor area that is responsible for holding the eyes still during fixation, persistent neural firing encodes the angular position of the eyes in a characteristic manner: below a threshold position the neuron is silent, and above it the firing rate is linearly related to position. Both the threshold and linear slope vary from neuron to neuron. We have reproduced this behavior in a biophysically plausible network model. Persistence depends on precise tuning of the strength of synaptic feedback, and a relatively long synaptic time constant improves the robustness to mistuning.     

A Calcium-Dependent Plasticity Rule for HCN Channels Maintains Activity Homeostasis and Stable Synaptic Learning

Theoretical and computational frameworks for synaptic plasticity and learning have a long and cherished history, with few parallels within the well-established literature for plasticity of voltage-gated ion channels. In this study, we derive rules for plasticity in the hyperpolarization-activated cyclic nucleotide-gated (HCN) channels, and assess the synergy between synaptic and HCN channel plasticity in establishing stability during synaptic learning. To do this, we employ a conductance-based model for the hippocampal pyramidal neuron, and incorporate synaptic plasticity through the well-established Bienenstock-Cooper-Munro (BCM)-like rule for synaptic plasticity, wherein the direction and strength of the plasticity is dependent on the concentration of calcium influx. Under this framework, we derive a rule for HCN channel plasticity to establish homeostasis in synaptically-driven firing rate, and incorporate such plasticity into our model. In demonstrating that this rule for HCN channel plasticity helps maintain firing rate homeostasis after bidirectional synaptic plasticity, we observe a linear relationship between synaptic plasticity and HCN channel plasticity for maintaining firing rate homeostasis. Motivated by this linear relationship, we derive a calcium-dependent rule for HCN-channel plasticity, and demonstrate that firing rate homeostasis is maintained in the face of synaptic plasticity when moderate and high levels of cytosolic calcium influx induced depression and potentiation of the HCN-channel conductance, respectively. Additionally, we show that such synergy between synaptic and HCN-channel plasticity enhances the stability of synaptic learning through metaplasticity in the BCM-like synaptic plasticity profile. Finally, we demonstrate that the synergistic interaction between synaptic and HCN-channel plasticity preserves robustness of information transfer across the neuron under a rate-coding schema. Our results establish specific physiological roles for experimentally observed plasticity in HCN channels accompanying synaptic plasticity in hippocampal neurons, and uncover potential links between HCN-channel plasticity and calcium influx, dynamic gain control and stable synaptic learning.     

Role of delayed nonsynaptic neuronal plasticity in long-term associative memory

BACKGROUND: It is now well established that persistent nonsynaptic neuronal plasticity occurs after learning and, like synaptic plasticity, it can be the substrate for long-term memory. What still remains unclear, though, is how nonsynaptic plasticity contributes to the altered neural network properties on which memory depends. Understanding how nonsynaptic plasticity is translated into modified network and behavioral output therefore represents an important objective of current learning and memory research. RESULTS: By using behavioral single-trial classical conditioning together with electrophysiological analysis and calcium imaging, we have explored the cellular mechanisms by which experience-induced nonsynaptic electrical changes in a neuronal soma remote from the synaptic region are translated into synaptic and circuit level effects. We show that after single-trial food-reward conditioning in the snail Lymnaea stagnalis, identified modulatory neurons that are extrinsic to the feeding network become persistently depolarized between 16 and 24 hr after training. This is delayed with respect to early memory formation but concomitant with the establishment and duration of long-term memory. The persistent nonsynaptic change is extrinsic to and maintained independently of synaptic effects occurring within the network directly responsible for the generation of feeding. Artificial membrane potential manipulation and calcium-imaging experiments suggest a novel mechanism whereby the somal depolarization of an extrinsic neuron recruits command-like intrinsic neurons of the circuit underlying the learned behavior. CONCLUSIONS: We show that nonsynaptic plasticity in an extrinsic modulatory neuron encodes information that enables the expression of long-term associative memory, and we describe how this information can be translated into modified network and behavioral output.     

Molecular mechanisms underlying emotional learning and memory in the lateral amygdala

Fear conditioning is a valuable behavioral paradigm for studying the neural basis of emotional learning and memory. The lateral nucleus of the amygdala (LA) is a crucial site of neural changes that occur during fear conditioning. Pharmacological manipulations of the LA, strategically timed with respect to training and testing, have shed light on the molecular events that mediate the acquisition of fear associations and the formation and maintenance of long-term memories of those associations. Similar mechanisms have been found to underlie long-term potentiation (LTP) in LA, an artificial means of inducing synaptic plasticity and a physiological model of learning and memory. Thus, LTP-like changes in synaptic plasticity may underlie fear conditioning. Given that the neural circuit underlying fear conditioning has been implicated in emotional disorders in humans, the molecular mechanisms of fear conditioning are potential targets for psychotherapeutic drug development.     

Learning of Precise Spike Times with Homeostatic Membrane Potential Dependent Synaptic Plasticity

Precise spatio-temporal patterns of neuronal action potentials underly e.g. sensory representations and control of muscle activities. However, it is not known how the synaptic efficacies in the neuronal networks of the brain adapt such that they can reliably generate spikes at specific points in time. Existing activity-dependent plasticity rules like Spike-Timing-Dependent Plasticity are agnostic to the goal of learning spike times. On the other hand, the existing formal and supervised learning algorithms perform a temporally precise comparison of projected activity with the target, but there is no known biologically plausible implementation of this comparison. Here, we propose a simple and local unsupervised synaptic plasticity mechanism that is derived from the requirement of a balanced membrane potential. Since the relevant signal for synaptic change is the postsynaptic voltage rather than spike times, we call the plasticity rule Membrane Potential Dependent Plasticity (MPDP). Combining our plasticity mechanism with spike after-hyperpolarization causes a sensitivity of synaptic change to pre- and postsynaptic spike times which can reproduce Hebbian spike timing dependent plasticity for inhibitory synapses as was found in experiments. In addition, the sensitivity of MPDP to the time course of the voltage when generating a spike allows MPDP to distinguish between weak (spurious) and strong (teacher) spikes, which therefore provides a neuronal basis for the comparison of actual and target activity. For spatio-temporal input spike patterns our conceptually simple plasticity rule achieves a surprisingly high storage capacity for spike associations. The sensitivity of the MPDP to the subthreshold membrane potential during training allows robust memory retrieval after learning even in the presence of activity corrupted by noise. We propose that MPDP represents a biophysically plausible mechanism to learn temporal target activity patterns.     

Somato-dendritic Synaptic Plasticity and Error-backpropagation in Active Dendrites

In the last decade dendrites of cortical neurons have been shown to nonlinearly combine synaptic inputs by evoking local dendritic spikes. It has been suggested that these nonlinearities raise the computational power of a single neuron, making it comparable to a 2-layer network of point neurons. But how these nonlinearities can be incorporated into the synaptic plasticity to optimally support learning remains unclear. We present a theoretically derived synaptic plasticity rule for supervised and reinforcement learning that depends on the timing of the presynaptic, the dendritic and the postsynaptic spikes. For supervised learning, the rule can be seen as a biological version of the classical error-backpropagation algorithm applied to the dendritic case. When modulated by a delayed reward signal, the same plasticity is shown to maximize the expected reward in reinforcement learning for various coding scenarios. Our framework makes specific experimental predictions and highlights the unique advantage of active dendrites for implementing powerful synaptic plasticity rules that have access to downstream information via backpropagation of action potentials.     

Estimating synaptic parameters from mean, variance, and covariance in trains of synaptic responses

Fluctuation analysis of synaptic transmission using the variance-mean approach has been restricted in the past to steady-state responses. Here we extend this method to short repetitive trains of synaptic responses, during which the response amplitudes are not stationary. We consider intervals between trains, long enough so that the system is in the same average state at the beginning of each train. This allows analysis of ensemble means and variances for each response in a train separately. Thus, modifications in synaptic efficacy during short-term plasticity can be attributed to changes in synaptic parameters. In addition, we provide practical guidelines for the analysis of the covariance between successive responses in trains. Explicit algorithms to estimate synaptic parameters are derived and tested by Monte Carlo simulations on the basis of a binomial model of synaptic transmission, allowing for quantal variability, heterogeneity in the release probability, and postsynaptic receptor saturation and desensitization. We find that the combined analysis of variance and covariance is advantageous in yielding an estimate for the number of release sites, which is independent of heterogeneity in the release probability under certain conditions. Furthermore, it allows one to calculate the apparent quantal size for each response in a sequence of stimuli.     

Operant conditioning in invertebrates

Learning to anticipate future events on the basis of past experience with the consequences of one's own behavior (operant conditioning) is a simple form of learning that humans share with most other animals, including invertebrates. Three model organisms have recently made significant contributions towards a mechanistic model of operant conditioning, because of their special technical advantages. Research using the fruit fly Drosophila melanogaster implicated the ignorant gene in operant conditioning in the heat-box, research on the sea slug Aplysia californica contributed a cellular mechanism of behavior selection at a convergence point of operant behavior and reward, and research on the pond snail Lymnaea stagnalis elucidated the role of a behavior-initiating neuron in operant conditioning. These insights demonstrate the usefulness of a variety of invertebrate model systems to complement and stimulate research in vertebrates.     

Stability of complex spike timing-dependent plasticity in cerebellar learning

Dynamics of spike-timing dependent synaptic plasticity are analyzed for excitatory and inhibitory synapses onto cerebellar Purkinje cells. The purpose of this study is to place theoretical constraints on candidate synaptic learning rules that determine the changes in synaptic efficacy due to pairing complex spikes with presynaptic spikes in parallel fibers and inhibitory interneurons. Constraints are derived for the timing between complex spikes and presynaptic spikes, constraints that result from the stability of the learning dynamics of the learning rule. Potential instabilities in the parallel fiber synaptic learning rule are found to be stabilized by synaptic plasticity at inhibitory synapses if the inhibitory learning rules are stable, and conditions for stability of inhibitory plasticity are given. Combining excitatory with inhibitory plasticity provides a mechanism for minimizing the overall synaptic input. Stable learning rules are shown to be able to sculpt simple-spike patterns by regulating the excitability of neurons in the inferior olive that give rise to climbing fibers.     

Synaptic plasticity and connectivity requirements to produce stimulus-pair specific responses in recurrent networks of spiking neurons

Animals must respond selectively to specific combinations of salient environmental stimuli in order to survive in complex environments. A task with these features, biconditional discrimination, requires responses to select pairs of stimuli that are opposite to responses to those stimuli in another combination. We investigate the characteristics of synaptic plasticity and network connectivity needed to produce stimulus-pair neural responses within randomly connected model networks of spiking neurons trained in biconditional discrimination. Using reward-based plasticity for synapses from the random associative network onto a winner-takes-all decision-making network representing perceptual decision-making, we find that reliably correct decision making requires upstream neurons with strong stimulus-pair selectivity. By chance, selective neurons were present in initial networks; appropriate plasticity mechanisms improved task performance by enhancing the initial diversity of responses. We find long-term potentiation of inhibition to be the most beneficial plasticity rule by suppressing weak responses to produce reliably correct decisions across an extensive range of networks.     

Activity-dependent regulation of receptive field properties of cat area 17 by supervised Hebbian learning.

Most algorithms currently used to model synaptic plasticity in self-organizing cortical networks suppose that the change in synaptic efficacy is governed by the same structuring factor, i.e., the temporal correlation of activity between pre- and postsynaptic neurons. Functional predictions generated by such algorithms have been tested electrophysiologically in the visual cortex of anesthetized and paralyzed cats. Supervised learning procedures were applied at the cellular level to change receptive field (RF) properties during the time of recording of an individual functionally identified cell. The protocols were devised as cellular analogs of the plasticity of RF properties, which is normally expressed during a critical period of postnatal development. We summarize here evidence demonstrating that changes in covariance between afferent input and postsynaptic response imposed during extracellular and intracellular conditioning can acutely induce selective long-lasting up- and down-regulations of visual responses. The functional properties that could be modified in 40% of cells submitted to differential pairing protocols include ocular dominance, orientation selectivity and orientation preference, interocular orientation disparity, and the relative dominance of ON and OFF responses. Since changes in RF properties can be induced in the adult as well, our findings also suggest that similar activity-dependent processes may occur during development and during active phases of learning under the supervision of behavioral attention or contextual signals. Such potential for plasticity in primary visual cortical neurons suggests the existence of a hidden connectivity expressing a wider functional competence than the one revealed at the spiking level. In particular, in the spatial domain the sensory synaptic integration field is larger than the classical discharge field. It can be shaped by supervised learning and its subthreshold extent can be unmasked by the pharmacological blockade of intracortical inhibition.     

Tag-Trigger-Consolidation: A Model of Early and Late Long-Term-Potentiation and Depression

Changes in synaptic efficacies need to be long-lasting in order to serve as a substrate for memory. Experimentally, synaptic plasticity exhibits phases covering the induction of long-term potentiation and depression (LTP/LTD) during the early phase of synaptic plasticity, the setting of synaptic tags, a trigger process for protein synthesis, and a slow transition leading to synaptic consolidation during the late phase of synaptic plasticity. We present a mathematical model that describes these different phases of synaptic plasticity. The model explains a large body of experimental data on synaptic tagging and capture, cross-tagging, and the late phases of LTP and LTD. Moreover, the model accounts for the dependence of LTP and LTD induction on voltage and presynaptic stimulation frequency. The stabilization of potentiated synapses during the transition from early to late LTP occurs by protein synthesis dynamics that are shared by groups of synapses. The functional consequence of this shared process is that previously stabilized patterns of strong or weak synapses onto the same postsynaptic neuron are well protected against later changes induced by LTP/LTD protocols at individual synapses.