Dopamine scales performance in the absence of new learning

Learning and motivation are integral in shaping an organism's adaptive behavior. The dopamine system has been implicated in both processes; however, dissociating the two, both experimentally and conceptually, has posed significant challenges. We have developed an animal model that dissociates expression or scaling of a learned behavior from learning itself. An inducible dopamine transporter (DAT) knockdown mouse line has been generated, which exhibits significantly slower reuptake of released dopamine and increased tonic firing of dopamine neurons without altering phasic burst firing. Mice were trained in experimental tasks prior to inducing a hyperdopaminergic tone and then retested. Elevated dopamine enhanced performance in goal-directed operant responses. These data demonstrate that alterations in dopaminergic tone can scale the performance of a previously learned behavior in the absence of new learning.     

Speed/accuracy trade-off between the habitual and the goal-directed processes

Instrumental responses are hypothesized to be of two kinds: habitual and goal-directed, mediated by the sensorimotor and the associative cortico-basal ganglia circuits, respectively. The existence of the two heterogeneous associative learning mechanisms can be hypothesized to arise from the comparative advantages that they have at different stages of learning. In this paper, we assume that the goal-directed system is behaviourally flexible, but slow in choice selection. The habitual system, in contrast, is fast in responding, but inflexible in adapting its behavioural strategy to new conditions. Based on these assumptions and using the computational theory of reinforcement learning, we propose a normative model for arbitration between the two processes that makes an approximately optimal balance between search-time and accuracy in decision making. Behaviourally, the model can explain experimental evidence on behavioural sensitivity to outcome at the early stages of learning, but insensitivity at the later stages. It also explains that when two choices with equal incentive values are available concurrently, the behaviour remains outcome-sensitive, even after extensive training. Moreover, the model can explain choice reaction time variations during the course of learning, as well as the experimental observation that as the number of choices increases, the reaction time also increases. Neurobiologically, by assuming that phasic and tonic activities of midbrain dopamine neurons carry the reward prediction error and the average reward signals used by the model, respectively, the model predicts that whereas phasic dopamine indirectly affects behaviour through reinforcing stimulus-response associations, tonic dopamine can directly affect behaviour through manipulating the competition between the habitual and the goal-directed systems and thus, affect reaction time.     

Transient changes in mesolimbic dopamine and their association with 'reward'

Mesolimbic dopaminergic neurons modulate complex circuitry in the ventral forebrain involved in reward processing, although the precise function of the dopaminergic input is debated. Electrophysiological measurements have revealed that mesolimbic dopaminergic neurons can fire in either tonic or phasic modes, and that phasic firing accompanies the alerting or anticipatory phases of reward. However, the neurochemical relevance of this rapid neuronal discharge within the reward processing circuitry is not yet clear, in part because of difficulty in interpretation of extracellular dopamine measurements. Herein, the nature of the information provided by different neurochemical techniques is critically discussed. Classical methods of monitoring dopamine reveal changes in extracellular dopamine resulting from tonic neuronal activity, but do not have the temporal resolution to distinguish concentration transients. However, recent advances in dopamine sensors now enable transient dopamine concentrations resulting from phasic firing to be positively identified and followed on a physiologically relevant timescale. This has enabled demonstrations of discrete, phasic dopamine signals accompanying rewarding or alerting stimuli. Thus, enhanced dopamine release at terminals appears to be coincident with phasic electrical activity at cell bodies. These accumulating data promise to help unravel the precise role of phasic dopamine transmission in reward processing.     

NMDA receptors in dopaminergic neurons are crucial for habit learning

Dopamine is crucial for habit learning. Activities of midbrain dopaminergic neurons are regulated by the cortical and subcortical signals among which glutamatergic afferents provide excitatory inputs. Cognitive implications of glutamatergic afferents in regulating and engaging dopamine signals during habit learning, however, remain unclear. Here, we show that mice with dopaminergic neuron-specific NMDAR1 deletion are impaired in a variety of habit-learning tasks, while normal in some other dopamine-modulated functions such as locomotor activities, goal-directed learning, and spatial reference memories. In vivo neural recording revealed that dopaminergic neurons in these mutant mice could still develop the cue-reward association responses; however, their conditioned response robustness was drastically blunted. Our results suggest that integration of glutamatergic inputs to DA neurons by NMDA receptors, likely by regulating associative activity patterns, is a crucial part of the cellular mechanism underpinning habit learning.     

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.     

Sources contributing to the average extracellular concentration of dopamine in the nucleus accumbens

Mesolimbic dopamine neurons fire in both tonic and phasic modes resulting in detectable extracellular levels of dopamine in the nucleus accumbens (NAc). In the past, different techniques have targeted dopamine levels in the NAc to establish a basal concentration. In this study, we used in vivo fast scan cyclic voltammetry (FSCV) in the NAc of awake, freely moving rats. The experiments were primarily designed to capture changes in dopamine caused by phasic firing - that is, the measurement of dopamine 'transients'. These FSCV measurements revealed for the first time that spontaneous dopamine transients constitute a major component of extracellular dopamine levels in the NAc. A series of experiments were designed to probe regulation of extracellular dopamine. Lidocaine was infused into the ventral tegmental area, the site of dopamine cell bodies, to arrest neuronal firing. While there was virtually no instantaneous change in dopamine concentration, longer sampling revealed a decrease in dopamine transients and a time-averaged decrease in the extracellular level. Dopamine transporter inhibition using intravenous GBR12909 injections increased extracellular dopamine levels changing both frequency and size of dopamine transients in the NAc. To further unmask the mechanics governing extracellular dopamine levels we used intravenous injection of the vesicular monoamine transporter (VMAT2) inhibitor, tetrabenazine, to deplete dopamine storage and increase cytoplasmic dopamine in the nerve terminals. Tetrabenazine almost abolished phasic dopamine release but increased extracellular dopamine to ～500?nM, presumably by inducing reverse transport by dopamine transporter (DAT). Taken together, data presented here show that average extracellular dopamine in the NAc is low (20-30?nM) and largely arises from phasic dopamine transients.     

Repeated exposure to methamphetamine causes long-lasting presynaptic corticostriatal depression that is renormalized with drug readministration

Addiction-associated behaviors such as drug craving and relapse are hypothesized to result from synaptic changes that persist long after withdrawal and are renormalized by drug reinstatement, although such chronic synaptic effects have not been identified. We report that exposure to the dopamine releaser methamphetamine for 10 days elicits a long-lasting (>4 month) depression at corticostriatal terminals that is reversed by methamphetamine readministration. Both methamphetamine-induced chronic presynaptic depression and the drug's selective renormalization in drug-experienced animals are independent of corresponding long-term changes in synaptic dopamine release but are due to alterations in D1 dopamine and cholinergic receptor systems. These mechanisms might provide a synaptic basis that underlies addiction and habit learning and their long-term maintenance.     

A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem—action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.     

Attenuated Tonic and Enhanced Phasic Release of Dopamine in Attention Deficit Hyperactivity Disorder

It is unclear whether attention deficit hyperactive disorder (ADHD) is a hypodopaminergic or hyperdopaminergic condition. Different sets of data suggest either hyperactive or hypoactive dopamine system. Since indirect methods used in earlier studies have arrived at contradictory conclusions, we directly measured the tonic and phasic release of dopamine in ADHD volunteers. The tonic release in ADHD and healthy control volunteers was measured and compared using dynamic molecular imaging technique. The phasic release during performance of Eriksen’s flanker task was measured in the two groups using single scan dynamic molecular imaging technique. In these experiments volunteers were positioned in a positron emission tomography (PET) camera and administered a dopamine receptor ligand ¹¹C-raclopride intravenously. After the injection PET data were acquired dynamically while volunteers either stayed still (tonic release experiments) or performed the flanker task (phasic release experiments). PET data were analyzed to measure dynamic changes in ligand binding potential (BP) and other receptor kinetic parameters. The analysis revealed that at rest the ligand BP was significantly higher in the right caudate of ADHD volunteers suggesting reduced tonic release. During task performance significantly lower ligand BP was observed in the same area, indicating increased phasic release. In ADHD tonic release of dopamine is attenuated and the phasic release is enhanced in the right caudate. By characterizing the nature of dysregulated dopamine neurotransmission in ADHD, the results explain earlier findings of reduced or increased dopaminergic activity.     

Examination of Rapid Dopamine Dynamics with Fast Scan Cyclic Voltammetry During Intra-oral Tastant Administration in Awake Rats

Rapid, phasic dopamine (DA) release in the mammalian brain plays a critical role in reward processing, reinforcement learning, and motivational control. Fast scan cyclic voltammetry (FSCV) is an electrochemical technique with high spatial and temporal (sub-second) resolution that has been utilized to examine phasic DA release in several types of preparations. In vitro experiments in single-cells and brain slices and in vivo experiments in anesthetized rodents have been used to identify mechanisms that mediate dopamine release and uptake under normal conditions and in disease models. Over the last 20 years, in vivo FSCV experiments in awake, freely moving rodents have also provided insight of dopaminergic mechanisms in reward processing and reward learning. One major advantage of the awake, freely moving preparation is the ability to examine rapid DA fluctuations that are time-locked to specific behavioral events or to reward or cue presentation. However, one limitation of combined behavior and voltammetry experiments is the difficulty of dissociating DA effects that are specific to primary rewarding or aversive stimuli from co-occurring DA fluctuations that mediate reward-directed or other motor behaviors. Here, we describe a combined method using in vivo FSCV and intra-oral infusion in an awake rat to directly investigate DA responses to oral tastants. In these experiments, oral tastants are infused directly to the palate of the rat – bypassing reward-directed behavior and voluntary drinking behavior – allowing for direct examination of DA responses to tastant stimuli.     

An association between prediction errors and risk-seeking: Theory and behavioral evidence

Reward prediction errors (RPEs) and risk preferences have two things in common: both can shape decision making behavior, and both are commonly associated with dopamine. RPEs drive value learning and are thought to be represented in the phasic release of striatal dopamine. Risk preferences bias choices towards or away from uncertainty; they can be manipulated with drugs that target the dopaminergic system. Based on the common neural substrate, we hypothesize that RPEs and risk preferences are linked on the level of behavior as well. Here, we develop this hypothesis theoretically and test it empirically. First, we apply a recent theory of learning in the basal ganglia to predict how RPEs influence risk preferences. We find that positive RPEs should cause increased risk-seeking, while negative RPEs should cause risk-aversion. We then test our behavioral predictions using a novel bandit task in which value and risk vary independently across options. Critically, conditions are included where options vary in risk but are matched for value. We find that our prediction was correct: participants become more risk-seeking if choices are preceded by positive RPEs, and more risk-averse if choices are preceded by negative RPEs. These findings cannot be explained by other known effects, such as nonlinear utility curves or dynamic learning rates.     

Real-time decoding of dopamine concentration changes in the caudate-putamen during tonic and phasic firing

The fundamental process that underlies volume transmission in the brain is the extracellular diffusion of neurotransmitters from release sites to distal target cells. Dopaminergic neurons display a range of activity states, from low-frequency tonic firing to bursts of high-frequency action potentials (phasic firing). However, it is not clear how this activity affects volume transmission on a subsecond time scale. To evaluate this, we developed a finite-difference model that predicts the lifetime and diffusion of dopamine in brain tissue. We first used this model to decode in vivo amperometric measurements of electrically evoked dopamine, and obtained rate constants for release and uptake as well as the extent of diffusion. Accurate predictions were made under a variety of conditions including different regions, different stimulation parameters and with uptake inhibited. Second, we used the decoded rate constants to predict how heterogeneity of dopamine release and uptake sites would affect dopamine concentration fluctuations during different activity states in the absence of an electrode. These simulations show that synchronous phasic firing can produce spatially and temporally heterogeneous concentration profiles whereas asynchronous tonic firing elicits uniform, steady-state dopamine concentrations.     

Getting formal with dopamine and reward

Recent neurophysiological studies reveal that neurons in certain brain structures carry specific signals about past and future rewards. Dopamine neurons display a short-latency, phasic reward signal indicating the difference between actual and predicted rewards. The signal is useful for enhancing neuronal processing and learning behavioral reactions. It is distinctly different from dopamine's tonic enabling of numerous behavioral processes. Neurons in the striatum, frontal cortex, and amygdala also process reward information but provide more differentiated information for identifying and anticipating rewards and organizing goal-directed behavior. The different reward signals have complementary functions, and the optimal use of rewards in voluntary behavior would benefit from interactions between the signals. Addictive psychostimulant drugs may exert their action by amplifying the dopamine reward signal.     

Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex

Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior.     

The lesion of the rat substantia nigra pars compacta dopaminergic neurons as a model for Parkinson's disease memory disabilities

1. In this article we review the studies of memory disabilities in a rat model o Parkinson's disease (PD).2. Intranigral administration of 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP) to rats causes a partial lesion in the substantia nigra, compact part (SNc) and a specific loss of dopamine and its metabolites in the striatum of rats.3. These animals present learning and memory deficits but no sensorimotor impairments, thus modeling the early phase of PD when cognitive impairments are observed but the motor symptoms of the disease are barely present.4. The cognitive deficits observed in these animals affect memory tasks proposed to model habit learning (the cued version of the water maze task and the two-way active avoidance task) and working memory (a working memory version of the water maze), but spare long-term spatial memory (the spatial reference version of the Morris water maze).5. The treatment of these animals with levodopa in a dose that restores the striatal level of dopamine does not reverse these memory impairments, probably because this treatment promotes a high level of dopamine in extrastriatal brain regions, such as the prefrontal cortex and the hippocampus.6. On the other hand, the adenosine receptor antagonist, caffeine, partly reverse the memory impairment effect of SNc lesion in these rats. This effect may be due to caffeine action on nigrostriatal neurons, since it induces dopamine release and modulates the interaction between adenosine and dopamine receptor activity.7. These results suggest that the MPTP SNc-lesioned rats are a good model to study memory disabilities related to PD and that caffeine and other selective A(2A) adenosine receptor antagonists are promising drugs to treat this symptoms in PD patients.     

Homeostatic regulation mechanism of spontaneous firing determines glutamate responsiveness in the midbrain dopamine neurons

Autonomous tonic firing of the midbrain dopamine neuron is essential for maintenance of ambient dopamine level in the brain, in which intracellular Ca²⁺ concentration ([Ca²⁺]c) plays a complex but pivotal role. However, little is known about Ca²⁺ signals by which dopamine neurons maintain an optimum spontaneous firing rate. In the midbrain dopamine neurons, we here show that spontaneous firing evoked [Ca²⁺]c changes in a phasic manner in the dendritic region but a tonic manner in the soma. Tonic levels of somatic [Ca²⁺]c strictly tallied with spontaneous firing rates. However, manipulatory raising or lowering of [Ca²⁺]c with caged compounds from the resting firing state proportionally suppressed or raised spontaneous firing rate, respectively, suggesting presence of the homeostatic regulation mechanism for spontaneous firing rate via tonic [Ca²⁺]c changes of the soma. More importantly, abolition of this homeostatic regulation mechanism significantly exaggerated the responses of tonic firings and high-frequency phasic discharges to glutamate. Therefore, we conclude that this Ca²⁺-dependent homeostatic regulation mechanism is responsible for not only maintaining optimum rate of spontaneous firing, but also proper responses to glutamate. Perturbation of this mechanism could cause dopamine neurons to be more vulnerable to glutamate and Ca²⁺ toxicities.     

Electrographic correlates of the antiamnestic action of nootropic agents following deprivation of the paradoxical sleep phase

During experiments conducted on albino rats the deficiency of passive avoidance retention was shown to correlate not only with the reduction in REM sleep and SWS, but also with disappearance of phasic component of theta-rhythm. Drugs with nootropic mode of action (cleregyl, centrophenoxin, antioxidant 3-xypyridine) recovered the deficiency of passive avoidance retention and increased phasic component of theta-Rhythm, while phenazepam enhanced tonic component of theta-rhythm, and failed to act upon learning deficits. It seems likely from these results that the electrophysiological correlates of antiamnestic effect is the maintenance of proper two-component theta-rhythm and the increase in its phasic component, whereas the destructuring of sleep, including REM sleep reduction is not considered to be key determinant in the action upon memory and learning procedure.     

Ghrelin regulates phasic dopamine and nucleus accumbens signaling evoked by food‐predictive stimuli

Environmental stimuli that signal food availability hold powerful sway over motivated behavior and promote feeding, in part, by activating the mesolimbic system. These food‐predictive cues evoke brief (phasic) changes in nucleus accumbens (NAc) dopamine concentration and in the activity of individual NAc neurons. Phasic fluctuations in mesolimbic signaling have been directly linked to goal‐directed behaviors, including behaviors elicited by food‐predictive cues. Food‐seeking behavior is also strongly influenced by physiological state (i.e., hunger vs. satiety). Ghrelin, a stomach hormone that crosses the blood‐brain barrier, is linked to the perception of hunger and drives food intake, including intake potentiated by environmental cues. Notwithstanding, whether ghrelin regulates phasic mesolimbic signaling evoked by food‐predictive stimuli is unknown. Here, rats underwent Pavlovian conditioning in which one cue predicted the delivery of rewarding food (CS+) and a second cue predicted nothing (CS?). After training, we measured the effect of ghrelin infused into the lateral ventricle (LV) on sub‐second fluctuations in NAc dopamine using fast‐scan cyclic voltammetry and individual NAc neuron activity using in vivo electrophysiology in separate groups of rats. LV ghrelin augmented both phasic dopamine and phasic increases in the activity of NAc neurons evoked by the CS+. Importantly, ghrelin did not affect the dopamine nor NAc neuron response to the CS?, suggesting that ghrelin selectively modulated mesolimbic signaling evoked by motivationally significant stimuli. These data demonstrate that ghrelin, a hunger signal linked to physiological state, can regulate cue‐evoked mesolimbic signals that underlie food‐directed behaviors.

     

The Effects of Acute Dopamine Precursor Depletion on the Reinforcing Value of Exercise in Anorexia Nervosa

This study investigated whether dopaminergic systems are involved in the motivation to engage in behaviours associated with anorexia nervosa (AN), specifically, the drive to exercise. Women recovered from AN (AN REC, n = 17) and healthy controls (HC, n = 15) were recruited. The acute phenylalanine/tyrosine depletion (APTD) method was used to transiently decrease dopamine synthesis and transmission. The effect of dopamine precursor depletion on drive to exercise was measured using a progressive ratio (PR) exercise breakpoint task. Both groups worked for the opportunity to exercise, and, at baseline, PR breakpoint scores were higher in AN REC than HC. Compared to values on the experimental control session, APTD did not decrease PR breakpoint scores in AN REC, but significantly decreased scores in HC. These data show that women recovered from AN are more motivated to exercise than HC, although in both groups, activity is more reinforcing than inactivity. Importantly, decreasing dopamine does not reduce the motivation to exercise in people recovered from AN, but in contrast, does so in HC. It is proposed that in AN, drive to exercise develops into a behaviour that is largely independent of dopamine mediated reward processes and becomes dependent on cortico-striatal neurocircuitry that regulates automated, habit- or compulsive-like behaviours. These data strengthen the case for the involvement of reward, learning, habit, and dopaminergic systems in the aetiology of AN.