Axiomatic methods, dopamine and reward prediction error

The phasic firing rate of midbrain dopamine neurons has been shown to respond both to the receipt of rewarding stimuli, and the degree to which such stimuli are anticipated by the recipient. This has led to the hypothesis that these neurons encode reward prediction error (RPE)-the difference between how rewarding an event is, and how rewarding it was expected to be. However, the RPE model is one of a number of competing explanations for dopamine activity that have proved hard to disentangle, mainly because they are couched in terms of latent, or unobservable, variables. This article describes techniques for dealing with latent variables common in economics and decision theory, and reviews work that uses these techniques to provide simple, non-parametric tests of the RPE hypothesis, allowing clear differentiation between competing explanations.     

Akt1 deficiency modulates reward learning and reward prediction error in mice

In contemporary reinforcement learning models, reward prediction error (RPE), the difference between the expected and actual reward, is thought to guide action value learning through the firing activity of dopaminergic neurons. Given the importance of dopamine in reward learning and the involvement of Akt1 in dopamine-dependent behaviors, the aim of this study was to investigate whether Akt1 deficiency modulates reward learning and the magnitude of RPE using Akt1 mutant mice as a model. In comparison to wild-type littermate controls, the expression of Akt1 proteins in mouse brains occurred in a gene-dosage-dependent manner and Akt1 heterozygous (HET) mice exhibited impaired striatal Akt1 activity under methamphetamine challenge. No genotypic difference was found in the basal levels of dopamine and its metabolites. In a series of reward-related learning tasks, HET mice displayed a relatively efficient method of updating reward information from the environment during the acquisition phase of the two natural reward tasks and in the reverse section of the dynamic foraging T-maze but not in methamphetamine-induced or aversive-related reward learning. The implementation of a standard reinforcement learning model and the Bayesian hierarchical parameter estimation show that HET mice have higher RPE magnitudes and that their action values are updated more rapidly among all three test sections in T-maze. These results indicate that Akt1 deficiency modulates natural reward learning and RPE. This study showed a promising avenue for investigating RPE in mutant mice and provided evidence for the potential link from genetic deficiency, to neurobiological abnormalities, to impairment in higher-order cognitive functioning.     

Stimulus-dependent adjustment of reward prediction error in the midbrain

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.     

Excitatory SST neurons in the medial paralemniscal nucleus control repetitive self-grooming and encode reward

1. Download : Download high-res image (234KB)
2. Download : Download full-size image

     

Dopamine neurons can represent context-dependent prediction error

Midbrain dopamine (DA) neurons are thought to encode reward prediction error. Reward prediction can be improved if any relevant context is taken into account. We found that monkey DA neurons can encode a context-dependent prediction error. In the first noncontextual task, a light stimulus was randomly followed by reward, with a fixed equal probability. The response of DA neurons was positively correlated with the number of preceding unrewarded trials and could be simulated by a conventional temporal difference (TD) model. In the second contextual task, a reward-indicating light stimulus was presented with the probability that, while fixed overall, was incremented as a function of the number of preceding unrewarded trials. The DA neuronal response then was negatively correlated with this number. This history effect corresponded to the prediction error based on the conditional probability of reward and could be simulated only by implementing the relevant context into the TD model.     

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.     

Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement

Activation of dopamine receptors in forebrain regions, for minutes or longer, is known to be sufficient for positive reinforcement of stimuli and actions. However, the firing rate of dopamine neurons is increased for only about 200 milliseconds following natural reward events that are better than expected, a response which has been described as a "reward prediction error" (RPE). Although RPE drives reinforcement learning (RL) in computational models, it has not been possible to directly test whether the transient dopamine signal actually drives RL. Here we have performed optical stimulation of genetically targeted ventral tegmental area (VTA) dopamine neurons expressing Channelrhodopsin-2 (ChR2) in mice. We mimicked the transient activation of dopamine neurons that occurs in response to natural reward by applying a light pulse of 200 ms in VTA. When a single light pulse followed each self-initiated nose poke, it was sufficient in itself to cause operant reinforcement. Furthermore, when optical stimulation was delivered in separate sessions according to a predetermined pattern, it increased locomotion and contralateral rotations, behaviors that are known to result from activation of dopamine neurons. All three of the optically induced operant and locomotor behaviors were tightly correlated with the number of VTA dopamine neurons that expressed ChR2, providing additional evidence that the behavioral responses were caused by activation of dopamine neurons. These results provide strong evidence that the transient activation of dopamine neurons provides a functional reward signal that drives learning, in support of RL theories of dopamine function.     

A pair of motion-sensitive neurons in the locust encode approaches of a looming object

Neurons in the locust visual system encode approaches of looming stimuli and are implicated in production of escape behaviours. The lobula giant movement detector (LGMD) and its postsynaptic partner, the descending contralateral movement detector (DCMD) compute characteristics of expanding edges across the locust eye during a loom and DCMD synapses onto motor elements associated with behaviour. We identified another descending interneuron within the locust ventral nerve cord. We named this neuron the late DCMD (LDCMD) as it responds later during an approach, with the firing rate peaking at about the time of collision. LDCMD produced lower amplitude, broader action potentials that were associated with an afterhyperpolarization, whereas DCMD action potentials showed a brief afterhyperpolarization often followed by an afterdepolarization. Within the mesothoracic ganglion, the primary LDCMD axon located adjacent to the DCMD axon, was thinner and lacked collateral projections to the lateral region of the neuropil. When compared with DCMD, LDCMD fired with fewer spikes during a loom and showed weaker habituation to repeated approaches. Coincidence of LDCMD and DCMD firing increased during object approach. Our findings indicate the presence of an additional motion-sensitive descending neuron in the locust that encodes temporally distinct properties of an approaching object.     

Serotonergic neurons transiently require a midline-derived FGF signal

In the grasshopper CNS, serotonergic growth cones cross the midline early in development and initiate expression of serotonin uptake activity, or SERT. To test if the midline contains an activity that induces SERT, cuts were made that separated serotonergic cell bodies from the midline. SERT activity is completely lost when the midline is separated but is then rescued by bath-applied FGF2 (fibroblast growth factor 2), which can activate the heartless FGF receptor. heartless is expressed specifically in serotonergic neurons. A candidate FGF-like molecule was identified that is expressed in a subset of midline glia. SERT-expressing severed growth cones continue to migrate to their correct targets, which indicates that by the time SERT is activated, the serotonergic growth cones are committed to target-directed growth.     

Random signal variation as a cause of systematic microdensitometric error

A high noise/signal ratio in microdensitometry, owing, for example, to the use of a very small measuring spot or light at an extreme of the spectrum, decreases the precision (increases the variability) of results. This can be compensated for by making more measurements, but under some circumstances random signal variations can also cause systematic errors that are not so easily corrected. The present article explains how such errors can arise when measuring either very pale or very dense objects and suggests methods to detect and avoid them. The principles described are applicable to many types of microdensitometer, not only the Vickers flying-spot instruments used in this investigation.     

A comparison of signal sequence prediction methods using a test set of signal peptides

We describe the creation of a test set containing secretory and non-secretory proteins. Five existing prediction programs for signal sequences and their cleavage sites are compared on the basis of this test set: SPScan, SigCleave, SignalP V1.1, SignalP V2.0. b2-HMM and SignalP V2.0.b2-NN.     

Edaravone guards dopamine neurons in a rotenone model for Parkinson's disease

3-methyl-1-phenyl-2-pyrazolin-5-one (edaravone), an effective free radical scavenger, provides neuroprotection in stroke models and patients. In this study, we investigated its neuroprotective effects in a chronic rotenone rat model for Parkinson's disease. Here we showed that a five-week treatment with edaravone abolished rotenone's activity to induce catalepsy, damage mitochondria and degenerate dopamine neurons in the midbrain of rotenone-treated rats. This abolishment was attributable at least partly to edaravone's inhibition of rotenone-induced reactive oxygen species production or apoptotic promoter Bax expression and its up-regulation of the vesicular monoamine transporter 2 (VMAT2) expression. Collectively, edaravone may provide novel clinical therapeutics for PD.     

Unique properties of mesoprefrontal neurons within a dual mesocorticolimbic dopamine system

The mesocorticolimbic dopamine system is essential for cognitive and emotive brain functions and is thus an important target in major brain diseases like schizophrenia, drug addiction, and attention deficit hyperactivity disorder. However, the cellular basis for the diversity in behavioral functions and associated dopamine-release pattern within the mesocorticolimbic system has remained unclear. Here, we report the identification of a type of dopaminergic neuron within the mesocorticolimbic dopamine system with unconventional fast-firing properties and small DAT/TH mRNA expression ratios that selectively projects to prefrontal cortex and nucleus accumbens core and medial shell as well as to basolateral amygdala. In contrast, well-described conventional slow-firing dopamine midbrain neurons only project to the lateral shell of the nucleus accumbens and the dorsolateral striatum. Among this dual dopamine midbrain system defined in this study by converging anatomical, electrophysiological, and molecular properties, mesoprefrontal dopaminergic neurons are unique, as only they do not possess functional somatodendritic Girk2-coupled dopamine D2 autoreceptors.     

Seventeen base pairs of region I encode a novel tripartite binding signal for SV40 T antigen

Three sequence components direct high affinity binding of dimeric SV40 T antigen to SV40 origin region I. Two signals are encoded by two directly repeated 5′-GAGGC-3′ pentanucleotides. Approximately equal contributions to binding stability are made by each pentanucleotide, and both spacing and orientation of the pentanucleotides are important for binding affinity. The third vital component is contained in a 5′-TTTTTTG-3′ spacer sequence that separates the pentanucleotides. Sequence-specific features of the spacer stabilize binding to the adjacent pentanucleotides. The asymmetry of the spacer suggests that a novel binding mechanism is involved. Because the alignment of T antigen on mutant and wild-type DNAs is similar, we propose that any two of the three sequence signals are sufficient to determine the unique arrangement of a bound protein dimer.