Getting formal with dopamine and reward

Recent neurophysiological studies reveal that neurons in certain brain structures carry specific signals about past and future rewards. Dopamine neurons display a short-latency, phasic reward signal indicating the difference between actual and predicted rewards. The signal is useful for enhancing neuronal processing and learning behavioral reactions. It is distinctly different from dopamine's tonic enabling of numerous behavioral processes. Neurons in the striatum, frontal cortex, and amygdala also process reward information but provide more differentiated information for identifying and anticipating rewards and organizing goal-directed behavior. The different reward signals have complementary functions, and the optimal use of rewards in voluntary behavior would benefit from interactions between the signals. Addictive psychostimulant drugs may exert their action by amplifying the dopamine reward signal.     

Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology

Neurons in a small number of brain structures detect rewards and reward-predicting stimuli and are active during the expectation of predictable food and liquid rewards. These neurons code the reward information according to basic terms of various behavioural theories that seek to explain reward-directed learning, approach behaviour and decision-making. The involved brain structures include groups of dopamine neurons, the striatum including the nucleus accumbens, the orbitofrontal cortex and the amygdala. The reward information is fed to brain structures involved in decision-making and organisation of behaviour, such as the dorsolateral prefrontal cortex and possibly the parietal cortex. The neural coding of basic reward terms derived from formal theories puts the neurophysiological investigation of reward mechanisms on firm conceptual grounds and provides neural correlates for the function of rewards in learning, approach behaviour and decision-making.     

Ventromedial prefrontal cortex activation is associated with memory formation for predictable rewards

During reinforcement learning, dopamine release shifts from the moment of reward consumption to the time point when the reward can be predicted. Previous studies provide consistent evidence that reward-predicting cues enhance long-term memory (LTM) formation of these items via dopaminergic projections to the ventral striatum. However, it is less clear whether memory for items that do not precede a reward but are directly associated with reward consumption is also facilitated. Here, we investigated this question in an fMRI paradigm in which LTM for reward-predicting and neutral cues was compared to LTM for items presented during consumption of reliably predictable as compared to less predictable rewards. We observed activation of the ventral striatum and enhanced memory formation during reward anticipation. During processing of less predictable as compared to reliably predictable rewards, the ventral striatum was activated as well, but items associated with less predictable outcomes were remembered worse than items associated with reliably predictable outcomes. Processing of reliably predictable rewards activated the ventromedial prefrontal cortex (vmPFC), and vmPFC BOLD responses were associated with successful memory formation of these items. Taken together, these findings show that consumption of reliably predictable rewards facilitates LTM formation and is associated with activation of the vmPFC.     

Reward representations and reward-related learning in the human brain: insights from neuroimaging

This review outlines recent findings from human neuroimaging concerning the role of a highly interconnected network of brain areas including orbital and medial prefrontal cortex, amygdala, striatum and dopaminergic mid-brain in reward processing. Distinct reward-related functions can be attributed to different components of this network. Orbitofrontal cortex is involved in coding stimulus reward value and in concert with the amygdala and ventral striatum is implicated in representing predicted future reward. Such representations can be used to guide action selection for reward, a process that depends, at least in part, on orbital and medial prefrontal cortex as well as dorsal striatum.     

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Pavlovian reward prediction and receipt in schizophrenia: relationship to anhedonia

Reward processing abnormalities have been implicated in the pathophysiology of negative symptoms such as anhedonia and avolition in schizophrenia. However, studies examining neural responses to reward anticipation and receipt have largely relied on instrumental tasks, which may confound reward processing abnormalities with deficits in response selection and execution. 25 chronic, medicated outpatients with schizophrenia and 20 healthy controls underwent functional magnetic resonance imaging using a pavlovian reward prediction paradigm with no response requirements. Subjects passively viewed cues that predicted subsequent receipt of monetary reward or non-reward, and blood-oxygen-level-dependent signal was measured at the time of cue presentation and receipt. At the group level, neural responses to both reward anticipation and receipt were largely similar between groups. At the time of cue presentation, striatal anticipatory responses did not differ between patients and controls. Right anterior insula demonstrated greater activation for nonreward than reward cues in controls, and for reward than nonreward cues in patients. At the time of receipt, robust responses to receipt of reward vs. nonreward were seen in striatum, midbrain, and frontal cortex in both groups. Furthermore, both groups demonstrated responses to unexpected versus expected outcomes in cortical areas including bilateral dorsolateral prefrontal cortex. Individual difference analyses in patients revealed an association between physical anhedonia and activity in ventral striatum and ventromedial prefrontal cortex during anticipation of reward, in which greater anhedonia severity was associated with reduced activation to money versus no-money cues. In ventromedial prefrontal cortex, this relationship held among both controls and patients, suggesting a relationship between anticipatory activity and anhedonia irrespective of diagnosis. These findings suggest that in the absence of response requirements, brain responses to reward receipt are largely intact in medicated individuals with chronic schizophrenia, while reward anticipation responses in left ventral striatum are reduced in those patients with greater anhedonia severity.     

Changes in brain monoamine oxidase activity in conditioned passive avoidance reaction in rats]

Activity of the enzyme monoamine oxidase (MAO) and kinetic parameters (Km, Vmax) for the 5-hydroxytryptamine (5-HT) and dopamine deamination were examined in the brain of rats with conditioned passive avoidance recall. Changes of the 5-HT and dopamine deamination were found in amygdala, striatum and frontal cortex. MAO activity was not changed in hippocampus. In amygdala the rate of 5-HT deamination was significantly increased and kinetic studies revealed increased affinity of the enzyme for 5-HT. The metabolism of dopamine in amygdala was unchanged. In frontal cortex the deamination of 5-HT was not changed, but the dopamine deamination significantly decreased. This decrease was due to lowering of MAO affinity for dopamine. In striatum the metabolism of both 5-HT and dopamine was reduced, and kinetic studies showed the lowering of Vmax for 5-HT and dopamine deamination.     

Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice

In choosing between different rewards expected after unequal delays, humans and animals often prefer the smaller but more immediate reward, indicating that the subjective value or utility of reward is depreciated according to its delay. Here, we show that neurons in the primate caudate nucleus and ventral striatum modulate their activity according to temporally discounted values of rewards with a similar time course. However, neurons in the caudate nucleus encoded the difference in the temporally discounted values of the two alternative targets more reliably than neurons in the ventral striatum. In contrast, neurons in the ventral striatum largely encoded the sum of the temporally discounted values, and therefore, the overall goodness of available options. These results suggest a more pivotal role for the dorsal striatum in action selection during intertemporal choice.     

Saccade reward signals in posterior cingulate cortex

Movement selection depends on the outcome of prior behavior. Posterior cingulate cortex (CGp) is strongly connected with both limbic and oculomotor circuitry, and CGp neurons respond following saccades, suggesting a role in signaling the motivational outcome of gaze shifts. To test this hypothesis, single CGp neurons were studied in monkeys while they shifted gaze to visual targets for liquid rewards that varied in size or were delivered probabilistically. CGp neurons responded following saccades as well as following reward delivery, and these responses were correlated with reward size. CGp neurons also responded following the omission of predicted rewards. The timing of CGp activation and its modulation by reward could provide signals useful for updating representations of expected saccade value.     

Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs

How do addictive drugs hijack the brain's reward system? This review speculates how normal, physiological reward processes may be affected by addictive drugs. Addictive drugs affect acute responses and plasticity in dopamine neurons and postsynaptic structures. These effects reduce reward discrimination, increase the effects of reward prediction error signals, and enhance neuronal responses to reward-predicting stimuli, which may contribute to compulsion. Addictive drugs steepen neuronal temporal reward discounting and create temporal myopia that impairs the control of drug taking. Tonically enhanced dopamine levels may disturb working memory mechanisms necessary for assessing background rewards and thus may generate inaccurate neuronal reward predictions. Drug-induced working memory deficits may impair neuronal risk signaling, promote risky behaviors, and facilitate preaddictive drug use. Malfunctioning adaptive reward coding may lead to overvaluation of drug rewards. Many of these malfunctions may result in inadequate neuronal decision mechanisms and lead to choices biased toward drug rewards.     

Food Reward-Induced Neurotransmitter Changes in Cognitive Brain Regions

Recent evidence indicates that mechanisms involved in reward and mechanisms involved in learning interact, in that reward includes learning processes and learning includes reward processes. In spite of such interactions, reward and learning represent distinct functions. In the present study, as part of an examination of the differences in learning and reward mechanisms, it was assumed that food principally affects reward mechanisms. After a brief period of fasting, we assayed the release of three neurotransmitters and their associated metabolites in eight brain areas associated with learning and memory as a response to feeding. Using microdialysis for the assay, we found changes in the hippocampus, cortex, amygdala, and the thalamic nucleus, (considered cognitive areas), in addition to those in the nucleus accumbens and ventral tegmental area (considered reward areas). Extracellular dopamine levels increased in the nucleus accumbens, ventral tegmental area, amygdala, and thalamic nucleus, while they decreased in the hippocampus and prefrontal cortex. Dopamine metabolites increased in all areas tested (except the dorsal hippocampus); changes in norepinephrine varied with decreases in the accumbens, dorsal hippocampus, amygdala, and thalamic nucleus, and increases in the prefrontal cortex; serotonin levels decreased in all the areas tested; although its metabolite 5HIAA increased in two regions (the medial temporal cortex, and thalamic nucleus). Our assays indicate that in reward activities such as feeding, in addition to areas usually associated with reward such as the mesolimbic dopamine system, other areas associated with cognition also participate. Results also indicate that several transmitter systems play a part, with several neurotransmitters and several receptors involved in the response to food in a number of brain structures, and the changes in transmitter levels may be affected by metabolism and transport in addition to changes in release in a regionally heterogeneous manner. Food reward represents a complex pattern of changes in the brain that involve cognitive processes. Although food reward elements overlap with other reward systems sharing some neurotransmitter compounds, it significantly differs indicating a specific reward to process for food consumption. Like in other rewards, both learning and cognitive areas play a significant part in food reward. Special issue dedicated to Dr. Moussa Youdim.     

Dissociable reward and timing signals in human midbrain and ventral striatum

Reward prediction error (RPE) signals are central to current models of reward-learning. Temporal difference (TD) learning models posit that these signals should be modulated by predictions, not only of magnitude but also timing of reward. Here we show that BOLD activity in the VTA conforms to such TD predictions: responses to unexpected rewards are modulated by a temporal hazard function and activity between a predictive stimulus and reward is depressed in proportion to predicted reward. By contrast, BOLD activity in ventral striatum (VS) does not reflect a TD RPE, but instead encodes a signal on the variable relevant for behavior, here timing but not magnitude of reward. The results have important implications for dopaminergic models of cortico-striatal learning and suggest a modification of the conventional view that VS BOLD necessarily reflects inputs from dopaminergic VTA neurons signaling an RPE.     

Quantitative immunochemistry on neuronal loss, reactive gliosis and BBB damage in cortex/striatum and hippocampus/amygdala after systemic kainic acid administration

Cell specific markers were quantified in the hippocampus, the amygdala/pyriform cortex, the frontal cerebral cortex and the striatum of the rat brain after systemic administration of kainic acid. Neuron specific enolase (NSE) reflects loss of neurons, glial fibrillary acidic protein (GFAP) reflects reactive gliosis, and brain levels of serum proteins measures blood-brain-barrier permeability. While the concentration of NSE remained unaffected in the frontal cerebral cortex and the striatum, their GFAP content increased during the first three days. In the hippocampus and amygdala, NSE levels decreased significantly. GFAP levels in the hippocampus were unaffected after one day and decreased in the amygdala/pyriform cortex. After that, GFAP increased strikingly until day 9 or, in the case of amygdala/pyriform cortex, even longer. This biphasic time course for GFAP was accompanied by a decrease of S-100 during days 1-9 followed by a significant increase at day 27 above the initial level. The regional differences in GFAP and S-100 could result from the degree of neuronal degeneration, the astrocytic receptor set-up and/or effects on the blood-brain barrier.     

Whole-brain mapping of direct inputs to midbrain dopamine neurons

Recent studies indicate that dopamine neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) convey distinct signals. To explore this difference, we comprehensively identified each area's monosynaptic inputs using the rabies virus. We show that dopamine neurons in both areas integrate inputs from a more diverse collection of areas than previously thought, including autonomic, motor, and somatosensory areas. SNc and VTA dopamine neurons receive contrasting excitatory inputs: the former from the somatosensory/motor cortex and subthalamic nucleus, which may explain their short-latency responses to salient events; and the latter from the lateral hypothalamus, which may explain their involvement in value coding. We demonstrate that neurons in the striatum that project directly to dopamine neurons form patches in both the dorsal and ventral striatum, whereas those projecting to GABAergic neurons are distributed in the matrix compartment. Neuron-type-specific connectivity lays a foundation for studying how dopamine neurons compute outputs.     

Somatostatin Content Increases Following Norepinephrine Depletion in Frontal Cortex of l-Methyl-4-Phenyl-1,2,3,6-Tetrahydropyridine-Treated Mice

The effects of 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP) on somatostatin (SS)-containing neurons were examined by measuring dopamine, norepinephrine (NE), SS, and SS mRNA in striatum and frontal cortex of C57/B16 mice at various times following treatment with MPTP-HCl (96 mg/kg i.p.). MPTP caused a 70% depletion of dopamine in striatum by 1 day and a 40% depletion of NE in frontal cortex within 3 days. SS content was increased in frontal cortex 4 days later, but not in striatum; there were no changes in SS mRNA. Maprotiline, a specific NE-uptake blocker, prevented both the depletion of NE and the increase of SS in frontal cortex due to MPTP administration. These results support the possibility that NE can regulate SS in frontal cortex and are discussed in terms of the decrease of SS seen in parkinsonian patients with dementia.     

The Neural Circuitry of Reward Processing in Complex Social Comparison: Evidence from an Event-Related fMRI Study

In this study, Functional magnetic resonance imaging (fMRI) was conducted to investigate the mechanisms by which the brain activity in a complex social comparison context. One true subject and two pseudo-subjects were asked to complete a simple number estimate task at the same time which including upward and downward comparisons. Two categories of social comparison rewards (fair and unfair rewards distributions) were mainly presented by comparing the true subject with other two pseudo-subjects. Particularly, there were five conditions of unfair distribution when all the three subjects were correct but received different rewards. Behavioral data indicated that the ability to self-regulate was important in satisfaction judgment when the subject perceived an unfair reward distribution. fMRI data indicated that the interaction between the ventral striatum and the prefrontal cortex was important in self-regulation under specific conditions in complex social comparison, especially under condition of reward processing when there were two different reward values and the subject failed to exhibit upward comparison.     

Neural responses during anticipation of a primary taste reward

The aim of this study was to determine the brain regions involved in anticipation of a primary taste reward and to compare these regions to those responding to the receipt of a taste reward. Using fMRI, we scanned human subjects who were presented with visual cues that signaled subsequent reinforcement with a pleasant sweet taste (1 M glucose), a moderately unpleasant salt taste (0.2 M saline), or a neutral taste. Expectation of a pleasant taste produced activation in dopaminergic midbrain, posterior dorsal amygdala, striatum, and orbitofrontal cortex (OFC). Apart from OFC, these regions were not activated by reward receipt. The findings indicate that when rewards are predictable, brain regions recruited during expectation are, in part, dissociable from areas responding to reward receipt.     

Dopamine and Serotonin Metabolism in Brain Sites Ipsi- and Contralateral to Direction of Conditioned Turning in Rats

Concentrations of dopamine, serotonin, and some of their metabolites were analyzed by means of HPLC in brain samples obtained from rats operantly conditioned to turn in circles to obtain water reinforcement. In experiment 1 using Wistar rats, no differences in the levels of transmitters or metabolites were detected between brain samples (frontal cortex, ventral striatum, dorsal striatum, septum, amygdala, substantia nigra) from the hemispheres located ipsi- and contralateral to the direction of turning. A higher dopamine metabolism (indicated by higher metabolite/transmitter ratios) in ventral striatum, dorsal striatum, and amygdala was found after 15 min than after 5 min of turning in both hemispheres. A higher dopamine metabolism was found in water-deprived rats compared to nondeprived rats independently of whether or not deprived rats were trained to turn for water reinforcement. In two additional experiments, no differences in dopamine metabolism were found between the ipsi- and contralateral striatum of Wistar rats after 25 min and Sprague-Dawley rats after 10 min of operantly conditioned turning. The present results confirm that dopamine metabolism can change with different behavioral or physiological states; they do not support the hypothesis that conditioned turning is correlated with asymmetrical changes in the metabolism of dopamine or serotonin in the brain.