Prefrontal coding of temporally discounted values during intertemporal choice

Reward from a particular action is seldom immediate, and the influence of such delayed outcome on choice decreases with delay. It has been postulated that when faced with immediate and delayed rewards, decision makers choose the option with maximum temporally discounted value. We examined the preference of monkeys for delayed reward in an intertemporal choice task and the neural basis for real-time computation of temporally discounted values in the dorsolateral prefrontal cortex. During this task, the locations of the targets associated with small or large rewards and their corresponding delays were randomly varied. We found that prefrontal neurons often encoded the temporally discounted value of reward expected from a particular option. Furthermore, activity tended to increase with [corrected] discounted values for targets [corrected] presented in the neuron's preferred direction, suggesting that activity related to temporally discounted values in the prefrontal cortex might determine the animal's behavior during intertemporal choice.     

Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology

Neurons in a small number of brain structures detect rewards and reward-predicting stimuli and are active during the expectation of predictable food and liquid rewards. These neurons code the reward information according to basic terms of various behavioural theories that seek to explain reward-directed learning, approach behaviour and decision-making. The involved brain structures include groups of dopamine neurons, the striatum including the nucleus accumbens, the orbitofrontal cortex and the amygdala. The reward information is fed to brain structures involved in decision-making and organisation of behaviour, such as the dorsolateral prefrontal cortex and possibly the parietal cortex. The neural coding of basic reward terms derived from formal theories puts the neurophysiological investigation of reward mechanisms on firm conceptual grounds and provides neural correlates for the function of rewards in learning, approach behaviour and decision-making.     

Modulation of caudate activity by action contingency

Research has increasingly implicated the striatum in the processing of reward-related information in both animals and humans. However, it is unclear whether human striatal activation is driven solely by the hedonic properties of rewards or whether such activation is reliant on other factors, such as anticipation of upcoming reward or performance of an action to earn a reward. We used event-related functional magnetic resonance imaging to investigate hemodynamic responses to monetary rewards and punishments in three experiments that made use of an oddball paradigm. We presented reward and punishment displays randomly in time, following an anticipatory cue, or following a button press response. Robust and differential activation of the caudate nucleus occurred only when a perception of contingency existed between the button press response and the outcome. This finding suggests that the caudate is involved in reinforcement of action potentially leading to reward, rather than in processing reward per se.     

Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum

Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the “win-stay, lose-switch” strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.     

Timing in reward and decision processes

Sensitivity to time, including the time of reward, guides the behaviour of all organisms. Recent research suggests that all major reward structures of the brain process the time of reward occurrence, including midbrain dopamine neurons, striatum, frontal cortex and amygdala. Neuronal reward responses in dopamine neurons, striatum and frontal cortex show temporal discounting of reward value. The prediction error signal of dopamine neurons includes the predicted time of rewards. Neurons in the striatum, frontal cortex and amygdala show responses to reward delivery and activities anticipating rewards that are sensitive to the predicted time of reward and the instantaneous reward probability. Together these data suggest that internal timing processes have several well characterized effects on neuronal reward processing.     

Somatodendritic dopamine release: recent mechanistic insights

Dopamine (DA) is a key transmitter in motor, reward and cogitative pathways, with DA dysfunction implicated in disorders including Parkinson''s disease and addiction. Located in midbrain, DA neurons of the substantia nigra pars compacta project via the medial forebrain bundle to the dorsal striatum (caudate putamen), and DA neurons in the adjacent ventral tegmental area project to the ventral striatum (nucleus accumbens) and prefrontal cortex. In addition to classical vesicular release from axons, midbrain DA neurons exhibit DA release from their cell bodies and dendrites. Somatodendritic DA release leads to activation of D2 DA autoreceptors on DA neurons that inhibit their firing via G-protein-coupled inwardly rectifying K⁺ channels. This helps determine patterns of DA signalling at distant axonal release sites. Somatodendritically released DA also acts via volume transmission to extrasynaptic receptors that modulate local transmitter release and neuronal activity in the midbrain. Thus, somatodendritic release is a pivotal intrinsic feature of DA neurons that must be well defined in order to fully understand the physiology and pathophysiology of DA pathways. Here, we review recent mechanistic aspects of somatodendritic DA release, with particular emphasis on the Ca²⁺ dependence of release and the potential role of exocytotic proteins.     

Ventromedial prefrontal cortex activation is associated with memory formation for predictable rewards

During reinforcement learning, dopamine release shifts from the moment of reward consumption to the time point when the reward can be predicted. Previous studies provide consistent evidence that reward-predicting cues enhance long-term memory (LTM) formation of these items via dopaminergic projections to the ventral striatum. However, it is less clear whether memory for items that do not precede a reward but are directly associated with reward consumption is also facilitated. Here, we investigated this question in an fMRI paradigm in which LTM for reward-predicting and neutral cues was compared to LTM for items presented during consumption of reliably predictable as compared to less predictable rewards. We observed activation of the ventral striatum and enhanced memory formation during reward anticipation. During processing of less predictable as compared to reliably predictable rewards, the ventral striatum was activated as well, but items associated with less predictable outcomes were remembered worse than items associated with reliably predictable outcomes. Processing of reliably predictable rewards activated the ventromedial prefrontal cortex (vmPFC), and vmPFC BOLD responses were associated with successful memory formation of these items. Taken together, these findings show that consumption of reliably predictable rewards facilitates LTM formation and is associated with activation of the vmPFC.     

Cholinergic interneuron characteristics and nicotinic properties in the striatum

The neostriatum (dorsal striatum) is composed of the caudate and putamen. The ventral striatum is the ventral conjunction of the caudate and putamen that merges into and includes the nucleus accumbens and striatal portions of the olfactory tubercle. About 2% of the striatal neurons are cholinergic. Most cholinergic neurons in the central nervous system make diffuse projections that sparsely innervate relatively broad areas. In the striatum, however, the cholinergic neurons are interneurons that provide very dense local innervation. The cholinergic interneurons provide an ongoing acetylcholine (ACh) signal by firing action potentials tonically at about 5 Hz. A high concentration of acetylcholinesterase in the striatum rapidly terminates the ACh signal, and thereby minimizes desensitization of nicotinic acetylcholine receptors. Among the many muscarinic and nicotinic striatal mechanisms, the ongoing nicotinic activity potently enhances dopamine release. This process is among those in the striatum that link the two extensive and dense local arbors of the cholinergic interneurons and dopaminergic afferent fibers. During a conditioned motor task, cholinergic interneurons respond with a pause in their tonic firing. It is reasonable to hypothesize that this pause in the cholinergic activity alters action potential dependent dopamine release. The correlated response of these two broad and dense neurotransmitter systems helps to coordinate the output of the striatum, and is likely to be an important process in sensorimotor planning and learning.     

Getting formal with dopamine and reward

Recent neurophysiological studies reveal that neurons in certain brain structures carry specific signals about past and future rewards. Dopamine neurons display a short-latency, phasic reward signal indicating the difference between actual and predicted rewards. The signal is useful for enhancing neuronal processing and learning behavioral reactions. It is distinctly different from dopamine's tonic enabling of numerous behavioral processes. Neurons in the striatum, frontal cortex, and amygdala also process reward information but provide more differentiated information for identifying and anticipating rewards and organizing goal-directed behavior. The different reward signals have complementary functions, and the optimal use of rewards in voluntary behavior would benefit from interactions between the signals. Addictive psychostimulant drugs may exert their action by amplifying the dopamine reward signal.     

CART peptide and the mesolimbic dopamine system

Over the past decade, CART peptide has been commonly associated with the rewarding and reinforcing properties of drugs of abuse and natural rewards such as food. The mesolimbic dopamine system is the predominant pathway involved in mediating reward and reinforcement. Many behavioral and neuroanatomical studies have been conducted in order to further elucidate the importance of CART-containing neurons within the mesolimbic dopamine system. This chapter will review the current knowledge of the localization, synaptic connectivity and neurochemical content of CART peptidecontaining neurons in nuclei of the mesolimbic reward pathway. These nuclei include the nucleus accumbens (NA), ventral midbrain, and the lateral hypothalamus (LH). In conclusion, an interconnected CART-containing loop between the NA, ventral midbrain and LH has evolved from these neuroanatomical studies that may have functional implications for CART peptide's involvement in reward and reinforcement.     

Food reward in the absence of taste receptor signaling

Food palatability and hedonic value play central roles in nutrient intake. However, postingestive effects can influence food preferences independently of palatability, although the neurobiological bases of such mechanisms remain poorly understood. Of central interest is whether the same brain reward circuitry that is responsive to palatable rewards also encodes metabolic value independently of taste signaling. Here we show that trpm5-/- mice, which lack the cellular machinery required for sweet taste transduction, can develop a robust preference for sucrose solutions based solely on caloric content. Sucrose intake induced dopamine release in the ventral striatum of these sweet-blind mice, a pattern usually associated with receipt of palatable rewards. Furthermore, single neurons in this same ventral striatal region showed increased sensitivity to caloric intake even in the absence of gustatory inputs. Our findings suggest that calorie-rich nutrients can directly influence brain reward circuits that control food intake independently of palatability or functional taste transduction.     

The Neural Circuitry of Reward Processing in Complex Social Comparison: Evidence from an Event-Related fMRI Study

In this study, Functional magnetic resonance imaging (fMRI) was conducted to investigate the mechanisms by which the brain activity in a complex social comparison context. One true subject and two pseudo-subjects were asked to complete a simple number estimate task at the same time which including upward and downward comparisons. Two categories of social comparison rewards (fair and unfair rewards distributions) were mainly presented by comparing the true subject with other two pseudo-subjects. Particularly, there were five conditions of unfair distribution when all the three subjects were correct but received different rewards. Behavioral data indicated that the ability to self-regulate was important in satisfaction judgment when the subject perceived an unfair reward distribution. fMRI data indicated that the interaction between the ventral striatum and the prefrontal cortex was important in self-regulation under specific conditions in complex social comparison, especially under condition of reward processing when there were two different reward values and the subject failed to exhibit upward comparison.     

Striatal Expression of Glutamic Acid Decarboxylase Gene in Alzheimer's Disease

Abstract: To examine potential alteration of GABAergic striatal neurons in Alzheimer's disease, we used quantitative in situ hybridization to analyze the messenger RNA coding for M_r 67,000 glutamic acid decarboxylase (GAD₆₇ mRNA) in the striatum of five patients with Alzheimer's disease (AD) and nine matched control subjects. We found a 51–57% increase in the optical density of hybridization signal in the caudate nucleus and putamen, corresponding to a 30–42% increase in the number of neurons expressing a detectable amount of GAD₆₇ mRNA. By contrast, no alteration was observed in the ventral striatum. The expression of GAD₆₇ mRNA per neuron was similar in AD and control subjects both in the dorsal and ventral striatum. Taken together, our data indicate that, in AD, GABAergic neurotransmission is increased in the dorsal striatum but not in the ventral striatum. We suggest that this increased GABAergic neurotransmission may explain extrapyramidal signs often observed in AD.     

Reward-Induced Phasic Dopamine Release in the Monkey Ventral Striatum and Putamen

In-vivo voltammetry has successfully been used to detect dopamine release in rodent brains, but its application to monkeys has been limited. We have previously detected dopamine release in the caudate of behaving Japanese monkeys using diamond microelectrodes (Yoshimi 2011); however it is not known whether the release pattern is the same in various areas of the forebrain. Recent studies have suggested variations in the dopaminergic projections to forebrain areas. In the present study, we attempted simultaneous recording at two locations in the striatum, using fast-scan cyclic voltammetry (FSCV) on carbon fibers, which has been widely used in rodents. Responses to unpredicted food and liquid rewards were detected repeatedly. The response to the liquid reward after conditioned stimuli was enhanced after switching the prediction cue. These characteristics were generally similar between the ventral striatum and the putamen. Overall, the technical application of FSCV recording in multiple locations was successful in behaving primates, and further voltammetric recordings in multiple locations will expand our knowledge of dopamine reward responses.     

Dissociable reward and timing signals in human midbrain and ventral striatum

Reward prediction error (RPE) signals are central to current models of reward-learning. Temporal difference (TD) learning models posit that these signals should be modulated by predictions, not only of magnitude but also timing of reward. Here we show that BOLD activity in the VTA conforms to such TD predictions: responses to unexpected rewards are modulated by a temporal hazard function and activity between a predictive stimulus and reward is depressed in proportion to predicted reward. By contrast, BOLD activity in ventral striatum (VS) does not reflect a TD RPE, but instead encodes a signal on the variable relevant for behavior, here timing but not magnitude of reward. The results have important implications for dopaminergic models of cortico-striatal learning and suggest a modification of the conventional view that VS BOLD necessarily reflects inputs from dopaminergic VTA neurons signaling an RPE.     

Subliminal instrumental conditioning demonstrated in the human brain

How the brain uses success and failure to optimize future decisions is a long-standing question in neuroscience. One computational solution involves updating the values of context-action associations in proportion to a reward prediction error. Previous evidence suggests that such computations are expressed in the striatum and, as they are cognitively impenetrable, represent an unconscious learning mechanism. Here, we formally test this by studying instrumental conditioning in a situation where we masked contextual cues, such that they were not consciously perceived. Behavioral data showed that subjects nonetheless developed a significant propensity to choose cues associated with monetary rewards relative to punishments. Functional neuroimaging revealed that during conditioning cue values and prediction errors, generated from a computational model, both correlated with activity in ventral striatum. We conclude that, even without conscious processing of contextual cues, our brain can learn their reward value and use them to provide a bias on decision making.     

Activation of VTA GABA neurons disrupts reward consumption

The activity of ventral tegmental area (VTA) dopamine (DA) neurons promotes behavioral responses to rewards and environmental stimuli that predict them. VTA GABA inputs synapse directly onto DA neurons and may regulate DA neuronal activity to alter reward-related behaviors; however, the functional consequences of selective activation of VTA GABA neurons remains unknown. Here, we show that in?vivo optogenetic activation of VTA GABA neurons disrupts reward consummatory behavior but not conditioned anticipatory behavior in response to reward-predictive cues. In addition, direct activation of VTA GABA projections to the nucleus accumbens (NAc) resulted in detectable GABA release but did not alter reward consumption. Furthermore, optogenetic stimulation of VTA GABA neurons directly suppressed the activity and excitability of neighboring DA neurons as well as the release of DA in the NAc, suggesting that the dynamic interplay between VTA DA and GABA neurons can control the initiation and termination of reward-related behaviors.     

Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Neural mechanisms underlying motivation of mental versus physical effort

Mental and physical efforts, such as paying attention and lifting weights, have been shown to involve different brain systems. These cognitive and motor systems, respectively, include cortical networks (prefronto-parietal and precentral regions) as well as subregions of the dorsal basal ganglia (caudate and putamen). Both systems appeared sensitive to incentive motivation: their activity increases when we work for higher rewards. Another brain system, including the ventral prefrontal cortex and the ventral basal ganglia, has been implicated in encoding expected rewards. How this motivational system drives the cognitive and motor systems remains poorly understood. More specifically, it is unclear whether cognitive and motor systems can be driven by a common motivational center or if they are driven by distinct, dedicated motivational modules. To address this issue, we used functional MRI to scan healthy participants while performing a task in which incentive motivation, cognitive, and motor demands were varied independently. We reasoned that a common motivational node should (1) represent the reward expected from effort exertion, (2) correlate with the performance attained, and (3) switch effective connectivity between cognitive and motor regions depending on task demand. The ventral striatum fulfilled all three criteria and therefore qualified as a common motivational node capable of driving both cognitive and motor regions of the dorsal striatum. Thus, we suggest that the interaction between a common motivational system and the different task-specific systems underpinning behavioral performance might occur within the basal ganglia.