Reward Anticipation in Ventral Striatum and Individual Sensitivity to Reward: A Pilot Study of a Child-Friendly fMRI Task

Reward processing has been implicated in developmental disorders. However, the classic task to probe reward anticipation, the monetary incentive delay task, has an abstract coding of reward and no storyline and may therefore be less appropriate for use with developmental populations. We modified the task to create a version appropriate for use with children. We investigated whether this child-friendly version could elicit ventral striatal activation during reward anticipation in typically developing children and young adolescents (aged 9.5–14.5). In addition, we tested whether our performance-based measure of reward sensitivity was associated with anticipatory activity in ventral striatum. Reward anticipation was related to activity in bilateral ventral striatum. Moreover, we found an association between individual reward sensitivity and activity in ventral striatum. We conclude that this task assesses ventral striatal activity in a child-friendly paradigm. The combination with a performance-based measure of reward sensitivity potentially makes the task a powerful tool for developmental imaging studies of reward processing.     

Pavlovian reward prediction and receipt in schizophrenia: relationship to anhedonia

Reward processing abnormalities have been implicated in the pathophysiology of negative symptoms such as anhedonia and avolition in schizophrenia. However, studies examining neural responses to reward anticipation and receipt have largely relied on instrumental tasks, which may confound reward processing abnormalities with deficits in response selection and execution. 25 chronic, medicated outpatients with schizophrenia and 20 healthy controls underwent functional magnetic resonance imaging using a pavlovian reward prediction paradigm with no response requirements. Subjects passively viewed cues that predicted subsequent receipt of monetary reward or non-reward, and blood-oxygen-level-dependent signal was measured at the time of cue presentation and receipt. At the group level, neural responses to both reward anticipation and receipt were largely similar between groups. At the time of cue presentation, striatal anticipatory responses did not differ between patients and controls. Right anterior insula demonstrated greater activation for nonreward than reward cues in controls, and for reward than nonreward cues in patients. At the time of receipt, robust responses to receipt of reward vs. nonreward were seen in striatum, midbrain, and frontal cortex in both groups. Furthermore, both groups demonstrated responses to unexpected versus expected outcomes in cortical areas including bilateral dorsolateral prefrontal cortex. Individual difference analyses in patients revealed an association between physical anhedonia and activity in ventral striatum and ventromedial prefrontal cortex during anticipation of reward, in which greater anhedonia severity was associated with reduced activation to money versus no-money cues. In ventromedial prefrontal cortex, this relationship held among both controls and patients, suggesting a relationship between anticipatory activity and anhedonia irrespective of diagnosis. These findings suggest that in the absence of response requirements, brain responses to reward receipt are largely intact in medicated individuals with chronic schizophrenia, while reward anticipation responses in left ventral striatum are reduced in those patients with greater anhedonia severity.     

网络游戏成瘾的奖赏加工缺陷及其神经机制

奖赏加工异常是网络游戏成瘾（internet gaming disorder，IGD）核心特征之一，近期研究结合认知神经科学技术对IGD的神经机制进行探讨发现，IGD与药物成瘾存在相似的神经基础，但仍然存在较多争议。本文从奖赏类型、奖赏加工阶段梳理了IGD人群奖赏加工的研究进展。IGD人群在游戏相关线索下，奖赏预期（reward anticipation）阶段的奖赏系统激活可能与注意偏向、情绪体验和渴求感增加有关。同时，IGD人群对自然奖赏表现出较为一致的低敏感性，该特征主要出现在结果评估（outcome evaluation）阶段。未来研究可以排除共病因素、结合生动的游戏奖赏刺激，进一步探究奖赏预期和结果评估异常如何推动IGD的发展。     

Adolescents,Adults and Rewards: Comparing Motivational Neurocircuitry Recruitment Using fMRI

Background

Adolescent risk-taking, including behaviors resulting in injury or death, has been attributed in part to maturational differences in mesolimbic incentive-motivational neurocircuitry, including ostensible oversensitivity of the nucleus accumbens (NAcc) to rewards.

Methodology/Principal Findings

To test whether adolescents showed increased NAcc activation by cues for rewards, or by delivery of rewards, we scanned 24 adolescents (age 12–17) and 24 adults age (22–42) with functional magnetic resonance imaging while they performed a monetary incentive delay (MID) task. The MID task was configured to temporally disentangle potential reward or potential loss anticipation-related brain signal from reward or loss notification-related signal. Subjects saw cues signaling opportunities to win or avoid losing $0, $.50, or $5 for responding quickly to a subsequent target. Subjects then viewed feedback of their trial success after a variable interval from cue presentation of between 6 to17 s. Adolescents showed reduced NAcc recruitment by reward-predictive cues compared to adult controls in a linear contrast with non-incentive cues, and in a volume-of-interest analysis of signal change in the NAcc. In contrast, adolescents showed little difference in striatal and frontocortical responsiveness to reward deliveries compared to adults.

Conclusions/Significance

In light of divergent developmental difference findings between neuroimaging incentive paradigms (as well as at different stages within the same task), these data suggest that maturational differences in incentive-motivational neurocircuitry: 1) may be sensitive to nuances of incentive tasks or stimuli, such as behavioral or learning contingencies, and 2) may be specific to the component of the instrumental behavior (such as anticipation versus notification).     

Early Effects of Reward Anticipation Are Modulated by Dopaminergic Stimulation

The abilities to predict future rewards and assess the value of reward delivery are crucial aspects of adaptive behavior. While the mesolimbic system, including dopaminergic midbrain, ventral striatum and prefrontal cortex have long been associated with reward processing, recent studies also indicate a prominent role of early visual brain regions. However, the precise underlying neural mechanisms still remain unclear. To address this issue, we presented participants with visual cues predicting rewards of high and low magnitudes and probability (2×2 factorial design), while neural activity was scanned using magnetoencephalography. Importantly, one group of participants received 150 mg of the dopamine precursor levodopa prior to the experiment, while another group received a placebo. For the placebo group, neural signals of reward probability (but not magnitude) emerged at ∼100 ms after cue presentation at occipital sensors in the event-related magnetic fields. Importantly, these probability signals were absent in the levodopa group indicating a close link. Moreover, levodopa administration reduced oscillatory power in the high (20–30 Hz) and low (13–20 Hz) beta band during both reward anticipation and delivery. Taken together, our findings indicate that visual brain regions are involved in coding prospective reward probability but not magnitude and that these effects are modulated by dopamine.     

Ventromedial prefrontal cortex activation is associated with memory formation for predictable rewards

During reinforcement learning, dopamine release shifts from the moment of reward consumption to the time point when the reward can be predicted. Previous studies provide consistent evidence that reward-predicting cues enhance long-term memory (LTM) formation of these items via dopaminergic projections to the ventral striatum. However, it is less clear whether memory for items that do not precede a reward but are directly associated with reward consumption is also facilitated. Here, we investigated this question in an fMRI paradigm in which LTM for reward-predicting and neutral cues was compared to LTM for items presented during consumption of reliably predictable as compared to less predictable rewards. We observed activation of the ventral striatum and enhanced memory formation during reward anticipation. During processing of less predictable as compared to reliably predictable rewards, the ventral striatum was activated as well, but items associated with less predictable outcomes were remembered worse than items associated with reliably predictable outcomes. Processing of reliably predictable rewards activated the ventromedial prefrontal cortex (vmPFC), and vmPFC BOLD responses were associated with successful memory formation of these items. Taken together, these findings show that consumption of reliably predictable rewards facilitates LTM formation and is associated with activation of the vmPFC.     

Subliminal instrumental conditioning demonstrated in the human brain

How the brain uses success and failure to optimize future decisions is a long-standing question in neuroscience. One computational solution involves updating the values of context-action associations in proportion to a reward prediction error. Previous evidence suggests that such computations are expressed in the striatum and, as they are cognitively impenetrable, represent an unconscious learning mechanism. Here, we formally test this by studying instrumental conditioning in a situation where we masked contextual cues, such that they were not consciously perceived. Behavioral data showed that subjects nonetheless developed a significant propensity to choose cues associated with monetary rewards relative to punishments. Functional neuroimaging revealed that during conditioning cue values and prediction errors, generated from a computational model, both correlated with activity in ventral striatum. We conclude that, even without conscious processing of contextual cues, our brain can learn their reward value and use them to provide a bias on decision making.     

Modulation of caudate activity by action contingency

Research has increasingly implicated the striatum in the processing of reward-related information in both animals and humans. However, it is unclear whether human striatal activation is driven solely by the hedonic properties of rewards or whether such activation is reliant on other factors, such as anticipation of upcoming reward or performance of an action to earn a reward. We used event-related functional magnetic resonance imaging to investigate hemodynamic responses to monetary rewards and punishments in three experiments that made use of an oddball paradigm. We presented reward and punishment displays randomly in time, following an anticipatory cue, or following a button press response. Robust and differential activation of the caudate nucleus occurred only when a perception of contingency existed between the button press response and the outcome. This finding suggests that the caudate is involved in reinforcement of action potentially leading to reward, rather than in processing reward per se.     

Reduced sensitivity to immediate reward during decision-making in older than younger adults

We examined whether older adults differ from younger adults in the degree to which they favor immediate over delayed rewards during decision-making. To examine the neural correlates of age-related differences in delay discounting we acquired functional MR images while participants made decisions between smaller but sooner and larger but later monetary rewards. The behavioral results show age-related reductions in delay discounting. Less impulsive decision-making in older adults was associated with lower ventral striatal activations to immediate reward. Furthermore, older adults showed an overall higher percentage of delayed choices and reduced activity in the dorsal striatum than younger adults. This points to a reduced reward sensitivity of the dorsal striatum in older adults. Taken together, our findings indicate that less impulsive decision-making in older adults is due to a reduced sensitivity of striatal areas to reward. These age-related changes in reward sensitivity may result from transformations in dopaminergic neuromodulation with age.     

Uncertainty–guided learning with scaled prediction errors in the basal ganglia

To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.     

Effect of Familiarity on Reward Anticipation in Children with and without Autism Spectrum Disorders

Background

Previous research on the reward system in autism spectrum disorders (ASD) suggests that children with ASD anticipate and process social rewards differently than typically developing (TD) children—but has focused on the reward value of unfamiliar face stimuli. Children with ASD process faces differently than their TD peers. Previous research has focused on face processing of unfamiliar faces, but less is known about how children with ASD process familiar faces. The current study investigated how children with ASD anticipate rewards accompanied by familiar versus unfamiliar faces.

Methods

The stimulus preceding negativity (SPN) of the event-related potential (ERP) was utilized to measure reward anticipation. Participants were 6- to 10-year-olds with (N = 14) and without (N = 14) ASD. Children were presented with rewards accompanied by incidental face or non-face stimuli that were either familiar (caregivers) or unfamiliar. All non-face stimuli were composed of scrambled face elements in the shape of arrows, controlling for visual properties.

Results

No significant differences between familiar versus unfamiliar faces were found for either group. When collapsing across familiarity, TD children showed larger reward anticipation to face versus non-face stimuli, whereas children with ASD did not show differential responses to these stimulus types. Magnitude of reward anticipation to faces was significantly correlated with behavioral measures of social impairment in the ASD group.

Conclusions

The findings do not provide evidence for differential reward anticipation for familiar versus unfamiliar face stimuli in children with or without ASD. These findings replicate previous work suggesting that TD children anticipate rewards accompanied by social stimuli more than rewards accompanied by non-social stimuli. The results do not support the idea that familiarity normalizes reward anticipation in children with ASD. Our findings also suggest that magnitude of reward anticipation to faces is correlated with levels of social impairment for children with ASD.     

Signatures of Value Comparison in Ventral Striatum Neurons

The ventral striatum (VS), like its cortical afferents, is closely associated with processing of rewards, but the relative contributions of striatal and cortical reward systems remains unclear. Most theories posit distinct roles for these structures, despite their similarities. We compared responses of VS neurons to those of ventromedial prefrontal cortex (vmPFC) Area 14 neurons, recorded in a risky choice task. Five major response patterns observed in vmPFC were also observed in VS: (1) offer value encoding, (2) value difference encoding, (3) preferential encoding of chosen relative to unchosen value, (4) a correlation between residual variance in responses and choices, and (5) prominent encoding of outcomes. We did observe some differences as well; in particular, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, the close match between vmPFC and VS suggests that cortex and its striatal targets make overlapping contributions to economic choice.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Reward-related dorsal striatal activity differences between former and current cocaine dependent individuals during an interactive competitive game

Cocaine addiction is characterized by impulsivity, impaired social relationships, and abnormal mesocorticolimbic reward processing, but their interrelationships relative to stages of cocaine addiction are unclear. We assessed blood-oxygenation-level dependent (BOLD) signal in ventral and dorsal striatum during functional magnetic resonance imaging (fMRI) in current (CCD; n?=?30) and former (FCD; n?=?28) cocaine dependent subjects as well as healthy control (HC; n?=?31) subjects while playing an interactive competitive Domino game involving risk-taking and reward/punishment processing. Out-of-scanner impulsivity-related measures were also collected. Although both FCD and CCD subjects scored significantly higher on impulsivity-related measures than did HC subjects, only FCD subjects had differences in striatal activation, specifically showing hypoactivation during their response to gains versus losses in right dorsal caudate, a brain region linked to habituation, cocaine craving and addiction maintenance. Right caudate activity in FCD subjects also correlated negatively with impulsivity-related measures of self-reported compulsivity and sensitivity to reward. These findings suggest that remitted cocaine dependence is associated with striatal dysfunction during social reward processing in a manner linked to compulsivity and reward sensitivity measures. Future research should investigate the extent to which such differences might reflect underlying vulnerabilities linked to cocaine-using propensities (e.g., relapses).     

The Conforming Brain and Deontological Resolve

Our personal values are subject to forces of social influence. Deontological resolve captures how strongly one relies on absolute rules of right and wrong in the representation of one''s personal values and may predict willingness to modify one''s values in the presence of social influence. Using fMRI, we found that a neurobiological metric for deontological resolve based on relative activity in the ventrolateral prefrontal cortex (VLPFC) during the passive processing of sacred values predicted individual differences in conformity. Individuals with stronger deontological resolve, as measured by greater VLPFC activity, displayed lower levels of conformity. We also tested whether responsiveness to social reward, as measured by ventral striatal activity during social feedback, predicted variability in conformist behavior across individuals but found no significant relationship. From these results we conclude that unwillingness to conform to others'' values is associated with a strong neurobiological representation of social rules.     

Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

BACKGROUND: Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. RESULTS: We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. CONCLUSIONS: The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.     

Amphetamine Sensitization Alters Reward Processing in the Human Striatum and Amygdala

Dysregulation of mesolimbic dopamine transmission is implicated in a number of psychiatric illnesses characterised by disruption of reward processing and goal-directed behaviour, including schizophrenia, drug addiction and impulse control disorders associated with chronic use of dopamine agonists. Amphetamine sensitization (AS) has been proposed to model the development of this aberrant dopamine signalling and the subsequent dysregulation of incentive motivational processes. However, in humans the effects of AS on the dopamine-sensitive neural circuitry associated with reward processing remains unclear. Here we describe the effects of acute amphetamine administration, following a sensitising dosage regime, on blood oxygen level dependent (BOLD) signal in dopaminoceptive brain regions during a rewarded gambling task performed by healthy volunteers. Using a randomised, double-blind, parallel-groups design, we found clear evidence for sensitization to the subjective effects of the drug, while rewarded reaction times were unchanged. Repeated amphetamine exposure was associated with reduced dorsal striatal BOLD signal during decision making, but enhanced ventromedial caudate activity during reward anticipation. The amygdala BOLD response to reward outcomes was blunted following repeated amphetamine exposure. Positive correlations between subjective sensitization and changes in anticipation- and outcome-related BOLD signal were seen for the caudate nucleus and amygdala, respectively. These data show for the first time in humans that AS changes the functional impact of acute stimulant exposure on the processing of reward-related information within dopaminoceptive regions. Our findings accord with pathophysiological models which implicate aberrant dopaminergic modulation of striatal and amygdala activity in psychosis and drug-related compulsive disorders.     

Prodromal Huntington Disease as a Model for Functional Compensation of Early Neurodegeneration

Functional compensation demonstrated as mechanism to offset neuronal loss in early Alzheimer disease may also occur in other adult-onset neurodegenerative diseases, particularly Huntington disease (HD) with its genetic determination and gradual changes in structural integrity. In HD, neurodegeneration typically initiates in the dorsal striatum, successively affecting ventral striatal areas. Investigating carriers of the HD mutation with evident dorsal, but only minimal or no ventral striatal atrophy, we expected to find evidence for compensation of ventral striatal functioning. We investigated 14 pre- or early symptomatic carriers of the mutation leading to HD and 18 matched healthy controls. Participants underwent structural T1 magnetic resonance imaging (MRI) and functional MRI during a reward task that probes ventral striatal functioning. Motor functioning and attention were assessed with reaction time (RT) tasks. Structural images confirmed a specific decrease of dorsal striatal but only marginal ventral striatal volume in HD relative to control subjects, paralleling prolonged RT in the motor response tasks. While behavioral performance in the reward task during fMRI scanning was unimpaired, reward-related fMRI signaling in the HD group was differentially enhanced in the bilateral ventral striatum and in bilateral orbitofrontal cortex/anterior insula, as another region sensitive to reward processing. We provide evidence for the concept of functional compensation in premanifest HD which may suggest a defense mechanism in neurodegeneration. Given the so far inevitable course of HD with its genetically determined endpoint, this disease may provide another model to study the different aspects of the concept of functional compensation.     

Relief as a reward: hedonic and neural responses to safety from pain

Relief fits the definition of a reward. Unlike other reward types the pleasantness of relief depends on the violation of a negative expectation, yet this has not been investigated using neuroimaging approaches. We hypothesized that the degree of negative expectation depends on state (dread) and trait (pessimism) sensitivity. Of the brain regions that are involved in mediating pleasure, the nucleus accumbens also signals unexpected reward and positive prediction error. We hypothesized that accumbens activity reflects the level of negative expectation and subsequent pleasant relief. Using fMRI and two purpose-made tasks, we compared hedonic and BOLD responses to relief with responses during an appetitive reward task in 18 healthy volunteers. We expected some similarities in task responses, reflecting common neural substrates implicated across reward types. However, we also hypothesized that relief responses would differ from appetitive rewards in the nucleus accumbens, since only relief pleasantness depends on negative expectations. The results confirmed these hypotheses. Relief and appetitive reward task activity converged in the ventromedial prefrontal cortex, which also correlated with appetitive reward pleasantness ratings. In contrast, dread and pessimism scores correlated with relief but not with appetitive reward hedonics. Moreover, only relief pleasantness covaried with accumbens activation. Importantly, the accumbens signal appeared to specifically reflect individual differences in anticipation of the adverse event (dread, pessimism) but was uncorrelated to appetitive reward hedonics. In conclusion, relief differs from appetitive rewards due to its reliance on negative expectations, the violation of which is reflected in relief-related accumbens activation.     

Neural economics and the biological substrates of valuation

A recent flurry of neuroimaging and decision-making experiments in humans, when combined with single-unit data from orbitofrontal cortex, suggests major additions to current models of reward processing. We review these data and models and use them to develop a specific computational relationship between the value of a predictor and the future rewards or punishments that it promises. The resulting computational model, the predictor-valuation model (PVM), is shown to anticipate a class of single-unit neural responses in orbitofrontal and striatal neurons. The model also suggests how neural responses in the orbitofrontal-striatal circuit may support the conversion of disparate types of future rewards into a kind of internal currency, that is, a common scale used to compare the valuation of future behavioral acts or stimuli.