Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice

In choosing between different rewards expected after unequal delays, humans and animals often prefer the smaller but more immediate reward, indicating that the subjective value or utility of reward is depreciated according to its delay. Here, we show that neurons in the primate caudate nucleus and ventral striatum modulate their activity according to temporally discounted values of rewards with a similar time course. However, neurons in the caudate nucleus encoded the difference in the temporally discounted values of the two alternative targets more reliably than neurons in the ventral striatum. In contrast, neurons in the ventral striatum largely encoded the sum of the temporally discounted values, and therefore, the overall goodness of available options. These results suggest a more pivotal role for the dorsal striatum in action selection during intertemporal choice.     

Led into Temptation? Rewarding Brand Logos Bias the Neural Encoding of Incidental Economic Decisions

Human decision-making is driven by subjective values assigned to alternative choice options. These valuations are based on reward cues. It is unknown, however, whether complex reward cues, such as brand logos, may bias the neural encoding of subjective value in unrelated decisions. In this functional magnetic resonance imaging (fMRI) study, we subliminally presented brand logos preceding intertemporal choices. We demonstrated that priming biased participants' preferences towards more immediate rewards in the subsequent temporal discounting task. This was associated with modulations of the neural encoding of subjective values of choice options in a network of brain regions, including but not restricted to medial prefrontal cortex. Our findings demonstrate the general susceptibility of the human decision making system to apparently incidental contextual information. We conclude that the brain incorporates seemingly unrelated value information that modifies decision making outside the decision-maker's awareness.     

What Do I Want and When Do I Want It: Brain Correlates of Decisions Made for Self and Other

A number of recent functional Magnetic Resonance Imaging (fMRI) studies on intertemporal choice behavior have demonstrated that so-called emotion- and reward-related brain areas are preferentially activated by decisions involving immediately available (but smaller) rewards as compared to (larger) delayed rewards. This pattern of activation was not seen, however, when intertemporal choices were made for another (unknown) individual, which speaks to that activation having been triggered by self-relatedness. In the present fMRI study, we investigated the brain correlates of individuals who passively observed intertemporal choices being made either for themselves or for an unknown person. We found higher activation within the ventral striatum, medial prefrontal and orbitofrontal cortex, pregenual anterior cingulate cortex, and posterior cingulate cortex when an immediate reward was possible for the observer herself, which is in line with findings from studies in which individuals actively chose immediately available rewards. Additionally, activation in the dorsal anterior cingulate cortex, posterior cingulate cortex, and precuneus was higher for choices that included immediate options than for choices that offered only delayed options, irrespective of who was to be the beneficiary. These results indicate that (1) the activations found in active intertemporal decision making are also present when the same decisions are merely observed, thus supporting the assumption that a robust brain network is engaged in immediate gratification; and (2) with immediate rewards, certain brain areas are activated irrespective of whether the observer or another person is the beneficiary of a decision, suggesting that immediacy plays a more general role for neural activation. An explorative analysis of participants’ brain activation corresponding to chosen rewards, further indicates that activation in the aforementioned brain areas depends on the mere presence, availability, or actual reception of immediate rewards.     

Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Signatures of Value Comparison in Ventral Striatum Neurons

The ventral striatum (VS), like its cortical afferents, is closely associated with processing of rewards, but the relative contributions of striatal and cortical reward systems remains unclear. Most theories posit distinct roles for these structures, despite their similarities. We compared responses of VS neurons to those of ventromedial prefrontal cortex (vmPFC) Area 14 neurons, recorded in a risky choice task. Five major response patterns observed in vmPFC were also observed in VS: (1) offer value encoding, (2) value difference encoding, (3) preferential encoding of chosen relative to unchosen value, (4) a correlation between residual variance in responses and choices, and (5) prominent encoding of outcomes. We did observe some differences as well; in particular, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, the close match between vmPFC and VS suggests that cortex and its striatal targets make overlapping contributions to economic choice.     

Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

BACKGROUND: Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. RESULTS: We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. CONCLUSIONS: The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.     

Will travel for food: spatial discounting in two new world monkeys

Nonhuman animals steeply discount the future, showing a preference for small, immediate over large, delayed rewards. Currently unclear is whether discounting functions depend on context. Here, we examine the effects of spatial context on discounting in cotton-top tamarins (Saguinus oedipus) and common marmosets (Callithrix jacchus), species known to differ in temporal discounting. We presented subjects with a choice between small, nearby rewards and large, distant rewards. Tamarins traveled farther for the large reward than marmosets, attending to the ratio of reward differences rather than their absolute values. This species difference contrasts with performance on a temporal task in which marmosets waited longer than tamarins for the large reward. These comparative data indicate that context influences choice behavior, with the strongest effect seen in marmosets who discounted more steeply over space than over time. These findings parallel details of each species' feeding ecology. Tamarins range over large distances and feed primarily on insects, which requires using quick, impulsive action. Marmosets range over shorter distances than tamarins and feed primarily on tree exudates, a clumped resource that requires patience to wait for sap to exude. These results show that discounting functions are context specific, shaped by a history of ecological pressures.     

Cross-country relationships between life expectancy,intertemporal choice and age at first birth

Humans, like other animals, typically discount the value of delayed rewards relative to those available in the present. From an evolutionary perspective, prioritising immediate rewards is a predictable response to high local mortality rates, as is an acceleration of reproductive scheduling. In a sample of 46 countries, we explored the cross-country relationships between average life expectancy, intertemporal choice, and women's age at first birth. We find that, across countries, lower life expectancy is associated with both a smaller percentage of people willing to wait for a larger but delayed reward, as well as a younger age at first birth. These results, which hold when controlling for region and economic pressure (GDP-per capita), dovetail with findings at the individual level to suggest that life expectancy is an important ecological predictor of both intertemporal and reproductive decision-making.     

Pathological gambling severity is associated with impulsivity in a delay discounting procedure

Research and clinical expertise indicates that impulsivity is an underlying feature of pathological gambling. This study examined the extent to which impulsive behavior, defined by the rate of discounting delayed monetary rewards, varies with pathological gambling severity, assessed by the South Oaks Gambling Screen (SOGS). Sixty-two pathological gamblers completed a delay discounting task, the SOGS, the Eysenck impulsivity scale, the Addiction Severity Index (ASI), and questions about gambling and substance use at intake to outpatient treatment for pathological gambling. In the delay discounting task, participants chose between a large delayed reward (US $1000) and smaller more immediate rewards (US $1-$999) across a range of delays (6h to 25 years). The rate at which the delayed reward was discounted (k value) was derived for each participant and linear regression was used to identify the variables that predicted k values. Age, gender, years of education, substance abuse treatment history, and cigarette smoking history failed to significantly predict k values. Scores on the Eysenck impulsivity scale and the SOGS both accounted for a significant proportion of the variance in k values. The predictive value of the SOGS was 1.4 times that of the Eysenck scale. These results indicate that of the measures tested, gambling severity was the best single predictor of impulsive behavior in a delay discounting task in this sample of pathological gamblers.     

Humans can adopt optimal discounting strategy under real-time constraints

Critical to our many daily choices between larger delayed rewards, and smaller more immediate rewards, are the shape and the steepness of the function that discounts rewards with time. Although research in artificial intelligence favors exponential discounting in uncertain environments, studies with humans and animals have consistently shown hyperbolic discounting. We investigated how humans perform in a reward decision task with temporal constraints, in which each choice affects the time remaining for later trials, and in which the delays vary at each trial. We demonstrated that most of our subjects adopted exponential discounting in this experiment. Further, we confirmed analytically that exponential discounting, with a decay rate comparable to that used by our subjects, maximized the total reward gain in our task. Our results suggest that the particular shape and steepness of temporal discounting is determined by the task that the subject is facing, and question the notion of hyperbolic reward discounting as a universal principle.     

Get It While It’s Hot: A Peak-First Bias in Self-Generated Choice Order in Rhesus Macaques

Animals typically must make a number of successive choices to achieve a goal: e.g., eating multiple food items before becoming satiated. However, it is unclear whether choosing the best first or saving the best for last represents the best choice strategy to maximize overall reward. Specifically, since outcomes can be evaluated prospectively (with future rewards discounted and more immediate rewards preferred) or retrospectively (with prior rewards discounted and more recent rewards preferred), the conditions under which each are used remains unclear. On the one hand, humans and non-human animals clearly discount future reward, preferring immediate rewards to delayed ones, suggesting prospective evaluation; on the other hand, it has also been shown that a sequence that ends well, i.e., with the best event or item last, is often preferred, suggesting retrospective evaluation. Here we hypothesized that when individuals are allowed to build the sequence themselves they are more likely to evaluate each item individually and therefore build a sequence using prospective evaluation. We examined the relationship between self-generated choice order and preference in rhesus monkeys in two experiments in which the distinctiveness of options were relatively high and low, respectively. We observed a positive linear relationship between choice order and preference among highly distinct options, indicating that the rhesus monkeys chose their preferred food first: i.e., a peak-first order preference. Overall, choice order depended on the degree of relative preference among alternatives and a peak-first bias, providing evidence for prospective evaluation when choice order is self-generated.     

The subjective value of delayed and probabilistic outcomes: Outcome size matters for gains but not for losses

The subjective value of a reward (gain) is related to factors such as its size, the delay to its receipt and the probability of its receipt. We examined whether the subjective value of losses was similarly affected by these factors in 128 adults. Participants chose between immediate/certain gains or losses and larger delayed/probabilistic gains or losses. Rewards of $100 were devalued as a function of their delay (“discounted”) relatively less than $10 gains while probabilistic $100 rewards were discounted relatively more than $10 rewards. However, there was no effect of outcome size on discounting of delayed or probabilistic losses. For delayed outcomes of each size, the degree to which gains were discounted was positively correlated with the degree to which losses were discounted, whereas for probabilistic outcomes, no such correlation was observed. These results suggest that the processes underlying the subjective valuation of losses are different from those underlying the subjective valuation of gains.     

Neural differentiation of expected reward and risk in human subcortical structures

In decision-making under uncertainty, economic studies emphasize the importance of risk in addition to expected reward. Studies in neuroscience focus on expected reward and learning rather than risk. We combined functional imaging with a simple gambling task to vary expected reward and risk simultaneously and in an uncorrelated manner. Drawing on financial decision theory, we modeled expected reward as mathematical expectation of reward, and risk as reward variance. Activations in dopaminoceptive structures correlated with both mathematical parameters. These activations differentiated spatially and temporally. Temporally, the activation related to expected reward was immediate, while the activation related to risk was delayed. Analyses confirmed that our paradigm minimized confounds from learning, motivation, and salience. These results suggest that the primary task of the dopaminergic system is to convey signals of upcoming stochastic rewards, such as expected reward and risk, beyond its role in learning, motivation, and salience.     

Exponential Versus Hyperbolic Discounting of Delayed Outcomes: Risk and Waiting Time

Frequently, animals must choose between more immediate, smallerrewards and more delayed, but larger rewards. For example, theyoften must decide between accepting a smaller prey item versuscontinuing to search for a larger one, or between entering aleaner patch versus travelling to a richer patch that is furtheraway. In both situations, choice of the more immediate, butsmaller reward may be interpreted as implying that the valueof the later reward is discounted; that is, the value of thelater reward decreases as the delay to its receipt increases.This decrease in value may occur because of the increased riskinvolved in waiting for rewards, or because of the decreasedrate of reward associated with increased waiting time. The presentresearch attempts to determine the form of the relation betweenvalue and delay, and examines implications of this relationfor mechanisms underlying risk-sensitive foraging. Two accounts of the relation between value and delay have beenproposed to describe the decrease in value resulting from increasesin delay: an exponential model and a hyperbolic model. Our researchdemonstrates that, of the two, a hyperbola-like discountingmodel consistently explains more of the variance in temporaldiscounting data at the group level and, importantly, at theindividual level as well. We show mathematically that the hyperbolicmodel shares fundamental features with models of prey and patchchoice. In addition, the present review highlights the implicationsof a psychological perspective for the behavioral biology ofrisksensitive foraging, as well as the implications of an ecologicalperspective for the behavioral psychology of risk-sensitivechoice and decision-making.     

The Neural Circuitry of Reward Processing in Complex Social Comparison: Evidence from an Event-Related fMRI Study

In this study, Functional magnetic resonance imaging (fMRI) was conducted to investigate the mechanisms by which the brain activity in a complex social comparison context. One true subject and two pseudo-subjects were asked to complete a simple number estimate task at the same time which including upward and downward comparisons. Two categories of social comparison rewards (fair and unfair rewards distributions) were mainly presented by comparing the true subject with other two pseudo-subjects. Particularly, there were five conditions of unfair distribution when all the three subjects were correct but received different rewards. Behavioral data indicated that the ability to self-regulate was important in satisfaction judgment when the subject perceived an unfair reward distribution. fMRI data indicated that the interaction between the ventral striatum and the prefrontal cortex was important in self-regulation under specific conditions in complex social comparison, especially under condition of reward processing when there were two different reward values and the subject failed to exhibit upward comparison.     

Monkeys Are More Patient in a Foraging Task than in a Standard Intertemporal Choice Task

Studies of animal impulsivity generally find steep subjective devaluation, or discounting, of delayed rewards – often on the order of a 50% reduction in value in a few seconds. Because such steep discounting is highly disfavored in evolutionary models of time preference, we hypothesize that discounting tasks provide a poor measure of animals’ true time preferences. One prediction of this hypothesis is that estimates of time preferences based on these tasks will lack external validity, i.e. fail to predict time preferences in other contexts. We examined choices made by four rhesus monkeys in a computerized patch-leaving foraging task interleaved with a standard intertemporal choice task. Monkeys were significantly more patient in the foraging task than in the intertemporal choice task. Patch-leaving behavior was well fit by parameter-free optimal foraging equations but poorly fit by the hyperbolic discount parameter obtained from the intertemporal choice task. Day-to-day variation in time preferences across the two tasks was uncorrelated with each other. These data are consistent with the conjecture that seemingly impulsive behavior in animals is an artifact of their difficulty understanding the structure of intertemporal choice tasks, and support the idea that animals are more efficient rate maximizers in the multi-second range than intertemporal choice tasks would suggest.     

Effects of multiple delayed rewards on delay discounting in an adjusting amount procedure

It has been suggested that when the delivery of several rewards is separated in time, e.g. one reward immediately and a second reward a few moments later, the value of an alternative that includes these "bundled" rewards will be the sum of the hyperbolic discount functions of the individual rewards. The current study examined this hypothesis using an adjusting amount procedure. In this procedure, rats chose between a delayed food alternative and an immediate food alternative, where the amount of immediate food altered according to each rat's choices. The size of the immediate reward when rats were indifferent between the delayed and immediate alternatives indexed the value of the delayed alternative. Discount functions describing the relationship between the indifference points and the delay to food were created for conditions in which the delay alternative consisted of a single reward (150mul of sucrose solution) delayed by 0, 2, 4, 8, or 16s following the reinforced response. These functions were used to predict the indifference points in other conditions for which an additional 150mul of sucrose solution was delivered at 0, 4, 8, or 16s following the reinforced response. The model fit the data well. However, there were systematic deviations that suggested animals were sensitive to the context within which delays were presented, in addition to the delays themselves. That is, preference for the delayed alternative was lower than predicted when the delay to the additional reward was long (8 or 16s) and higher than the predicted values when it was short (0 or 4s).     

The Value of Emotion: How Does Episodic Prospection Modulate Delay Discounting?

Background

Humans often show impatience when making intertemporal choice for monetary rewards, preferring small rewards delivered immediately to larger rewards delivered after a delay, which reflects a fundamental psychological principle: delay discounting. However, we propose that episodic prospection humans can vividly envisage exerts a strong and broad influence on individuals'' delay discounting. Specifically, episodic prospection may affect individuals'' intertemporal choice by the negative or positive emotion of prospection.

Methodology/Principal Findings

The present study explored how episodic prospection modulated delay discounting by emotion. Study 1 showed that participants were more inclined to choose the delayed but larger rewards when they imaged positive future events than when they did not image events; Study 2 showed that participants were more inclined to choose the immediate but smaller rewards when they imaged negative future events than when they did not image events; In contrast, study 3 showed that choice preferences of participants when they imaged neutral future events were the same as when they did not image events.

Conclusions/Significance

By manipulating the emotion valence of episodic prospection, our findings suggested that positive emotion made individuals tend to choose delayed rewards, while negative emotion made individuals tend to choose immediate rewards. Only imaging events with neutral emotion did not affect individuals'' choice preference. Thus, the valence of imaged future events'' emotion might play an important role in individuals'' intertemporal choice. It is possible that the valence of emotion may affect the changed direction (promote or inhibit) of individuals'' delay discounting, while the ability to image future events affects the changed degree of individuals'' delay discounting.     

The present values of delayed rewards are approximately additive

In two experiments, human subjects were asked to estimate their present values of single delayed rewards and their present values of temporal sequences of three rewards. Present values were solicited by asking subjects to indicate an amount of money v for which they would be indifferent between receiving v at the end of the session and receiving the delayed reward(s). A procedure was used for which responding the true value of v was the optimal strategy, and the actual payoff that each subject received was determined by one randomly selected trial. In Experiment 1 (n=29) each delayed reward was 9.90 dollars in cash. In Experiment 2 (n=19) the delayed rewards were dated 15 dollars gift certificates to a local restaurant. In both experiments, the present values of the sequences were approximately equal to the sums of the present values of their component rewards. The presence of outliers suggests that a few subjects may have valued sequences less than the sums of their single rewards. Effects of a preference for uniform sequences, if any, were too small to be detected. Discounting of sequences was well fit by a parallel hyperbolic discounting equation, consistent with Mazur's [Mazur, J.E., 1986. Choice between single and multiple delayed reinforcers. J. Exp. Anal. Behav. 46 (1), 67-77] results using multiple reinforcers.