Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Prefrontal coding of temporally discounted values during intertemporal choice

Reward from a particular action is seldom immediate, and the influence of such delayed outcome on choice decreases with delay. It has been postulated that when faced with immediate and delayed rewards, decision makers choose the option with maximum temporally discounted value. We examined the preference of monkeys for delayed reward in an intertemporal choice task and the neural basis for real-time computation of temporally discounted values in the dorsolateral prefrontal cortex. During this task, the locations of the targets associated with small or large rewards and their corresponding delays were randomly varied. We found that prefrontal neurons often encoded the temporally discounted value of reward expected from a particular option. Furthermore, activity tended to increase with [corrected] discounted values for targets [corrected] presented in the neuron's preferred direction, suggesting that activity related to temporally discounted values in the prefrontal cortex might determine the animal's behavior during intertemporal choice.     

Effects of multiple delayed rewards on delay discounting in an adjusting amount procedure

It has been suggested that when the delivery of several rewards is separated in time, e.g. one reward immediately and a second reward a few moments later, the value of an alternative that includes these "bundled" rewards will be the sum of the hyperbolic discount functions of the individual rewards. The current study examined this hypothesis using an adjusting amount procedure. In this procedure, rats chose between a delayed food alternative and an immediate food alternative, where the amount of immediate food altered according to each rat's choices. The size of the immediate reward when rats were indifferent between the delayed and immediate alternatives indexed the value of the delayed alternative. Discount functions describing the relationship between the indifference points and the delay to food were created for conditions in which the delay alternative consisted of a single reward (150mul of sucrose solution) delayed by 0, 2, 4, 8, or 16s following the reinforced response. These functions were used to predict the indifference points in other conditions for which an additional 150mul of sucrose solution was delivered at 0, 4, 8, or 16s following the reinforced response. The model fit the data well. However, there were systematic deviations that suggested animals were sensitive to the context within which delays were presented, in addition to the delays themselves. That is, preference for the delayed alternative was lower than predicted when the delay to the additional reward was long (8 or 16s) and higher than the predicted values when it was short (0 or 4s).     

Led into Temptation? Rewarding Brand Logos Bias the Neural Encoding of Incidental Economic Decisions

Human decision-making is driven by subjective values assigned to alternative choice options. These valuations are based on reward cues. It is unknown, however, whether complex reward cues, such as brand logos, may bias the neural encoding of subjective value in unrelated decisions. In this functional magnetic resonance imaging (fMRI) study, we subliminally presented brand logos preceding intertemporal choices. We demonstrated that priming biased participants' preferences towards more immediate rewards in the subsequent temporal discounting task. This was associated with modulations of the neural encoding of subjective values of choice options in a network of brain regions, including but not restricted to medial prefrontal cortex. Our findings demonstrate the general susceptibility of the human decision making system to apparently incidental contextual information. We conclude that the brain incorporates seemingly unrelated value information that modifies decision making outside the decision-maker's awareness.     

Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice

In choosing between different rewards expected after unequal delays, humans and animals often prefer the smaller but more immediate reward, indicating that the subjective value or utility of reward is depreciated according to its delay. Here, we show that neurons in the primate caudate nucleus and ventral striatum modulate their activity according to temporally discounted values of rewards with a similar time course. However, neurons in the caudate nucleus encoded the difference in the temporally discounted values of the two alternative targets more reliably than neurons in the ventral striatum. In contrast, neurons in the ventral striatum largely encoded the sum of the temporally discounted values, and therefore, the overall goodness of available options. These results suggest a more pivotal role for the dorsal striatum in action selection during intertemporal choice.     

Measurement of delay discounting using trial-by-trial consequences

DELAY DISCOUNTING IN HUMANS WAS INVESTIGATED USING THREE DIFFERENT PROCEDURES: a frequently used discounting procedure with hypothetical rewards and delays; a procedure with hypothetical rewards and delays compressed down to much smaller values; and a contingent procedure in which each choice had a direct consequence. In the contingent procedure, on every trial, participants actually experienced the delay and obtained the reward amount associated with their choice. Each participant was exposed to all three procedures. Orderly temporal discounting patterns were obtained in all three procedures and described well by a hyperbolic model. Comparisons of the data revealed patterns unique to each procedure. The distributions of the discounting measures differed across the three procedures. In the contingent procedure, several subjects showed no discounting, e.g. complete self-control. Procedural factors in studies of impulsivity are discussed, and suggestions are offered for experiments in which the contingent-discounting procedure may prove useful.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Neural Sensitivity to Absolute and Relative Anticipated Reward in Adolescents

Adolescence is associated with a dramatic increase in risky and impulsive behaviors that have been attributed to developmental differences in neural processing of rewards. In the present study, we sought to identify age differences in anticipation of absolute and relative rewards. To do so, we modified a commonly used monetary incentive delay (MID) task in order to examine brain activity to relative anticipated reward value (neural sensitivity to the value of a reward as a function of other available rewards). This design also made it possible to examine developmental differences in brain activation to absolute anticipated reward magnitude (the degree to which neural activity increases with increasing reward magnitude). While undergoing fMRI, 18 adolescents and 18 adult participants were presented with cues associated with different reward magnitudes. After the cue, participants responded to a target to win money on that trial. Presentation of cues was blocked such that two reward cues associated with $.20, $1.00, or $5.00 were in play on a given block. Thus, the relative value of the $1.00 reward varied depending on whether it was paired with a smaller or larger reward. Reflecting age differences in neural responses to relative anticipated reward (i.e., reference dependent processing), adults, but not adolescents, demonstrated greater activity to a $1 reward when it was the larger of the two available rewards. Adults also demonstrated a more linear increase in ventral striatal activity as a function of increasing absolute reward magnitude compared to adolescents. Additionally, reduced ventral striatal sensitivity to absolute anticipated reward (i.e., the difference in activity to medium versus small rewards) correlated with higher levels of trait Impulsivity. Thus, ventral striatal activity in anticipation of absolute and relative rewards develops with age. Absolute reward processing is also linked to individual differences in Impulsivity.     

Will travel for food: spatial discounting in two new world monkeys

Nonhuman animals steeply discount the future, showing a preference for small, immediate over large, delayed rewards. Currently unclear is whether discounting functions depend on context. Here, we examine the effects of spatial context on discounting in cotton-top tamarins (Saguinus oedipus) and common marmosets (Callithrix jacchus), species known to differ in temporal discounting. We presented subjects with a choice between small, nearby rewards and large, distant rewards. Tamarins traveled farther for the large reward than marmosets, attending to the ratio of reward differences rather than their absolute values. This species difference contrasts with performance on a temporal task in which marmosets waited longer than tamarins for the large reward. These comparative data indicate that context influences choice behavior, with the strongest effect seen in marmosets who discounted more steeply over space than over time. These findings parallel details of each species' feeding ecology. Tamarins range over large distances and feed primarily on insects, which requires using quick, impulsive action. Marmosets range over shorter distances than tamarins and feed primarily on tree exudates, a clumped resource that requires patience to wait for sap to exude. These results show that discounting functions are context specific, shaped by a history of ecological pressures.     

The role of the rat medial prefrontal cortex in adapting to changes in instrumental contingency

In order to select actions appropriate to current needs, a subject must identify relationships between actions and events. Control over the environment is determined by the degree to which action consequences can be predicted, as described by action-outcome contingencies--i.e. performing an action should affect the probability of the outcome. We evaluated in a first experiment adaptation to contingency changes in rats with neurotoxic lesions of the medial prefrontal cortex. Results indicate that this brain region is not critical to adjust instrumental responding to a negative contingency where the rats must refrain from pressing a lever, as this action prevents reward delivery. By contrast, this brain region is required to reduce responding in a non-contingent situation where the same number of rewards is freely delivered and actions do not affect the outcome any more. In a second experiment, we determined that this effect does not result from a different perception of temporal relationships between actions and outcomes since lesioned rats adapted normally to gradually increasing delays in reward delivery. These data indicate that the medial prefrontal cortex is not directly involved in evaluating the correlation between action--and reward--rates or in the perception of reward delays. The deficit in lesioned rats appears to consist of an abnormal response to the balance between contingent and non-contingent rewards. By highlighting the role of prefrontal regions in adapting to the causal status of actions, these data contribute to our understanding of the neural basis of choice tasks.     

Strain differences in delay discounting using inbred rats

A heightened aversion to delayed rewards is associated with substance abuse and numerous other neuropsychiatric disorders. Many of these disorders are heritable, raising the possibility that delay aversion may also have a significant genetic or heritable component. To examine this possibility, we compared delay discounting in six inbred strains of rats (Brown Norway, Copenhagen, Lewis, Fischer, Noble and Wistar Furth) using the adjusting amount procedure, which provides a measure of the subjective value of delayed rewards. The subjective value of rewards decreased as the delay to receipt increased for all strains. However, a main effect of strain and a strain × delay interaction indicated that some strains were more sensitive to the imposition of delays than others. Fitting a hyperbolic discount equation showed significant strain differences in sensitivity to delay ( k ). These data indicate that there are significant strain differences in delay discounting. All strains strongly preferred the 10% sucrose solution (the reinforcer in the delay discounting task) over water and the amount of sucrose consumed was correlated with sensitivity to delay. Locomotor activity was not correlated with delay discounting behavior. Additional research will be required to disentangle genetic influences from maternal effects and to determine how these factors influence the underlying association between heightened delay discounting and neuropsychiatric disorders.     

The ecology and evolution of patience in two New World monkeys

Decision making often involves choosing between small, short-term rewards and large, long-term rewards. All animals, humans included, discount future rewards--the present value of delayed rewards is viewed as less than the value of immediate rewards. Despite its ubiquity, there exists considerable but unexplained variation between species in their capacity to wait for rewards--that is, to exert patience or self-control. Using two closely related primates--common marmosets (Callithrix jacchus) and cotton-top tamarins (Saguinus oedipus)--we uncover a variable that may explain differences in how species discount future rewards. Both species faced a self-control paradigm in which individuals chose between taking an immediate small reward and waiting a variable amount of time for a large reward. Under these conditions, marmosets waited significantly longer for food than tamarins. This difference cannot be explained by life history, social behaviour or brain size. It can, however, be explained by feeding ecology: marmosets rely on gum, a food product acquired by waiting for exudate to flow from trees, whereas tamarins feed on insects, a food product requiring impulsive action. Foraging ecology, therefore, may provide a selective pressure for the evolution of self-control.     

Delay discounting and gambling

Delay discounting describes the decline in the value of a reinforcer as the delay to that reinforcer increases. A review of the available studies revealed that steep delay discounting is positively correlated with problem or pathological gambling. One hypothesis regarding this correlation derives from the discounting equation proposed by Mazur (1989). According to the equation, steeper discounting renders the difference between fixed-delayed rewards and gambling-like variable-delayed rewards larger; with the latter being more valuable. The present study was designed to test this prediction by first assessing rats’ impulsive choices across four delays to a larger-later reinforcer. A second condition quantified strength of preference for mixed- over fixed-delays, with the duration of the latter adjusted between sessions to achieve indifference. Strength of preference for the mixed-delay alternative is given by the fixed delay at indifference (lower fixed-delay values reflect stronger preferences). Percent impulsive choice was not correlated with the value of the fixed delay at indifference and, therefore, the prediction of the hyperbolic model of gambling was not supported. A follow-up assessment revealed a significant decrease in impulsive choice after the second condition. This shift in impulsive choice could underlie the failure to observe the predicted correlation between impulsive choice and degree of preference for mixed- over fixed delays.     

Humans can adopt optimal discounting strategy under real-time constraints

Critical to our many daily choices between larger delayed rewards, and smaller more immediate rewards, are the shape and the steepness of the function that discounts rewards with time. Although research in artificial intelligence favors exponential discounting in uncertain environments, studies with humans and animals have consistently shown hyperbolic discounting. We investigated how humans perform in a reward decision task with temporal constraints, in which each choice affects the time remaining for later trials, and in which the delays vary at each trial. We demonstrated that most of our subjects adopted exponential discounting in this experiment. Further, we confirmed analytically that exponential discounting, with a decay rate comparable to that used by our subjects, maximized the total reward gain in our task. Our results suggest that the particular shape and steepness of temporal discounting is determined by the task that the subject is facing, and question the notion of hyperbolic reward discounting as a universal principle.     

Response bias is unaffected by delay length in a delay discounting paradigm

This study examined the contribution of response bias to measures of delay discounting in Long-Evans rats (n = 8) using the adjusting amount procedure. Under this procedure, we assessed preference for 150 μl of 10% sucrose solution delivered following a delay over a variable-amount alternative delivered immediately. Bias was calculated based on relative preference when reinforcers were delivered immediately from both alternatives. We extended this assessment procedure to examine preference when rewards from both alternatives were equally delayed (2, 4, 8, or 16 s) in addition to assessing a traditional delay discounting function. Relative preference was similar across delays and slightly larger than 150 μl. These results indicate that response bias was stable and suggests a relative aversion for the adjusting alternative, which may be due to the variability in reward size associated with that alternative.     

Pathological gambling severity is associated with impulsivity in a delay discounting procedure

Research and clinical expertise indicates that impulsivity is an underlying feature of pathological gambling. This study examined the extent to which impulsive behavior, defined by the rate of discounting delayed monetary rewards, varies with pathological gambling severity, assessed by the South Oaks Gambling Screen (SOGS). Sixty-two pathological gamblers completed a delay discounting task, the SOGS, the Eysenck impulsivity scale, the Addiction Severity Index (ASI), and questions about gambling and substance use at intake to outpatient treatment for pathological gambling. In the delay discounting task, participants chose between a large delayed reward (US $1000) and smaller more immediate rewards (US $1-$999) across a range of delays (6h to 25 years). The rate at which the delayed reward was discounted (k value) was derived for each participant and linear regression was used to identify the variables that predicted k values. Age, gender, years of education, substance abuse treatment history, and cigarette smoking history failed to significantly predict k values. Scores on the Eysenck impulsivity scale and the SOGS both accounted for a significant proportion of the variance in k values. The predictive value of the SOGS was 1.4 times that of the Eysenck scale. These results indicate that of the measures tested, gambling severity was the best single predictor of impulsive behavior in a delay discounting task in this sample of pathological gamblers.     

Dopaminergic Lesions of the Dorsolateral Striatum in Rats Increase Delay Discounting in an Impulsive Choice Task

Dysregulated dopamine transmission in striatal circuitry is associated with impulsivity. The current study evaluated the influence of dopaminergic inputs to the dorsolateral striatum on impulsive choice, one aspect of impulsive behavior. We implemented an operant task that measures impulsive choice in rats via delay discounting wherein intracranial self-stimulation (ICSS) was used as the positive reinforcer. To do so, rats were anesthetized to allow implanting of a stimulating electrode within the lateral hypothalamus of one hemisphere and bilateral dorsal striatal injections of the dopaminergic toxin, 6-OHDA (lesioned) or its vehicle (sham). Following recovery, rats were trained in a delay discounting task wherein they selected between a small ICSS current presented immediately after lever pressing, and a large ICSS current presented following a 0 to 15s delay upon pressing the alternate lever. Task acquisition and reinforcer discrimination were similar for lesioned and sham rats. All rats exhibited an initial preference for the large reinforcer, and as the delay was increased, preference for the large reinforcer was decreased indicating that the subjective value of the large reinforcer was discounted as a function of delay time. However, this discounting effect was significantly enhanced in lesioned rats for the longer delays. These data reveal a contribution of dopaminergic inputs to the dorsolateral striatum on impulsive choice behavior, and provide new insights into neural substrates underlying discounting behaviors.     

The subjective value of delayed and probabilistic outcomes: Outcome size matters for gains but not for losses

The subjective value of a reward (gain) is related to factors such as its size, the delay to its receipt and the probability of its receipt. We examined whether the subjective value of losses was similarly affected by these factors in 128 adults. Participants chose between immediate/certain gains or losses and larger delayed/probabilistic gains or losses. Rewards of $100 were devalued as a function of their delay (“discounted”) relatively less than $10 gains while probabilistic $100 rewards were discounted relatively more than $10 rewards. However, there was no effect of outcome size on discounting of delayed or probabilistic losses. For delayed outcomes of each size, the degree to which gains were discounted was positively correlated with the degree to which losses were discounted, whereas for probabilistic outcomes, no such correlation was observed. These results suggest that the processes underlying the subjective valuation of losses are different from those underlying the subjective valuation of gains.     

The evolution of social discounting in hierarchically clustered populations

The expression of a social behaviour may affect the fitness of actors and recipients living in the present and in the future of the population. When there is a risk that a future reward will not be experienced in such a context, the value of that reward should be discounted; but by how much? Here, we evaluate social discount rates for delayed fitness rewards to group of recipients living at different positions in both space and time than the actor in a hierarchically clustered population. This is a population where individuals are grouped into families, families into villages, villages into clans, and so on, possibly ad infinitum. The group-wide fitness effects are assumed to either increase or decrease the fecundity or the survival of recipients and can be arbitrarily extended in space and time. We find that actions changing the survival of individuals living in the future are generally more strongly discounted than fecundity-changing actions for all future times and that the value of future rewards increases as individuals live longer. We also find that delayed fitness effects may not only be discounted by a constant factor per unit delay (exponential discounting), but that, as soon as there is localized dispersal in a population, discounting per unit delay is likely to fall rapidly for small delays and then slowly for longer delays (hyperbolic discounting). As dispersal tends to be localized in natural populations, our results suggest that evolution is likely to favour individuals that express present-biased behaviours and that may be time-inconsistent with respect to their group-wide effects.