Prefrontal coding of temporally discounted values during intertemporal choice

Reward from a particular action is seldom immediate, and the influence of such delayed outcome on choice decreases with delay. It has been postulated that when faced with immediate and delayed rewards, decision makers choose the option with maximum temporally discounted value. We examined the preference of monkeys for delayed reward in an intertemporal choice task and the neural basis for real-time computation of temporally discounted values in the dorsolateral prefrontal cortex. During this task, the locations of the targets associated with small or large rewards and their corresponding delays were randomly varied. We found that prefrontal neurons often encoded the temporally discounted value of reward expected from a particular option. Furthermore, activity tended to increase with [corrected] discounted values for targets [corrected] presented in the neuron's preferred direction, suggesting that activity related to temporally discounted values in the prefrontal cortex might determine the animal's behavior during intertemporal choice.     

Effects of multiple delayed rewards on delay discounting in an adjusting amount procedure

It has been suggested that when the delivery of several rewards is separated in time, e.g. one reward immediately and a second reward a few moments later, the value of an alternative that includes these "bundled" rewards will be the sum of the hyperbolic discount functions of the individual rewards. The current study examined this hypothesis using an adjusting amount procedure. In this procedure, rats chose between a delayed food alternative and an immediate food alternative, where the amount of immediate food altered according to each rat's choices. The size of the immediate reward when rats were indifferent between the delayed and immediate alternatives indexed the value of the delayed alternative. Discount functions describing the relationship between the indifference points and the delay to food were created for conditions in which the delay alternative consisted of a single reward (150mul of sucrose solution) delayed by 0, 2, 4, 8, or 16s following the reinforced response. These functions were used to predict the indifference points in other conditions for which an additional 150mul of sucrose solution was delivered at 0, 4, 8, or 16s following the reinforced response. The model fit the data well. However, there were systematic deviations that suggested animals were sensitive to the context within which delays were presented, in addition to the delays themselves. That is, preference for the delayed alternative was lower than predicted when the delay to the additional reward was long (8 or 16s) and higher than the predicted values when it was short (0 or 4s).     

Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value

Risky decision-making is altered in humans and animals with damage to the orbitofrontal cortex. However, the cellular function of the intact orbitofrontal cortex in processing information relevant for risky decisions is unknown. We recorded responses of single orbitofrontal neurons while monkeys viewed visual cues representing the key decision parameters, reward risk and value. Risk was defined as the mathematical variance of binary symmetric probability distributions of reward magnitudes; value was defined as non-risky reward magnitude. Monkeys displayed graded behavioral preferences for risky outcomes, as they did for value. A population of orbitofrontal neurons showed a distinctive risk signal: their cues and reward responses covaried monotonically with the variance of the different reward distributions without monotonically coding reward value. Furthermore, a small but statistically significant fraction of risk responses also coded reward value. These risk signals may provide physiological correlates for the role of the orbitofrontal cortex in risk processing.     

Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology

Neurons in a small number of brain structures detect rewards and reward-predicting stimuli and are active during the expectation of predictable food and liquid rewards. These neurons code the reward information according to basic terms of various behavioural theories that seek to explain reward-directed learning, approach behaviour and decision-making. The involved brain structures include groups of dopamine neurons, the striatum including the nucleus accumbens, the orbitofrontal cortex and the amygdala. The reward information is fed to brain structures involved in decision-making and organisation of behaviour, such as the dorsolateral prefrontal cortex and possibly the parietal cortex. The neural coding of basic reward terms derived from formal theories puts the neurophysiological investigation of reward mechanisms on firm conceptual grounds and provides neural correlates for the function of rewards in learning, approach behaviour and decision-making.     

Neural economics and the biological substrates of valuation

A recent flurry of neuroimaging and decision-making experiments in humans, when combined with single-unit data from orbitofrontal cortex, suggests major additions to current models of reward processing. We review these data and models and use them to develop a specific computational relationship between the value of a predictor and the future rewards or punishments that it promises. The resulting computational model, the predictor-valuation model (PVM), is shown to anticipate a class of single-unit neural responses in orbitofrontal and striatal neurons. The model also suggests how neural responses in the orbitofrontal-striatal circuit may support the conversion of disparate types of future rewards into a kind of internal currency, that is, a common scale used to compare the valuation of future behavioral acts or stimuli.     

Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

BACKGROUND: Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. RESULTS: We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. CONCLUSIONS: The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.     

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.     

Delay of gratification by chimpanzees (Pan troglodytes) in working and waiting situations

We tested four chimpanzees in a self-control task in which food rewards accumulated as long as they were not eaten. In one condition, the chimpanzees had to perform a computer task that directly led to the delivery of the food rewards. In another condition, working on the computerized task was not required and any such work was not linked to the delivery of rewards. The third condition offered no computerized task (chimpanzees simply waited for food rewards to be delivered). Three of four chimpanzees showed no effect of the work scenario on delay of gratification. The one chimpanzee that showed an influence of work scenario on self-control was the overall poorest performing animal. This animal delayed gratification the longest, however, when work was required and reward delivery was directly linked to that work. Therefore, although there is little evidence linking delay of gratification to work requirements in chimpanzees, chimpanzees with lower overall self-control might benefit from having some work available if reward accumulation is contingent on performing that work.     

Saccade reward signals in posterior cingulate cortex

Movement selection depends on the outcome of prior behavior. Posterior cingulate cortex (CGp) is strongly connected with both limbic and oculomotor circuitry, and CGp neurons respond following saccades, suggesting a role in signaling the motivational outcome of gaze shifts. To test this hypothesis, single CGp neurons were studied in monkeys while they shifted gaze to visual targets for liquid rewards that varied in size or were delivered probabilistically. CGp neurons responded following saccades as well as following reward delivery, and these responses were correlated with reward size. CGp neurons also responded following the omission of predicted rewards. The timing of CGp activation and its modulation by reward could provide signals useful for updating representations of expected saccade value.     

Signatures of Value Comparison in Ventral Striatum Neurons

The ventral striatum (VS), like its cortical afferents, is closely associated with processing of rewards, but the relative contributions of striatal and cortical reward systems remains unclear. Most theories posit distinct roles for these structures, despite their similarities. We compared responses of VS neurons to those of ventromedial prefrontal cortex (vmPFC) Area 14 neurons, recorded in a risky choice task. Five major response patterns observed in vmPFC were also observed in VS: (1) offer value encoding, (2) value difference encoding, (3) preferential encoding of chosen relative to unchosen value, (4) a correlation between residual variance in responses and choices, and (5) prominent encoding of outcomes. We did observe some differences as well; in particular, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, the close match between vmPFC and VS suggests that cortex and its striatal targets make overlapping contributions to economic choice.     

Timing in reward and decision processes

Sensitivity to time, including the time of reward, guides the behaviour of all organisms. Recent research suggests that all major reward structures of the brain process the time of reward occurrence, including midbrain dopamine neurons, striatum, frontal cortex and amygdala. Neuronal reward responses in dopamine neurons, striatum and frontal cortex show temporal discounting of reward value. The prediction error signal of dopamine neurons includes the predicted time of rewards. Neurons in the striatum, frontal cortex and amygdala show responses to reward delivery and activities anticipating rewards that are sensitive to the predicted time of reward and the instantaneous reward probability. Together these data suggest that internal timing processes have several well characterized effects on neuronal reward processing.     

What Do I Want and When Do I Want It: Brain Correlates of Decisions Made for Self and Other

A number of recent functional Magnetic Resonance Imaging (fMRI) studies on intertemporal choice behavior have demonstrated that so-called emotion- and reward-related brain areas are preferentially activated by decisions involving immediately available (but smaller) rewards as compared to (larger) delayed rewards. This pattern of activation was not seen, however, when intertemporal choices were made for another (unknown) individual, which speaks to that activation having been triggered by self-relatedness. In the present fMRI study, we investigated the brain correlates of individuals who passively observed intertemporal choices being made either for themselves or for an unknown person. We found higher activation within the ventral striatum, medial prefrontal and orbitofrontal cortex, pregenual anterior cingulate cortex, and posterior cingulate cortex when an immediate reward was possible for the observer herself, which is in line with findings from studies in which individuals actively chose immediately available rewards. Additionally, activation in the dorsal anterior cingulate cortex, posterior cingulate cortex, and precuneus was higher for choices that included immediate options than for choices that offered only delayed options, irrespective of who was to be the beneficiary. These results indicate that (1) the activations found in active intertemporal decision making are also present when the same decisions are merely observed, thus supporting the assumption that a robust brain network is engaged in immediate gratification; and (2) with immediate rewards, certain brain areas are activated irrespective of whether the observer or another person is the beneficiary of a decision, suggesting that immediacy plays a more general role for neural activation. An explorative analysis of participants’ brain activation corresponding to chosen rewards, further indicates that activation in the aforementioned brain areas depends on the mere presence, availability, or actual reception of immediate rewards.     

Brain mechanisms underlying flavour and appetite

Complementary neurophysiological recordings in macaques and functional neuroimaging in humans show that the primary taste cortex in the rostral insula and adjoining frontal operculum provides separate and combined representations of the taste, temperature and texture (including viscosity and fat texture) of food in the mouth independently of hunger and thus of reward value and pleasantness. One synapse on, in the orbitofrontal cortex, these sensory inputs are for some neurons combined by learning with olfactory and visual inputs. Different neurons respond to different combinations, providing a rich representation of the sensory properties of food. In the orbitofrontal cortex, feeding to satiety with one food decreases the responses of these neurons to that food, but not to other foods, showing that sensory-specific satiety is computed in the primate (including human) orbitofrontal cortex. Consistently, activation of parts of the human orbitofrontal cortex correlates with subjective ratings of the pleasantness of the taste and smell of food. Cognitive factors, such as a word label presented with an odour, influence the pleasantness of the odour and the activation produced by the odour in the orbitofrontal cortex. These findings provide a basis for understanding how what is in the mouth is represented by independent information channels in the brain; how the information from these channels is combined; and how and where the reward and subjective affective value of food is represented and is influenced by satiety signals. Activation of these representations in the orbitofrontal cortex may provide the goal for eating, and understanding them helps to provide a basis for understanding appetite and its disorders.     

Functions of the orbitofrontal and pregenual cingulate cortex in taste, olfaction, appetite and emotion

Complementary neurophysiological recordings in macaques and functional neuroimaging in humans show that the primary taste cortex in the rostral insula and adjoining frontal operculum provides separate and combined representations of the taste, temperature, and texture (including viscosity and fat texture) of food in the mouth independently of hunger and thus of reward value and pleasantness. One synapse on, in the orbitofrontal cortex, these sensory inputs are for some neurons combined by learning with olfactory and visual inputs. Different neurons respond to different combinations, providing a rich representation of the sensory properties of food. The representation of taste and other food-related stimuli in the orbitofrontal cortex of macaques is found from its lateral border throughout area 13 to within 7 mm of the midline, and in humans the representation of food-related and other pleasant stimuli is found particularly in the medial orbitofrontal cortex. In the orbitofrontal cortex, feeding to satiety with one food decreases the responses of these neurons to that food, but not to other foods, showing that sensory-specific satiety is computed in the primate (including human) orbitofrontal cortex. Consistently, activation of parts of the human orbitofrontal cortex correlates with subjective ratings of the pleasantness of the taste and smell of food. Cognitive factors, such as a word label presented with an odour, influence the pleasantness of the odour, and the activation produced by the odour in the orbitofrontal cortex. Food intake is thus controlled by building a multimodal representation of the sensory properties of food in the orbitofrontal cortex, and gating this representation by satiety signals to produce a representation of the pleasantness or reward value of food which drives food intake. A neuronal representation of taste is also found in the pregenual cingulate cortex, which receives inputs from the orbitofrontal cortex, and in humans many pleasant stimuli activate the pregenual cingulate cortex, pointing towards this as an important area in motivation and emotion.     

The value of success: acquiring gains, avoiding losses, and simply being successful

A large network of spatially contiguous, yet anatomically distinct regions in medial frontal cortex is involved in reward processing. Although it is clear these regions play a role in critical aspects of reward-related learning and decision-making, the individual contributions of each component remains unclear. We explored dissociations in reward processing throughout several key regions in the reward system and aimed to clarify the nature of previously observed outcome-related activity in a portion of anterior medial orbitofrontal cortex (mOFC). Specifically, we tested whether activity in anterior mOFC was related to processing successful actions, such that this region would respond similarly to rewards with and without tangible benefits, or whether this region instead encoded only quantifiable outcome values (e.g., money). Participants performed a task where they encountered monetary gains and losses (and non-gains and non-losses) during fMRI scanning. Critically, in addition to the outcomes with monetary consequences, the task included trials that provided outcomes without tangible benefits (participants were simply told that they were correct or incorrect). We found that anterior mOFC responded to all successful outcomes regardless of whether they carried tangible benefits (monetary gains and non-losses) or not (controls). These results support the hypothesis that anterior mOFC processes rewards in terms of a common currency and is capable of providing reward-based signals for everything we value, whether it be primary or secondary rewards or simply a successful experience without objectively quantifiable benefits.     

Neural correlates of variations in event processing during learning in central nucleus of amygdala

Attention or variations in event processing help drive learning. Lesion studies have implicated the central nucleus of the amygdala (CeA) in this process, particularly when expected rewards are omitted. However, lesion studies cannot specify how information processing in CeA supports such learning. To address these questions, we recorded CeA neurons in rats performing a task in which rewards were delivered or omitted unexpectedly. We found that activity in CeA neurons increased selectively at the time of omission and declined again with learning. Increased firing correlated with CeA-inactivation sensitive measures of attention. Notably CeA neurons did not fire to the cues or in response to unexpected rewards. These results indicate that CeA contributes to learning in response to reward omission due to a specific role in signaling actual omission rather than a more general involvement in signaling expectancies, errors, or reward value.     

Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice

In choosing between different rewards expected after unequal delays, humans and animals often prefer the smaller but more immediate reward, indicating that the subjective value or utility of reward is depreciated according to its delay. Here, we show that neurons in the primate caudate nucleus and ventral striatum modulate their activity according to temporally discounted values of rewards with a similar time course. However, neurons in the caudate nucleus encoded the difference in the temporally discounted values of the two alternative targets more reliably than neurons in the ventral striatum. In contrast, neurons in the ventral striatum largely encoded the sum of the temporally discounted values, and therefore, the overall goodness of available options. These results suggest a more pivotal role for the dorsal striatum in action selection during intertemporal choice.     

Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Response bias is unaffected by delay length in a delay discounting paradigm

This study examined the contribution of response bias to measures of delay discounting in Long-Evans rats (n = 8) using the adjusting amount procedure. Under this procedure, we assessed preference for 150 μl of 10% sucrose solution delivered following a delay over a variable-amount alternative delivered immediately. Bias was calculated based on relative preference when reinforcers were delivered immediately from both alternatives. We extended this assessment procedure to examine preference when rewards from both alternatives were equally delayed (2, 4, 8, or 16 s) in addition to assessing a traditional delay discounting function. Relative preference was similar across delays and slightly larger than 150 μl. These results indicate that response bias was stable and suggests a relative aversion for the adjusting alternative, which may be due to the variability in reward size associated with that alternative.     

Exponential Versus Hyperbolic Discounting of Delayed Outcomes: Risk and Waiting Time

Frequently, animals must choose between more immediate, smallerrewards and more delayed, but larger rewards. For example, theyoften must decide between accepting a smaller prey item versuscontinuing to search for a larger one, or between entering aleaner patch versus travelling to a richer patch that is furtheraway. In both situations, choice of the more immediate, butsmaller reward may be interpreted as implying that the valueof the later reward is discounted; that is, the value of thelater reward decreases as the delay to its receipt increases.This decrease in value may occur because of the increased riskinvolved in waiting for rewards, or because of the decreasedrate of reward associated with increased waiting time. The presentresearch attempts to determine the form of the relation betweenvalue and delay, and examines implications of this relationfor mechanisms underlying risk-sensitive foraging. Two accounts of the relation between value and delay have beenproposed to describe the decrease in value resulting from increasesin delay: an exponential model and a hyperbolic model. Our researchdemonstrates that, of the two, a hyperbola-like discountingmodel consistently explains more of the variance in temporaldiscounting data at the group level and, importantly, at theindividual level as well. We show mathematically that the hyperbolicmodel shares fundamental features with models of prey and patchchoice. In addition, the present review highlights the implicationsof a psychological perspective for the behavioral biology ofrisksensitive foraging, as well as the implications of an ecologicalperspective for the behavioral psychology of risk-sensitivechoice and decision-making.