Impulsiveness without discounting: the ecological rationality hypothesis

Observed animal impulsiveness challenges ideas from foraging theory about the fitness value of food rewards, and may play a role in important behavioural phenomena such as cooperation and addiction. Behavioural ecologists usually invoke temporal discounting to explain the evolution of animal impulsiveness. According to the discounting hypothesis, delay reduces the fitness value of the delayed food. We develop an alternative model for the evolution of impulsiveness that does not require discounting. We show that impulsive or short-sighted rules can maximize long-term rates of food intake. The advantages of impulsive rules come from two sources. First, naturally occurring choices have a foreground-background structure that reduces the long-term cost of impulsiveness. Second, impulsive rules have a discrimination advantage because they tend to compare smaller quantities. Discounting contributes little to this result. Although we find that impulsive rules are optimal in a simple foreground-background choice situation in the absence of discounting, in contrast we do not find comparable impulsiveness in binary choice situations even when there is strong discounting.     

Insulin,Central Dopamine D2 Receptors,and Monetary Reward Discounting in Obesity

Animal research finds that insulin regulates dopamine signaling and reward behavior, but similar research in humans is lacking. We investigated whether individual differences in body mass index, percent body fat, pancreatic β-cell function, and dopamine D2 receptor binding were related to reward discounting in obese and non-obese adult men and women. Obese (n = 27; body mass index>30) and non-obese (n = 20; body mass index<30) adults were assessed for percent body fat with dual-energy X-ray absorptiometry and for β-cell function using disposition index. Choice of larger, but delayed or less certain, monetary rewards relative to immediate, certain smaller monetary rewards was measured using delayed and probabilistic reward discounting tasks. Positron emission tomography using a non-displaceable D2-specific radioligand, [¹¹C](N-methyl)benperidol quantified striatal D2 receptor binding. Groups differed in body mass index, percent body fat, and disposition index, but not in striatal D2 receptor specific binding or reward discounting. Higher percent body fat in non-obese women related to preference for a smaller, certain reward over a larger, less likely one (greater probabilistic discounting). Lower β-cell function in the total sample and lower insulin sensitivity in obese related to stronger preference for an immediate and smaller monetary reward over delayed receipt of a larger one (greater delay discounting). In obese adults, higher striatal D2 receptor binding related to greater delay discounting. Interestingly, striatal D2 receptor binding was not significantly related to body mass index, percent body fat, or β-cell function in either group. Our findings indicate that individual differences in percent body fat, β-cell function, and striatal D2 receptor binding may each contribute to altered reward discounting behavior in non-obese and obese individuals. These results raise interesting questions about whether and how striatal D2 receptor binding and metabolic factors, including β-cell function, interact to affect reward discounting in humans.     

Foraging juncos: interaction of reward mean and variability

Recent stochastic theories of feeding strategy propose that a risk-averse forager will trade off increases in reward variability against increases in average reward. We show that two models of risk-aversion (variance discounting and the z-score model) imply different forms for this interaction of mean and variability. Variance discounting suggests constant risk-aversion, since the effect of reward variance on expected fitness is assumed to be independent of the mean. The z-score model suggests decreasing risk-aversion, since the effect of a given level of reward variance on expected fitness is assumed to decrease as the mean increases. We report two series of experiments with dark-eyed juncos (Junco hyemalis). Each series investigated the mean-variability trade-off under physiological conditions promoting risk-aversion; reward sizes were generally greater in the second series. The results clearly indicate a trade-off, but within a series the data do not discriminate between the two models' predictions. Comparing the two series suggests that the juncos' level of aversion to reward variability decreases after an overall increment in environmental profitability.     

The evolution of social discounting in hierarchically clustered populations

The expression of a social behaviour may affect the fitness of actors and recipients living in the present and in the future of the population. When there is a risk that a future reward will not be experienced in such a context, the value of that reward should be discounted; but by how much? Here, we evaluate social discount rates for delayed fitness rewards to group of recipients living at different positions in both space and time than the actor in a hierarchically clustered population. This is a population where individuals are grouped into families, families into villages, villages into clans, and so on, possibly ad infinitum. The group-wide fitness effects are assumed to either increase or decrease the fecundity or the survival of recipients and can be arbitrarily extended in space and time. We find that actions changing the survival of individuals living in the future are generally more strongly discounted than fecundity-changing actions for all future times and that the value of future rewards increases as individuals live longer. We also find that delayed fitness effects may not only be discounted by a constant factor per unit delay (exponential discounting), but that, as soon as there is localized dispersal in a population, discounting per unit delay is likely to fall rapidly for small delays and then slowly for longer delays (hyperbolic discounting). As dispersal tends to be localized in natural populations, our results suggest that evolution is likely to favour individuals that express present-biased behaviours and that may be time-inconsistent with respect to their group-wide effects.     

Individual flexibility and choice of foraging strategy in Polyrhachis laboriosa F. Smith (Hymenoptera, Formicidae)

Summary: We report in this study that the tree-dwelling African ant Polyrhachis laboriosa (Formicinae) uses different foraging strategies according to the size of the available food sources. We demonstrate that a recruitment behaviour can be induced with a 125 7l alimentary reward and that foraging remains solitary when rewards are smaller. Small rewards do not elicit trail-laying behaviour, and exploration behaviour is considerable. With large permanent food sources, scouts use group recruitment and there is less exploration around the reward. The choice of the foraging strategy is determined by the first forager, which modifies its behaviour according to the volume of the food supply. Independently of the size of the reward, the forager shows many exploratory displays during the first visit to the source, and contrary to most ants, it never lays a trail during its first return to the nest. Visual cues remain mainly used for individual orientation; information collected during the first trips are then transmitted to nestmates thanks to temporary trail laying behaviour.     

Floral preferences of bumblebees (Bombus edwardsii) in relation to intermittent versus continuous rewards

Worker bumblebees, Bombus edwardsii, preferably feed from artificial flowers yielding the same (continuous) reward on each visit rather than from flowers yielding variable (intermittent) rewards, even though the long-term expectation of reward is the same at each type of flower. However, variation in degree of preference among individual bees is high. Preferences after long foraging experience correspond closely to early preferences. Rate of flower visitation increases as mean reward increases, and may accelerate preference formation. Preferences are discussed in light of processes thought to control learning in honeybees. From these findings we propose that reward variance and expected time between reinforcements be considered as constraints in models of optimal foraging behaviour.     

Interruptions to foraging and learning in a changing environment

Many resources are both stochastic and variable in their average profitability. Animals have to sample them to track their current states, but whether it is economic to attempt this depends on many factors. Furthermore, there are many interruptions and distractions from foraging (e.g. escape from predators, bad weather, displacement by competitors) which interfere with the acquisition of information. We present a dynamic model of foraging in a stochastic and varying environment, under the constant threat of interruption, to investigate this very general problem. A forager faces two foraging options, one of which provides a known and constant reward, the other providing a reward that is not only stochastic, but whose mean payoff varies in time. The forager has to learn which option has the highest current payoff by sampling. However, interruptions to foraging can occur at any time, the timing and duration of which are beyond the animal's control. When there is a small probability of foraging being interrupted, the forager should forage extensively on the unknown option, but as the probability of interruptions is increased, there is a sudden transition to foraging only on the known option. This occurs because interruptions affect both the level of information required to make exploitation of the unknown option profitable, and the ability to acquire and maintain that information. At what probability of being interrupted this threshold emerges is affected by the value of learning about the unknown option and the duration of interruptions. We discuss the generality of our results with reference to the pervasive problem of updating information in the face of different types of interruption. Copyright 1999 The Association for the Study of Animal Behaviour.     

Monkeys Are More Patient in a Foraging Task than in a Standard Intertemporal Choice Task

Studies of animal impulsivity generally find steep subjective devaluation, or discounting, of delayed rewards – often on the order of a 50% reduction in value in a few seconds. Because such steep discounting is highly disfavored in evolutionary models of time preference, we hypothesize that discounting tasks provide a poor measure of animals’ true time preferences. One prediction of this hypothesis is that estimates of time preferences based on these tasks will lack external validity, i.e. fail to predict time preferences in other contexts. We examined choices made by four rhesus monkeys in a computerized patch-leaving foraging task interleaved with a standard intertemporal choice task. Monkeys were significantly more patient in the foraging task than in the intertemporal choice task. Patch-leaving behavior was well fit by parameter-free optimal foraging equations but poorly fit by the hyperbolic discount parameter obtained from the intertemporal choice task. Day-to-day variation in time preferences across the two tasks was uncorrelated with each other. These data are consistent with the conjecture that seemingly impulsive behavior in animals is an artifact of their difficulty understanding the structure of intertemporal choice tasks, and support the idea that animals are more efficient rate maximizers in the multi-second range than intertemporal choice tasks would suggest.     

Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Timing in reward and decision processes

Sensitivity to time, including the time of reward, guides the behaviour of all organisms. Recent research suggests that all major reward structures of the brain process the time of reward occurrence, including midbrain dopamine neurons, striatum, frontal cortex and amygdala. Neuronal reward responses in dopamine neurons, striatum and frontal cortex show temporal discounting of reward value. The prediction error signal of dopamine neurons includes the predicted time of rewards. Neurons in the striatum, frontal cortex and amygdala show responses to reward delivery and activities anticipating rewards that are sensitive to the predicted time of reward and the instantaneous reward probability. Together these data suggest that internal timing processes have several well characterized effects on neuronal reward processing.     

The twin threshold model: risk-intermediate foraging by rufous hummingbirds, Selasphorus rufus

I developed two versions of the twin threshold model (TTM) to assess risk-sensitive foraging decisions by rufous hummingbirds. The model incorporates energy thresholds for both starvation and reproduction and assesses how three reward distributions with a common mean but different levels of variance interact with these critical thresholds to determine fitness. Fitness, a combination of survival and reproduction, is influenced by both the amount of variance in the distributions and the relative position of the common mean between the thresholds. The model predicts that risk-intermediate foraging is often the optimal policy, and that risk aversion is favoured as the common mean of the distributions approaches the starvation threshold, whereas risk preference is favoured as the common mean approaches the reproduction threshold. Tests with free-living hummingbirds supported these predictions. Hummingbirds were presented with three distributions of nectar rewards that had a common mean but Nil, Moderate or High levels of variance. Birds preferred intermediate levels of variance (Moderate) when presented with all three rewards simultaneously, and became more risk-averse as the mean of the distributions was decreased but more risk-prone as the mean was increased. Birds preferred Nil when it was paired with Moderate or with High, but preferred Moderate in the presence of Nil and High together. This reversal of preference is a violation of regularity, conventionally interpreted as irrational choice behaviour. I provide an alternative version of the TTM demonstrating that violations of regularity can occur when relative instead of absolute evaluation mechanisms are used.     

Floral Temperature and Optimal Foraging: Is Heat a Feasible Floral Reward for Pollinators?

As well as nutritional rewards, some plants also reward ectothermic pollinators with warmth. Bumble bees have some control over their temperature, but have been shown to forage at warmer flowers when given a choice, suggesting that there is some advantage to them of foraging at warm flowers (such as reducing the energy required to raise their body to flight temperature before leaving the flower). We describe a model that considers how a heat reward affects the foraging behaviour in a thermogenic central-place forager (such as a bumble bee). We show that although the pollinator should spend a longer time on individual flowers if they are warm, the increase in total visit time is likely to be small. The pollinator's net rate of energy gain will be increased by landing on warmer flowers. Therefore, if a plant provides a heat reward, it could reduce the amount of nectar it produces, whilst still providing its pollinator with the same net rate of gain. We suggest how heat rewards may link with plant life history strategies.     

Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

BACKGROUND: Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. RESULTS: We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. CONCLUSIONS: The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.     

A Temporal Dimension to the Influence of Pollen Rewards on Bee Behaviour and Fecundity in Aloe tenuior

The net effect of pollen production on fecundity in plants can range from negative – when self-pollen interferes with fecundity due to incompatibility mechanisms, to positive – when pollen availability is associated with increased pollinator visitation and fecundity due to its utilization as a reward. We investigated the responses of bees to pollen and nectar rewards, and the effects of these rewards on pollen deposition and fecundity in the hermaphroditic succulent shrub Aloe tenuior. Self-pollinated plants failed to set fruit, but their ovules were regularly penetrated by self-pollen tubes, which uniformly failed to develop into seeds as expected from ovarian self-incompatibility (or strong early inbreeding depression). Bees consistently foraged for pollen during the morning and early afternoon, but switched to nectar in the late afternoon. As a consequence of this differential foraging, we were able to test the relative contribution to fecundity of pollen- versus nectar-collecting flower visitors. We exposed emasculated and intact flowers in either the morning or late afternoon to foraging bees and showed that emasculation reduced pollen deposition by insects in the morning, but had little effect in the afternoon. Despite the potential for self-pollination to result in ovule discounting due to late-acting self-sterility, fecundity was severely reduced in artificially emasculated plants. Although there were temporal fluctuations in reward preference, most bee visits were for pollen rewards. Therefore the benefit of providing pollen that is accessible to bee foragers outweighs any potential costs to fitness in terms of gender interference in this species.     

Humans can adopt optimal discounting strategy under real-time constraints

Critical to our many daily choices between larger delayed rewards, and smaller more immediate rewards, are the shape and the steepness of the function that discounts rewards with time. Although research in artificial intelligence favors exponential discounting in uncertain environments, studies with humans and animals have consistently shown hyperbolic discounting. We investigated how humans perform in a reward decision task with temporal constraints, in which each choice affects the time remaining for later trials, and in which the delays vary at each trial. We demonstrated that most of our subjects adopted exponential discounting in this experiment. Further, we confirmed analytically that exponential discounting, with a decay rate comparable to that used by our subjects, maximized the total reward gain in our task. Our results suggest that the particular shape and steepness of temporal discounting is determined by the task that the subject is facing, and question the notion of hyperbolic reward discounting as a universal principle.     

Exponential Versus Hyperbolic Discounting of Delayed Outcomes: Risk and Waiting Time

Frequently, animals must choose between more immediate, smallerrewards and more delayed, but larger rewards. For example, theyoften must decide between accepting a smaller prey item versuscontinuing to search for a larger one, or between entering aleaner patch versus travelling to a richer patch that is furtheraway. In both situations, choice of the more immediate, butsmaller reward may be interpreted as implying that the valueof the later reward is discounted; that is, the value of thelater reward decreases as the delay to its receipt increases.This decrease in value may occur because of the increased riskinvolved in waiting for rewards, or because of the decreasedrate of reward associated with increased waiting time. The presentresearch attempts to determine the form of the relation betweenvalue and delay, and examines implications of this relationfor mechanisms underlying risk-sensitive foraging. Two accounts of the relation between value and delay have beenproposed to describe the decrease in value resulting from increasesin delay: an exponential model and a hyperbolic model. Our researchdemonstrates that, of the two, a hyperbola-like discountingmodel consistently explains more of the variance in temporaldiscounting data at the group level and, importantly, at theindividual level as well. We show mathematically that the hyperbolicmodel shares fundamental features with models of prey and patchchoice. In addition, the present review highlights the implicationsof a psychological perspective for the behavioral biology ofrisksensitive foraging, as well as the implications of an ecologicalperspective for the behavioral psychology of risk-sensitivechoice and decision-making.     

Will travel for food: spatial discounting in two new world monkeys

Nonhuman animals steeply discount the future, showing a preference for small, immediate over large, delayed rewards. Currently unclear is whether discounting functions depend on context. Here, we examine the effects of spatial context on discounting in cotton-top tamarins (Saguinus oedipus) and common marmosets (Callithrix jacchus), species known to differ in temporal discounting. We presented subjects with a choice between small, nearby rewards and large, distant rewards. Tamarins traveled farther for the large reward than marmosets, attending to the ratio of reward differences rather than their absolute values. This species difference contrasts with performance on a temporal task in which marmosets waited longer than tamarins for the large reward. These comparative data indicate that context influences choice behavior, with the strongest effect seen in marmosets who discounted more steeply over space than over time. These findings parallel details of each species' feeding ecology. Tamarins range over large distances and feed primarily on insects, which requires using quick, impulsive action. Marmosets range over shorter distances than tamarins and feed primarily on tree exudates, a clumped resource that requires patience to wait for sap to exude. These results show that discounting functions are context specific, shaped by a history of ecological pressures.     

Delay discounting and probability discounting as related to cigarette smoking status in adults

This study examined relations between adult smokers and non-smokers and the devaluation of monetary rewards as a function of delay (delay discounting, DD) or probability (probability discounting, PD). The extent to which individuals discount value, either as a function of a reward being delayed or probabilistic, has been taken to reflect individual differences in impulsivity. Those who discount most are considered most impulsive. Previous research has shown that adult smokers discount the value of delayed rewards more than adult non-smokers. However, in the one published study that examined probability discounting in adult smokers and non-smokers, the smokers did not discount the value of probabilistic rewards more than the non-smoker controls. From this past research, it was hypothesized that measures of delay discounting would differentiate between smokers and non-smokers but that probability discounting would not. Participants were 54 (25 female) adult smokers (n = 25) and non-smokers (n = 29). The smokers all reported smoking at least 20 cigarettes per day, and the non-smokers reported having never smoked. The results indicated that the smokers discounted significantly more than the non-smokers by both delay and probability. Unlike past findings, these results suggest that both delay and probability discounting are related to adult cigarette smoking; however, it also was determined that DD was a significantly stronger predictor of smoking than PD.     

Response bias is unaffected by delay length in a delay discounting paradigm

This study examined the contribution of response bias to measures of delay discounting in Long-Evans rats (n = 8) using the adjusting amount procedure. Under this procedure, we assessed preference for 150 μl of 10% sucrose solution delivered following a delay over a variable-amount alternative delivered immediately. Bias was calculated based on relative preference when reinforcers were delivered immediately from both alternatives. We extended this assessment procedure to examine preference when rewards from both alternatives were equally delayed (2, 4, 8, or 16 s) in addition to assessing a traditional delay discounting function. Relative preference was similar across delays and slightly larger than 150 μl. These results indicate that response bias was stable and suggests a relative aversion for the adjusting alternative, which may be due to the variability in reward size associated with that alternative.     

Reduced sensitivity to immediate reward during decision-making in older than younger adults

We examined whether older adults differ from younger adults in the degree to which they favor immediate over delayed rewards during decision-making. To examine the neural correlates of age-related differences in delay discounting we acquired functional MR images while participants made decisions between smaller but sooner and larger but later monetary rewards. The behavioral results show age-related reductions in delay discounting. Less impulsive decision-making in older adults was associated with lower ventral striatal activations to immediate reward. Furthermore, older adults showed an overall higher percentage of delayed choices and reduced activity in the dorsal striatum than younger adults. This points to a reduced reward sensitivity of the dorsal striatum in older adults. Taken together, our findings indicate that less impulsive decision-making in older adults is due to a reduced sensitivity of striatal areas to reward. These age-related changes in reward sensitivity may result from transformations in dopaminergic neuromodulation with age.