Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

BACKGROUND: Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. RESULTS: We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. CONCLUSIONS: The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.     

Reduced sensitivity to immediate reward during decision-making in older than younger adults

We examined whether older adults differ from younger adults in the degree to which they favor immediate over delayed rewards during decision-making. To examine the neural correlates of age-related differences in delay discounting we acquired functional MR images while participants made decisions between smaller but sooner and larger but later monetary rewards. The behavioral results show age-related reductions in delay discounting. Less impulsive decision-making in older adults was associated with lower ventral striatal activations to immediate reward. Furthermore, older adults showed an overall higher percentage of delayed choices and reduced activity in the dorsal striatum than younger adults. This points to a reduced reward sensitivity of the dorsal striatum in older adults. Taken together, our findings indicate that less impulsive decision-making in older adults is due to a reduced sensitivity of striatal areas to reward. These age-related changes in reward sensitivity may result from transformations in dopaminergic neuromodulation with age.     

Stochastic patch exploitation model

A solitary animal is foraging in a patch consisting of discrete prey items. We develop a stochastic model for the accumulation of gain as a function of elapsed time in the patch. The model is based on the waiting times between subsequent encounters with the prey items. The novelty of the model is in that it renders possible–via parameterization of the waiting time distributions: the incorporation of different foraging situations and patch structures into the gain process. The flexibility of the model is demonstrated with different foraging scenarios. Dependence of gain expectation and variance of the parameters of the waiting times is studied under these conditions. The model allows us to comment upon some of the basic concepts in contemporary foraging theory.     

The ecology and evolution of patience in two New World monkeys

Decision making often involves choosing between small, short-term rewards and large, long-term rewards. All animals, humans included, discount future rewards--the present value of delayed rewards is viewed as less than the value of immediate rewards. Despite its ubiquity, there exists considerable but unexplained variation between species in their capacity to wait for rewards--that is, to exert patience or self-control. Using two closely related primates--common marmosets (Callithrix jacchus) and cotton-top tamarins (Saguinus oedipus)--we uncover a variable that may explain differences in how species discount future rewards. Both species faced a self-control paradigm in which individuals chose between taking an immediate small reward and waiting a variable amount of time for a large reward. Under these conditions, marmosets waited significantly longer for food than tamarins. This difference cannot be explained by life history, social behaviour or brain size. It can, however, be explained by feeding ecology: marmosets rely on gum, a food product acquired by waiting for exudate to flow from trees, whereas tamarins feed on insects, a food product requiring impulsive action. Foraging ecology, therefore, may provide a selective pressure for the evolution of self-control.     

Ability to delay gratification and BMI in preadolescence

Delay of gratification tasks require an individual to forgo an immediate reward and wait for a more desirable delayed reward. This study used an ecologically valid measure of delayed gratification to test the hypothesis that preadolescents with higher BMI would be less likely to delay gratification. Healthy Hawks is a 12-week educational/behavioral obesity intervention at the University of Kansas Medical Center. Each week, children earn a point if they complete their goals worksheet. They can spend that point immediately on a small toy prize or save points to use on a larger prize. We retrospectively calculated the percentage of points saved over the 12 weeks for 59 children (28 females) ages 8-12 years old (mean = 10.29 ± 1.39). Spearman correlation revealed that higher BMI percentile was associated with reduced point savings (r = 0.33, P = 0.01). Similarly, obese preadolescents saved significantly fewer points than healthy weight (HW) and overweight preadolescents (t (57) = 3.14, P < 0.01). Results from our ecologically valid measure support the theory that obese children are less likely to delay gratification than overweight and HW children. Even for nonfood rewards, preadolescent children with higher BMIs prefer the immediate reward over a delayed, larger reward. This has implications for developing specific strategies within obesity treatments aimed at improving delayed gratification.     

Foraging for intermittently refuged prey: theory and field observations of a parasitoid

Many insect herbivores feed in concealed locations but become accessible intermittently, creating windows of greater vulnerability to attack, and generating a proportion of the prey population that is readily accessible to foraging natural enemies. We incorporated accessible prey into an extant optimal foraging model, and found that this addition allowed opportunistic exploitation of prey that have already emerged from refugia (the leaving strategy) as a viable strategy, in addition to waiting at refugia for prey to emerge (the waiting strategy). We parameterized the model empirically for the parasitoid Macrocentrus grandii and its host, Ostrinia nubilalis, under field conditions. The model predicted that M. grandii should adopt a leaving strategy when host patch density is high (travel time between patches is short), but a waiting strategy when host patch density is low (travel time between patches is long). Field observations of M. grandii patch tenure were consistent with model predictions, indicating that M. grandii exhibited flexible behaviour based on experience within a foraging bout, and that these behavioural shifts improved foraging efficiency. Behaviour of M. grandii was responsive to heterogeneity in host emergence rates, and appeared to be driven by the relatively small proportion of the host population that became accessible at a fast rate. Therefore understanding forager responses to intermittently refuged prey may require characterization of the behaviour of a subset of the prey population, rather than the average prey individual. The model can potentially be used as a framework for comparative studies across forager taxa, to understand when foragers on intermittently accessible prey should adopt fixed waiting or leaving strategies vs. a flexible strategy that is responsive to the current environment.     

The Role of Serotonin in the Regulation of Patience and Impulsivity

Classic theories suggest that central serotonergic neurons are involved in the behavioral inhibition that is associated with the prediction of negative rewards or punishment. Failed behavioral inhibition can cause impulsive behaviors. However, the behavioral inhibition that results from predicting punishment is not sufficient to explain some forms of impulsive behavior. In this article, we propose that the forebrain serotonergic system is involved in “waiting to avoid punishment” for future punishments and “waiting to obtain reward” for future rewards. Recently, we have found that serotonergic neurons increase their tonic firing rate when rats await food and water rewards and conditioned reinforcer tones. The rate of tonic firing during the delay period was significantly higher when rats were waiting for rewards than for tones, and rats were unable to wait as long for tones as for rewards. These results suggest that increased serotonergic neuronal firing facilitates waiting behavior when there is the prospect of a forthcoming reward and that serotonergic activation contributes to the patience that allows rats to wait longer. We propose a working hypothesis to explain how the serotonergic system regulates patience while waiting for future rewards.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Effects of multiple delayed rewards on delay discounting in an adjusting amount procedure

It has been suggested that when the delivery of several rewards is separated in time, e.g. one reward immediately and a second reward a few moments later, the value of an alternative that includes these "bundled" rewards will be the sum of the hyperbolic discount functions of the individual rewards. The current study examined this hypothesis using an adjusting amount procedure. In this procedure, rats chose between a delayed food alternative and an immediate food alternative, where the amount of immediate food altered according to each rat's choices. The size of the immediate reward when rats were indifferent between the delayed and immediate alternatives indexed the value of the delayed alternative. Discount functions describing the relationship between the indifference points and the delay to food were created for conditions in which the delay alternative consisted of a single reward (150mul of sucrose solution) delayed by 0, 2, 4, 8, or 16s following the reinforced response. These functions were used to predict the indifference points in other conditions for which an additional 150mul of sucrose solution was delivered at 0, 4, 8, or 16s following the reinforced response. The model fit the data well. However, there were systematic deviations that suggested animals were sensitive to the context within which delays were presented, in addition to the delays themselves. That is, preference for the delayed alternative was lower than predicted when the delay to the additional reward was long (8 or 16s) and higher than the predicted values when it was short (0 or 4s).     

Do kleptoparasites reduce their own foraging effort in order to detect kleptoparasitic opportunities?: An empirical test of a key assumption of kleptoparasitic models

Determining if, or when, individuals trade off time spent personally feeding against time spent monitoring others for kleptoparasitism opportunities is essential to an understanding of the evolution of scrounging and usurpation behaviours. We provide a first field test of whether kleptoparasites reduce their personal foraging effort in situations where the frequency and rewards of kleptoparasitism increase. We provided experimental food patches for wild European blackbirds that varied in the distribution of prey and that had a potentially high rate of kleptoparasitism within pairs of blackbirds feeding in them. Although individuals differed in their rate of kleptoparasitism, they did not vary in the size of the reward that they gained from kleptoparasitism. As prey became more clumped, kleptoparasitism rate and its reward per incident increased on average. There was, however, no evidence that individuals that were kleptoparasitising more quickly and/or at a higher frequency had lower personal foraging effort. In contrast, foraging effort increased in both birds compared to when they were foraging alone, independent of dominance, kleptoparasitic opportunity or reward. Our evidence suggests that in some circumstances a kleptoparasite can detect kleptoparasitic opportunities without compromising its own personal foraging rate.     

The subjective value of delayed and probabilistic outcomes: Outcome size matters for gains but not for losses

The subjective value of a reward (gain) is related to factors such as its size, the delay to its receipt and the probability of its receipt. We examined whether the subjective value of losses was similarly affected by these factors in 128 adults. Participants chose between immediate/certain gains or losses and larger delayed/probabilistic gains or losses. Rewards of $100 were devalued as a function of their delay (“discounted”) relatively less than $10 gains while probabilistic $100 rewards were discounted relatively more than $10 rewards. However, there was no effect of outcome size on discounting of delayed or probabilistic losses. For delayed outcomes of each size, the degree to which gains were discounted was positively correlated with the degree to which losses were discounted, whereas for probabilistic outcomes, no such correlation was observed. These results suggest that the processes underlying the subjective valuation of losses are different from those underlying the subjective valuation of gains.     

Experimental determination of the spatial scale of a prey patch from the predator’s perspective

Foraging theory predicts that predators should prefer foraging in habitat patches with higher prey densities. However, density depends on the spatial scale at which a “patch” is defined by an observer. Ecologists strive to measure prey densities at the same scale that predators do, but many natural landscapes lack obvious, well-defined prey patches. Thus one must determine the scale at which predators define patches of prey. We estimated the scale at which guppies, Poecilia reticulata, selected patches of zooplankton prey using a behavioral assay. Guppies could choose between two prey arrays, each manipulated to have a density that depended on the spatial scale at which density was calculated. We estimated the scale of guppy foraging by comparing guppy preferences across a series of trials in which we systematically varied the scale associated with “high” prey density. This approach enables the application of foraging theory to non-discrete habitats and prey landscapes.     

Competition in a group of equal foragers

Abstract Using techniques from renewal process theory, we build a stochastic model for gain accumulation in a group of equal competitors foraging in a patchy environment. The model for gain of the individuals is based on the waiting times between subsequent prey encounters by the group. These waiting times depend on the number of foragers in the group. A single parameter of this dependency encompasses a variety of foraging scenarios, from co-operation to scramble. With constant patch size, correlations between gains of any pair of foragers are negative. This dependency is most intense in small groups. Increased variation in patch size makes correlations in gains between group members positive irrespective of the group size. For a solitary forager, variance in gain approaches zero with increasing time in the patch. For an individual member in a group, variance grows monotonically. Thus, depending on the patch departure rule controlling the time to be spent in the patch, solitary foragers may have a smaller variance in gain than members in a group. As solitary foragers also potentially harvest all prey in the patch, it is hard to believe that grouping behavior would evolve solely on the basis of foraging.     

How should a clever baboon choose and move rocks?

Chacma baboons (Papio ursinus) intentionally overturn rocks to feed on the invertebrates beneath. However, baboons do not move all the rocks they encounter, with this presumably reflecting cost–benefit (or effort–reward) trade‐offs in their foraging behavior. We ask, how do “clever baboons” choose rock sizes and shapes and move these rocks? Using optimal foraging theory, we predicted that baboons would prefer to move medium‐sized rocks, a trade‐off between moving larger rocks that might require more effort to move, and smaller rocks that likely do not provide enough prey (the reward) to make the effort worthwhile. We also expected baboons to prefer rounded rocks as these will require less energy to move by rolling (rather than being flipped as for flat rocks) and that the effort of rock movement might be offset by moving rocks along the shortest axis. We show that baboons have clear preferences for specific rock sizes (medium‐sized) and shapes (angular and flat when these were medium‐sized), and the way in which rocks are moved (along the shortest axis). Prey occurred infrequently under rocks. The low predictability of prey beneath rocks suggests that such prey, when encountered, is of considerable value to baboons for them to expend the search effort, and also explains the extensive nature of rock movement by baboons in the landscape. Our study provides a novel application of the optimal foraging theory concept and has important implications for understanding and predicting how animals choose to move rocks.     

Genetic variation in foraging traits among inbred lines of a predatory mite

Response of predators to herbivore-induced plant volatiles can affect the length of time a predator spends in a prey patch and the probability of a predator finding a new prey patch. Variation in response to herbivore-induced plant volatiles may lead to different foraging decisions among individuals, thereby affecting both within-patch dynamics and between-patch dispersal. We found significant phenotypic and additive genetic variation in two behavioral assays of response to herbivore-induced plant volatiles among inbred isofemale lines of the predatory mite, Phytoseiulus persimilis. In wind-tunnel tests to measure patch residence time, adult female predators from certain lines left prey patches sooner than others when a distant source of herbivore-induced plant volatiles was presented; whereas such variation disappeared when no distant volatiles were presented. In a measure of patch location, certain lines were more likely than others to locate a prey-infested leaf disc; again there was no difference when uninfested leaf discs were used. Patch location was negatively correlated with patch residence. That is, lines that were more likely to leave a prey patch in the presence of distant volatiles were also more likely to find an odor source (ie, prey patch) from a distance of 20 cm. These two foraging-related behaviors are heritable. A continuous distribution of both behaviors indicated that several to many loci may be responsible for these behavioral traits. Our line-crossing experiments suggested that maternal influence could be excluded. Substantial phenotypic variation in two other foraging-related traits, consumption and oviposition, were also detected among inbred lines. Consumption and oviposition were positively correlated; however, the relationship (slope) varied among inbred lines, suggesting that predatory mites vary in food conversion efficiency. A relationship was detected between patch residence and consumption. Patch location, as one important foraging trait, appeared to be negatively related to consumption, suggesting a trade-off between searching for patches and reproduction.     

Ecological consequences of the bold–shy continuum: the effect of predator boldness on prey risk

Although the existence of different personality traits within and between animal populations has been relatively well established, the ecological implications of this variation remain neglected. In this study we tested whether differences in the boldness of pairs of three-spined sticklebacks led to differential predation risk in their prey, Chironomidae larvae. Bolder pairs, those that left a refuge and crossed the tank mid-line sooner, ate a greater proportion of prey in 10 min than less bold fish (therefore prey were at a greater per capita risk). Fish crossed the mid-line more rapidly when a larger number of prey were presented, suggesting they accepted greater risk in return for a larger foraging reward. Perception of predation risk also affected the differences between fish in boldness, as larger fish crossed the mid-line sooner after leaving the refuge (larger fish are less at risk from predation). Hence, an interesting trophic interaction occurs, where the risk experienced by the chironomid larvae is determined by the risk perceived by their predators. Through the variation generated by boldness, a form of behaviourally mediated trophic cascade can occur within (as well as between) communities.     

Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice

In choosing between different rewards expected after unequal delays, humans and animals often prefer the smaller but more immediate reward, indicating that the subjective value or utility of reward is depreciated according to its delay. Here, we show that neurons in the primate caudate nucleus and ventral striatum modulate their activity according to temporally discounted values of rewards with a similar time course. However, neurons in the caudate nucleus encoded the difference in the temporally discounted values of the two alternative targets more reliably than neurons in the ventral striatum. In contrast, neurons in the ventral striatum largely encoded the sum of the temporally discounted values, and therefore, the overall goodness of available options. These results suggest a more pivotal role for the dorsal striatum in action selection during intertemporal choice.     

Harder than Expected: Increased Conflict in Clearly Disadvantageous Delayed Choices in a Computer Game

When choosing between immediate and temporally delayed goods, people sometimes decide disadvantageously. Here, we aim to provide process-level insight into differences between individually determined advantageous and disadvantageous choices. Participants played a computer game, deciding between two different rewards of varying size and distance by moving an agent towards the chosen reward. We calculated individual models of advantageous choices and characterized the decision process by analyzing mouse movements. The larger amount of participants’ choices was classified as advantageous and the disadvantageous choices were biased towards choosing sooner/smaller rewards. The deflection of mouse movements indicated more conflict in disadvantageous choices compared with advantageous choices when the utilities of the options differed clearly. Further process oriented analysis revealed that disadvantageous choices were biased by a tendency for choice-repetition and an undervaluation of the value information in favour of the delay information, making rather simple choices harder than could be expected from the properties of the decision situation.     

Tolerance of understory plants subject to herbivory by roe deer

Classical foraging theory states that animals feeding in a patchy environment can maximise their long term prey capture rates by quitting food patches when they have depleted prey to a certain threshold level. Theory suggests that social foragers may be better able to do this if all individuals in a group have access to the prey capture information of all other group members. This will allow all foragers to make a more accurate estimation of the patch quality over time and hence enable them to quit patches closer to the optimal prey threshold level. We develop a model to examine the foraging efficiency of three strategies that could be used by a cohesive foraging group to initiate quitting a patch, where foragers do not use such information, and compare these with a fourth strategy in which foragers use public information of all prey capture events made by the group. We carried out simulations in six different prey environments, in which we varied the mean number of prey per patch and the variance of prey number between patches. Groups sharing public information were able to consistently quit patches close to the optimal prey threshold level, and obtained constant prey capture rates, in groups of all sizes. In contrast all groups not sharing public information quit patches progressively earlier than the optimal prey threshold value, and experienced decreasing prey capture rates, as group size increased. This is more apparent as the variance in prey number between patches increases. Thus in a patchy environment, where uncertainty is high, although public information use does not increase the foraging efficiency of groups over that of a lone forager, it certainly offers benefits over groups which do not, and particularly where group size is large.