How the value of the environment controls persistence in visual search

Classic foraging theory predicts that humans and animals aim to gain maximum reward per unit time. However, in standard instrumental conditioning tasks individuals adopt an apparently suboptimal strategy: they respond slowly when the expected value is low. This reward-related bias is often explained as reduced motivation in response to low rewards. Here we present evidence this behavior is associated with a complementary increased motivation to search the environment for alternatives. We trained monkeys to search for reward-related visual targets in environments with different values. We found that the reward-related bias scaled with environment value, was consistent with persistent searching after the target was already found, and was associated with increased exploratory gaze to objects in the environment. A novel computational model of foraging suggests that this search strategy could be adaptive in naturalistic settings where both environments and the objects within them provide partial information about hidden, uncertain rewards.     

Monkeys Are More Patient in a Foraging Task than in a Standard Intertemporal Choice Task

Studies of animal impulsivity generally find steep subjective devaluation, or discounting, of delayed rewards – often on the order of a 50% reduction in value in a few seconds. Because such steep discounting is highly disfavored in evolutionary models of time preference, we hypothesize that discounting tasks provide a poor measure of animals’ true time preferences. One prediction of this hypothesis is that estimates of time preferences based on these tasks will lack external validity, i.e. fail to predict time preferences in other contexts. We examined choices made by four rhesus monkeys in a computerized patch-leaving foraging task interleaved with a standard intertemporal choice task. Monkeys were significantly more patient in the foraging task than in the intertemporal choice task. Patch-leaving behavior was well fit by parameter-free optimal foraging equations but poorly fit by the hyperbolic discount parameter obtained from the intertemporal choice task. Day-to-day variation in time preferences across the two tasks was uncorrelated with each other. These data are consistent with the conjecture that seemingly impulsive behavior in animals is an artifact of their difficulty understanding the structure of intertemporal choice tasks, and support the idea that animals are more efficient rate maximizers in the multi-second range than intertemporal choice tasks would suggest.     

Harder than Expected: Increased Conflict in Clearly Disadvantageous Delayed Choices in a Computer Game

When choosing between immediate and temporally delayed goods, people sometimes decide disadvantageously. Here, we aim to provide process-level insight into differences between individually determined advantageous and disadvantageous choices. Participants played a computer game, deciding between two different rewards of varying size and distance by moving an agent towards the chosen reward. We calculated individual models of advantageous choices and characterized the decision process by analyzing mouse movements. The larger amount of participants’ choices was classified as advantageous and the disadvantageous choices were biased towards choosing sooner/smaller rewards. The deflection of mouse movements indicated more conflict in disadvantageous choices compared with advantageous choices when the utilities of the options differed clearly. Further process oriented analysis revealed that disadvantageous choices were biased by a tendency for choice-repetition and an undervaluation of the value information in favour of the delay information, making rather simple choices harder than could be expected from the properties of the decision situation.     

Managing uncertainty: information and insurance under the risk of starvation

In an uncertain world, animals face both unexpected opportunities and danger. Such outcomes can select for two potential strategies: collecting information to reduce uncertainty, or insuring against it. We investigate the relative value of information and insurance (energy reserves) under starvation risk by offering model foragers a choice between constant and varying food sources over finite foraging bouts. We show that sampling the variable option (choosing it when it is not expected to be good) should decline both with lower reserves and late in foraging bouts; in order to be able to reap the reduction in uncertainty associated with exploiting a variable resource effectively, foragers must be able to afford and compensate for an initial increase in the risk of an energetic shortfall associated with choosing the option when it is bad. Consequently, expected exploitation of the varying option increases as it becomes less variable, and when the overall risk of energetic shortfall is reduced. In addition, little activity on the variable alternative is expected until reserves are built up early in a foraging bout. This indicates that gathering information is a luxury while insurance is a necessity, at least when foraging on stochastic and variable food under the risk of starvation.     

Temporal discounting predicts risk sensitivity in rhesus macaques

Humans and animals tend both to avoid uncertainty and to prefer immediate over future rewards. The comorbidity of psychiatric disorders such as impulsivity, problem gambling, and addiction suggests that a common mechanism may underlie risk sensitivity and temporal discounting. Nonetheless, the precise relationship between these two traits remains largely unknown. To examine whether risk sensitivity and temporal discounting reflect a common process, we recorded choices made by two rhesus macaques in a visual gambling task while we varied the delay between trials. We found that preference for the risky option declined with increasing delay between sequential choices in the task, even when all other task parameters were held constant. These results were quantitatively predicted by a model that assumed that the subjective expected utility of the risky option is evaluated based on the expected time of the larger payoff. The importance of the larger payoff in this model suggests that the salience of larger payoffs played a critical role in determining the value of risky options. These data suggest that risk sensitivity may be a product of other cognitive processes, and specifically that myopia for the future and the salience of jackpots control the propensity to take a gamble.     

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.     

Maintaining Homeostasis by Decision-Making

Living organisms need to maintain energetic homeostasis. For many species, this implies taking actions with delayed consequences. For example, humans may have to decide between foraging for high-calorie but hard-to-get, and low-calorie but easy-to-get food, under threat of starvation. Homeostatic principles prescribe decisions that maximize the probability of sustaining appropriate energy levels across the entire foraging trajectory. Here, predictions from biological principles contrast with predictions from economic decision-making models based on maximizing the utility of the endpoint outcome of a choice. To empirically arbitrate between the predictions of biological and economic models for individual human decision-making, we devised a virtual foraging task in which players chose repeatedly between two foraging environments, lost energy by the passage of time, and gained energy probabilistically according to the statistics of the environment they chose. Reaching zero energy was framed as starvation. We used the mathematics of random walks to derive endpoint outcome distributions of the choices. This also furnished equivalent lotteries, presented in a purely economic, casino-like frame, in which starvation corresponded to winning nothing. Bayesian model comparison showed that—in both the foraging and the casino frames—participants’ choices depended jointly on the probability of starvation and the expected endpoint value of the outcome, but could not be explained by economic models based on combinations of statistical moments or on rank-dependent utility. This implies that under precisely defined constraints biological principles are better suited to explain human decision-making than economic models based on endpoint utility maximization.     

Testosterone levels in feces predict risk‐sensitive foraging in hummingbirds

Risk taking decisions related to the unpredictability of resource availability (risk‐sensitive foraging theory) have typically been explained by behavioral ecology and psychology approaches. However, little attention has been given to the physiological condition of animals as a factor that can influence the direction of foraging preferences. We evaluated the role of steroid hormones testosterone (T) and corticosterone (CORT) on the foraging preferences expressed by white‐eared hummingbirds Hylocharis leucotis in a risk‐sensitivity experiment. We recorded choices made by male individuals to floral arrays with constant and variable rewards (sugar concentration), and associated these with steroid hormone levels quantified at the start of the experiments. We found that males with higher T levels behave as risk‐prone foragers as they perform more visits to flower arrays with variable rewards. Interestingly, CORT levels were similar regardless whether individuals visited both types of array. According to our results, T seems to influence the foraging preferences of male hummingbirds. Individuals with higher levels of this hormone, made more rapid, frequent visits to flowers with variable rewards, and behave consistently as risk‐prone foragers, compared to males with low T levels. These are exciting avenues for future work, particularly considering recent evidence that individuals may exhibit behavioral differences, denoting an apparent personality, which may be associated with phisiological condition influencing how they respond behaviorally to environmental variation.     

Effects of recent experience on foraging decisions by bumble bees

The temporal and spatial scales employed by foraging bees in sampling their environment and making foraging decisions should depend both on the limits of bumble bee memory and on the spatial and temporal pattern of rewards in the habitat. We analyzed data from previous experiments to determine how recent foraging experience by bumble bees affects their flight distances to subsequent flowers. A single visit to a flower as sufficient to affect the flight distance to the next flower. However, longer sequences of two or three visits had an additional effect on the subsequent flight distance of individual foragers. This suggests that bumble bees can integrate information from at least three flowers for making a subsequent foraging decision. The existence of memory for floral characteristics at least at this scale may have significance for floral selection in natural environments.     

Colour association influences honey bee choice between sucrose concentrations

Certain colours associated with floral food resources are more quickly learned by honey bees (Apis mellifera) than are other colours. But the impact of colour, and other floral cues, on bee choice behaviour has not yet been determined. In these experiments, colour association and sugar concentration of reward were varied to assess how they interact to affect bee choice behaviour. Thirty-five bees were individually given binary choices between blue and yellow artificial flowers that contained either the same rewards or rewards of different sucrose concentrations. Honey bee choice between sucrose concentrations was affected by colour association and this effect was greatest when absolute difference between rewards was the lowest. The honey bee's ability to maximize energetic profitability during foraging is constrained by floral cue effectiveness.     

Adaptive strategies for managing uncertainty may explain personality‐related differences in behavioural plasticity

There is growing evidence that individuals within populations show consistent differences in their behaviour across contexts (personality), and that personality is associated with the extent to which individuals adjust their behaviour as function of changing conditions (behavioural plasticity). We propose an evolutionary explanation for a link between personality and plasticity based upon how individuals manage uncertainty. Individuals can employ three categories of tactics to manage uncertainty. They can 1) gather information (sample) to reduce uncertainty, 2) show strategic (state‐dependent) preferences for options that differ in their associated variances in rewards (i.e. variance‐sensitivity), or 3) invest in insurance to mitigate the consequences of uncertainty. We explicitly outline how individual differences in the use of any of these tactics can generate personality‐related differences in behavioural plasticity. For example, sampling effort is likely to co‐vary with individual activity and exploration behaviours, while simultaneously creating population variation in reactions to changes in environmental conditions. Individual differences in the use of insurance may be associated with differences in risk‐taking behaviours, such as boldness in the face of predation, thereby influencing the degree of adaptive plasticity across individuals. Population variation in responsiveness to environmental changes may also reflect individual differences in variance‐sensitivity, because stochastic change in the environment increases variances in rewards, which may both attract and benefit variance‐prone individuals, but not variance‐averse individuals. We review the existing evidence that individual variation in strategies for managing uncertainty exist, and describe how positive‐feedbacks between sampling, variance‐sensitivity and insurance can maintain and exaggerate even small initial differences between individuals in the relative use of these tactics. Given the pervasiveness of the problem of uncertainty, alternative strategies for managing uncertainty may provide a powerful explanation for consistent differences in behaviour and behavioural plasticity for a wide range of traits.     

The foraging benefits of information and the penalty of ignorance

Patch use theory and the marginal value theorem predict that a foraging patch should be abandoned when the costs and benefits of foraging in the patch are equal. This has generally been interpreted as all patches being abandoned when their instantaneous intake rate equals the foraging costs. Bayesian foraging – patch departure is based on a prior estimate of patch qualities and sampling information from the current patch – predicts that instantaneous quitting harvest rates sometimes are not constant across patches but increase with search time in the patch. That is, correct Bayesian foraging theory has appeared incompatible with the widely accepted cost–benefit theories of foraging. In this paper we reconcile Bayesian foraging with cost–benefit theories. The general solution is that a patch should be left not when instantaneous quitting harvest rate reaches a constant level, but when potential quitting harvest rate does. That is, the forager should base its decision on the value now and in the future until the patch is left. We define the difference between potential and instantaneous quitting harvest rates as the foraging benefit of information, FBI. For clumped prey the FBI is positive, and by including this additional benefit of patch harvest the forager is able to reduce its penalty of ignorance.     

Adaptive Management and the Value of Information: Learning Via Intervention in Epidemiology

Optimal intervention for disease outbreaks is often impeded by severe scientific uncertainty. Adaptive management (AM), long-used in natural resource management, is a structured decision-making approach to solving dynamic problems that accounts for the value of resolving uncertainty via real-time evaluation of alternative models. We propose an AM approach to design and evaluate intervention strategies in epidemiology, using real-time surveillance to resolve model uncertainty as management proceeds, with foot-and-mouth disease (FMD) culling and measles vaccination as case studies. We use simulations of alternative intervention strategies under competing models to quantify the effect of model uncertainty on decision making, in terms of the value of information, and quantify the benefit of adaptive versus static intervention strategies. Culling decisions during the 2001 UK FMD outbreak were contentious due to uncertainty about the spatial scale of transmission. The expected benefit of resolving this uncertainty prior to a new outbreak on a UK-like landscape would be £45–£60 million relative to the strategy that minimizes livestock losses averaged over alternate transmission models. AM during the outbreak would be expected to recover up to £20.1 million of this expected benefit. AM would also recommend a more conservative initial approach (culling of infected premises and dangerous contact farms) than would a fixed strategy (which would additionally require culling of contiguous premises). For optimal targeting of measles vaccination, based on an outbreak in Malawi in 2010, AM allows better distribution of resources across the affected region; its utility depends on uncertainty about both the at-risk population and logistical capacity. When daily vaccination rates are highly constrained, the optimal initial strategy is to conduct a small, quick campaign; a reduction in expected burden of approximately 10,000 cases could result if campaign targets can be updated on the basis of the true susceptible population. Formal incorporation of a policy to update future management actions in response to information gained in the course of an outbreak can change the optimal initial response and result in significant cost savings. AM provides a framework for using multiple models to facilitate public-health decision making and an objective basis for updating management actions in response to improved scientific understanding.     

Is exploration a metric for information gathering? Attraction to novelty and plasticity in black-capped chickadees

Animals can learn about the value of resources and predation risk by exploring novel environments or exploring novel stimuli in their regular environments. Still, there is a disconnect in the way that exploration has been defined and measured; exploration is defined in terms of information acquisition, while measured in terms of movement speed and diversity of contacted items in a novel environment. If exploration is indeed a measurement of information gathering, fast explorers should seek to reduce uncertainty about their environment more than slow explorers. Exploration speed has also been linked to behavioral plasticity, where fast explorers move fast but collect less detailed information, thereby forming routines and expressing less plasticity than slow explorers. We test these two hypotheses by comparing exploration in a novel environment to individuals' attraction to novelty and behavioral plasticity. Our results support the view that exploration is a measurement of information-gathering tendencies as fast explorers were more likely to collect novel information, which should reduce uncertainty further than sampling familiar information sources, compared with slower explorers. Furthermore, faster explorers switched to sampling novel information more quickly than slow explorers when the value of the familiar option decreased, opposing the widely held view that faster explorers present more routine-like behavior. By providing familiar and novel foraging options in close spatial contiguity, we demonstrate an attraction to novelty in faster explorers that cannot be confounded by activity rate, thereby suggesting that these individuals seek to reduce uncertainty. In conclusion, our results support the biological validity of the term “exploration” through its association with information gathering.     

Hedging your bets by learning reward correlations in the human brain

Human subjects are proficient at tracking the mean and variance of rewards and updating these via prediction errors. Here, we addressed whether humans can also learn about higher-order relationships between distinct environmental outcomes, a defining ecological feature of contexts where multiple sources of rewards are available. By manipulating the degree to which distinct outcomes are correlated, we show that subjects implemented an explicit model-based strategy to learn the associated outcome correlations and were adept in using that information to dynamically adjust their choices in a task that required a minimization of outcome variance. Importantly, the experimentally generated outcome correlations were explicitly represented neuronally in right midinsula with a learning prediction error signal expressed in rostral anterior cingulate cortex. Thus, our data show that the human brain represents higher-order correlation structures between rewards, a core adaptive ability whose immediate benefit is optimized sampling.     

Model based planners reflect on their model-free propensities

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.     

Fixation patterns in simple choice reflect optimal information sampling

Simple choices (e.g., eating an apple vs. an orange) are made by integrating noisy evidence that is sampled over time and influenced by visual attention; as a result, fluctuations in visual attention can affect choices. But what determines what is fixated and when? To address this question, we model the decision process for simple choice as an information sampling problem, and approximate the optimal sampling policy. We find that it is optimal to sample from options whose value estimates are both high and uncertain. Furthermore, the optimal policy provides a reasonable account of fixations and choices in binary and trinary simple choice, as well as the differences between the two cases. Overall, the results show that the fixation process during simple choice is influenced dynamically by the value estimates computed during the decision process, in a manner consistent with optimal information sampling.