Theory of Choice in Bandit,Information Sampling and Foraging Tasks

Decision making has been studied with a wide array of tasks. Here we examine the theoretical structure of bandit, information sampling and foraging tasks. These tasks move beyond tasks where the choice in the current trial does not affect future expected rewards. We have modeled these tasks using Markov decision processes (MDPs). MDPs provide a general framework for modeling tasks in which decisions affect the information on which future choices will be made. Under the assumption that agents are maximizing expected rewards, MDPs provide normative solutions. We find that all three classes of tasks pose choices among actions which trade-off immediate and future expected rewards. The tasks drive these trade-offs in unique ways, however. For bandit and information sampling tasks, increasing uncertainty or the time horizon shifts value to actions that pay-off in the future. Correspondingly, decreasing uncertainty increases the relative value of actions that pay-off immediately. For foraging tasks the time-horizon plays the dominant role, as choices do not affect future uncertainty in these tasks.     

Cortical and Hippocampal Correlates of Deliberation during Model-Based Decisions for Rewards in Humans

How do we use our memories of the past to guide decisions we''ve never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward — such as when planning routes using a cognitive map or chess moves using predicted countermoves — and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation.     

Implications of poor taxonomy in conservation

Red Lists are legal instruments that guide decisions within the context of conservation. Such lists can indicate the most appropriate conservation actions, leading to choices of priority areas for conservation, or create alternatives to ensure the preservation of threatened species. However, if Red Lists are effective tools in the conservation of threatened species, then it is important that we have guaranteed the quality of taxonomic delimitation of these taxa. Here we discuss a case of taxonomic confusion, emphasizing the implications of misidentification in Red Lists. To avoid mistakes in species conservation, we advocate that a minimum level of taxonomic knowledge is needed to ensure success in preserving target species. Therefore, complementarity of taxonomy and conservation guarantee stronger conservation actions.     

Mushroom Bodies Regulate Habit Formation in Drosophila

To make good decisions, we evaluate past choices to guide later decisions. In most situations, we have the opportunity to simultaneously learn about both the consequences of our choice (i.e., operantly) and the stimuli associated with correct or incorrect choices (i.e., classically) [1]. Interestingly, in many species, including humans, these learning processes occasionally lead to irrational decisions [2]. An extreme case is the habitual drug user consistently administering the drug despite the negative consequences, but we all have experience with our own, less severe habits. The standard animal model employs a combination of operant and classical learning components to bring about habit formation in rodents [3] and [4]. After extended training, these animals will press a lever even if the outcome associated with lever-pressing is no longer desired [5]. In this study, experiments with wild-type and transgenic flies revealed that a prominent insect neuropil, the mushroom bodies (MBs), regulates habit formation in flies by inhibiting the operant learning system when a predictive stimulus is present. This inhibition enables generalization of the classical memory and prevents premature habit formation. Extended training in wild-type flies produced a phenocopy of MB-impaired flies, such that generalization was abolished and goal-directed actions were transformed into habitual responses.     

The influence of recent decisions on future goal selection

Recent decisions about actions and goals can have effects on future choices. Several studies have shown an effect of the previous trial history on neural activity in a subsequent trial. Often, but not always, these effects originate from task requirements that make it necessary to maintain access to previous trial information to make future decisions. Maintaining the information about recent decisions and their outcomes can play an important role in both adapting to new contingencies and learning. Previous goal decisions must be distinguished from goals that are currently being planned to avoid perseveration or more general errors. Output monitoring is probably based on this separation of accomplished past goals from pending future goals that are being pursued. Behaviourally, it has been shown that the history context can influence the location, error rate and latency of successive responses. We will review the neurophysiological studies in the literature, including data from our laboratory, which support a role for the frontal lobe in tracking previous goal selections and outputs when new goals need to be accomplished.     

Neuromodulatory Control of a Goal-Directed Decision

Many cost-benefit decisions reduce to simple choices between approach or avoidance (or active disregard) to salient stimuli. Physiologically, critical factors in such decisions are modulators of the homeostatic neural networks that bias decision processes from moment to moment. For the predatory sea-slug Pleurobranchaea, serotonin (5-HT) is an intrinsic modulatory promoter of general arousal and feeding. We correlated 5-HT actions on appetitive state with its effects on the approach-avoidance decision in Pleurobranchaea. 5-HT and its precursor 5-hydroxytryptophan (5-HTP) augmented general arousal state and reduced feeding thresholds in intact animals. Moreover, 5-HT switched the turn response to chemosensory stimulation from avoidance to orienting in many animals. In isolated CNSs, bath application of 5-HT both stimulated activity in the feeding motor network and switched the fictive turn response to unilateral sensory nerve stimulation from avoidance to orienting. Previously, it was shown that increasing excitation state of the feeding network reversibly switched the turn motor network response from avoidance to orienting, and that 5-HT levels vary inversely with nutritional state. A simple model posits a critical role for 5-HT in control of the turn network response by corollary output of the feeding network. In it, 5-HT acts as an intrinsic neuromodulatory factor coupled to nutritional status and regulates approach-avoidance via the excitation state of the feeding network. Thus, the neuromodulator is a key organizing element in behavioral choice of approach or avoidance through its actions in promoting appetitive state, in large part via the homeostatic feeding network.     

Design, Development and Decisions

Evolutionary ideas and modern biological knowledge have important roles to play in the understanding of human behaviour. Nevertheless, it is deeply misleading to regard humans as robots in the grip of their genes. A well designed brain should respond to the consequences of behaviour; if an understanding of the likely consequences can be achieved without actually performing the act, then a person who knows that they will be rewarded or punished for certain acts is bound to be influenced by that knowledge. A brain designed in that way facilitates the evolution of societies with explicit social approval of certain activities and explicit disapproval of others. The evolutionary approach to psychology does not imply that individuals do not make free choices. Individuals clearly do make a big difference to what happens in their lives through their decisions. They may be surprised by the consequences of their own actions. A well designed brain should be able to anticipate the consequences of various courses of action and choose between them on the basis of their likely costs and benefits. Planning before doing is clearly of great advantage. People do make well considered decisions and they benefit from doing so.     

A selective emotional decision-making bias elicited by facial expressions

Emotional and social information can sway otherwise rational decisions. For example, when participants decide between two faces that are probabilistically rewarded, they make biased choices that favor smiling relative to angry faces. This bias may arise because facial expressions evoke positive and negative emotional responses, which in turn may motivate social approach and avoidance. We tested a wide range of pictures that evoke emotions or convey social information, including animals, words, foods, a variety of scenes, and faces differing in trustworthiness or attractiveness, but we found only facial expressions biased decisions. Our results extend brain imaging and pharmacological findings, which suggest that a brain mechanism supporting social interaction may be involved. Facial expressions appear to exert special influence over this social interaction mechanism, one capable of biasing otherwise rational choices. These results illustrate that only specific types of emotional experiences can best sway our choices.     

Follow the straggler: zebrafish use a simple heuristic for collective decision-making

Animal groups often make decisions sequentially, from the front to the back of the group. In such cases, individuals can use the choices made by earlier ranks, a form of social information, to inform their own choice. The optimal strategy for such decisions has been explored in models which differ on, for example, whether or not agents take into account the sequence of observed choices. The models demonstrate that choices made later in a sequence are more informative, but it is not clear if animals use this information or rely instead on simpler heuristics, such as quorum rules. We show that a simple rule ‘copy the last observed choice'', gives similar predictions to those of optimal models for most likely sequences. We trained groups of zebrafish to choose one arm of a Y-maze and used them to demonstrate various sequences to naive fish. We show that the naive fish appear to use a simple rule, most often copying the choice of the last demonstrator, which results in near-optimal choices at a fraction of the computational cost.     

Shared or unshared consensus decision in macaques?

Members of a social group have to make collective decisions in order to synchronise their activities. In a shared consensus decision, all group members can take part in the decision whereas in an unshared consensus decision, one individual, usually a dominant member of the group, takes the decision for the rest of the group. It has been suggested that the type of decision-making of a species could be influenced by its social style. To investigate this further, we studied collective movements in two species with opposed social systems, the Tonkean macaque (Macaca tonkeana) and the rhesus macaque (Macaca mulatta). From our results, it appears that the decision to move is the result of the choices and actions of several individuals in both groups. However, this consensus decision involved nearly all group members in Tonkean macaques whereas dominant and old individuals took a prominent role in rhesus macaques. Thus, we suggest that Tonkean macaques display equally shared consensus decisions to move, whereas in the same context rhesus macaque exhibit partially shared consensus decisions. Such a difference in making a collective decision might be linked to the different social systems of the two studied species.     

Amygdaloid and non-amygdaloid fear both influence avoidance of risky foraging in hungry rats

Considerable evidence seems to show that emotional and reflex reactions to feared situations are mediated by the amygdala. It might therefore seem plausible to expect that amygdala-coded fear should also influence decisions when animals make choices about instrumental actions. However, there is not good evidence of this. In particular, it appears, though the literature is conflicted, that once learning is complete, the amygdala may often not be involved in instrumental avoidance behaviours. It is therefore of interest that we have found in rats living for extended periods in a semi-naturalistic ‘closed economy’, where they were given random shocks in regions that had to be entered to obtain food, choices about feeding behaviour were in fact influenced by amygdala-coded fear, in spite of the null effect of amygdalar lesions on fear of dangerous location per se. We suggest that avoidance of highly motivated voluntary behaviour does depend in part on fear signals originating in the amygdala. Such signalling may be one role of well-known projections from amygdala to cortico-striate circuitry.     

Voting Intention and Choices: Are Voters Always Rational and Deliberative?

Human rationality–the ability to behave in order to maximize the achievement of their presumed goals (i.e., their optimal choices)–is the foundation for democracy. Research evidence has suggested that voters may not make decisions after exhaustively processing relevant information; instead, our decision-making capacity may be restricted by our own biases and the environment. In this paper, we investigate the extent to which humans in a democratic society can be rational when making decisions in a serious, complex situation–voting in a local political election. We believe examining human rationality in a political election is important, because a well-functioning democracy rests largely upon the rational choices of individual voters. Previous research has shown that explicit political attitudes predict voting intention and choices (i.e., actual votes) in democratic societies, indicating that people are able to reason comprehensively when making voting decisions. Other work, though, has demonstrated that the attitudes of which we may not be aware, such as our implicit (e.g., subconscious) preferences, can predict voting choices, which may question the well-functioning democracy. In this study, we systematically examined predictors on voting intention and choices in the 2014 mayoral election in Taipei, Taiwan. Results indicate that explicit political party preferences had the largest impact on voting intention and choices. Moreover, implicit political party preferences interacted with explicit political party preferences in accounting for voting intention, and in turn predicted voting choices. Ethnic identity and perceived voting intention of significant others were found to predict voting choices, but not voting intention. In sum, to the comfort of democracy, voters appeared to engage mainly explicit, controlled processes in making their decisions; but findings on ethnic identity and perceived voting intention of significant others may suggest otherwise.     

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.     

Visual extrapolation under risk: human observers estimate and compensate for exogenous uncertainty

Humans commonly face choices between multiple options with uncertain outcomes. Such situations occur in many contexts, from purely financial decisions (which shares should I buy?) to perceptuo-motor decisions between different actions (where should I aim my shot at goal?). Regardless of context, successful decision-making requires that the uncertainty at the heart of the decision-making problem is taken into account. Here, we ask whether humans can recover an estimate of exogenous uncertainty and then use it to make good decisions. Observers viewed a small dot that moved erratically until it disappeared behind an occluder. We varied the size of the occluder and the unpredictability of the dot''s path. The observer attempted to capture the dot as it emerged from behind the occluded region by setting the location and extent of a ‘catcher’ along the edge of the occluder. The reward for successfully catching the dot was reduced as the size of the catcher increased. We compared human performance with that of an agent maximizing expected gain and found that observers consistently selected catcher size close to this theoretical solution. These results suggest that humans are finely tuned to exogenous uncertainty information and can exploit it to guide action.     

Linking Time-Use Data to Explore Health Outcomes: Choosing to Vaccinate Against Influenza

To inform public health and medical decision makers concerning vaccination interventions, a methodology for merging and analyzing detailed activity data and health outcomes is presented. The objective is to investigate relationships between individual’s activity choices and their decision to receive an influenza vaccination. Data from the Behavioral Risk Factor Surveillance System (BRFSS) are used to predict vaccination rates in the American Time Use Survey (ATUS) data between 2003 and 2013 by using combined socioeconomic and demographic characteristics. The correlations between the extensive (do or not do) and intensive (how much) decisions to perform activities and influenza vaccination are further explored. Significant positive and negative correlations were found between several activities and vaccination. For some activities, the sign of the correlation flips when considering either the intensive or the extensive decision. This flip occurs with highly studied activities, like smoking. Correlations between activities and vaccination can provide an additional metric for targeting those least likely to vaccinate. The methodology outlined in this paper can be replicated to explore correlation among actions and other health outcomes.     

Facial attractiveness: evolutionary based research

Face preferences affect a diverse range of critical social outcomes, from mate choices and decisions about platonic relationships to hiring decisions and decisions about social exchange. Firstly, we review the facial characteristics that influence attractiveness judgements of faces (e.g. symmetry, sexually dimorphic shape cues, averageness, skin colour/texture and cues to personality) and then review several important sources of individual differences in face preferences (e.g. hormone levels and fertility, own attractiveness and personality, visual experience, familiarity and imprinting, social learning). The research relating to these issues highlights flexible, sophisticated systems that support and promote adaptive responses to faces that appear to function to maximize the benefits of both our mate choices and more general decisions about other types of social partners.     

Policy decision options during the first 5 years following certification of polio eradication

Policy makers face a number of difficult choices as they develop policies to ensure maintenance of a polio-free world following global eradication and certification. These policy decisions include choices about immunization, outbreak response (including whether to create a vaccine stockpile), surveillance, containment, management of chronic excretors, and investment in future research. This paper focuses on identifying the categories of decisions and characterizing the actual factors that country-level policy makers must weigh to manage polio risks during the first 5 years after certification. Building on a comprehensive literature review, we report the results of the first qualitative analysis to: (1) systematically characterize each type of decision and the relevant options during the first 5 years after certification, (2) clearly identify critical factors that influence the choices, and (3) specifically demonstrate the interdependence among the decisions to produce a reduced set of decision options. This paper explicitly focuses on the different perspectives of developed and developing countries in characterizing the options. While the management of polio risk in the postcertification period presents important challenges, this comprehensive approach helps simplify the process by focusing on critical decisions.     

Anticipation and Choice Heuristics in the Dynamic Consumption of Pain Relief

Humans frequently need to allocate resources across multiple time-steps. Economic theory proposes that subjects do so according to a stable set of intertemporal preferences, but the computational demands of such decisions encourage the use of formally less competent heuristics. Few empirical studies have examined dynamic resource allocation decisions systematically. Here we conducted an experiment involving the dynamic consumption over approximately 15 minutes of a limited budget of relief from moderately painful stimuli. We had previously elicited the participants’ time preferences for the same painful stimuli in one-off choices, allowing us to assess self-consistency. Participants exhibited three characteristic behaviors: saving relief until the end, spreading relief across time, and early spending, of which the last was markedly less prominent. The likelihood that behavior was heuristic rather than normative is suggested by the weak correspondence between one-off and dynamic choices. We show that the consumption choices are consistent with a combination of simple heuristics involving early-spending, spreading or saving of relief until the end, with subjects predominantly exhibiting the last two.     

Continuous decisions

Humans and other animals evolved to make decisions that extend over time with continuous and ever-changing options. Nonetheless, the academic study of decision-making is mostly limited to the simple case of choice between two options. Here, we advocate that the study of choice should expand to include continuous decisions. Continuous decisions, by our definition, involve a continuum of possible responses and take place over an extended period of time during which the response is continuously subject to modification. In most continuous decisions, the range of options can fluctuate and is affected by recent responses, making consideration of reciprocal feedback between choices and the environment essential. The study of continuous decisions raises new questions, such as how abstract processes of valuation and comparison are co-implemented with action planning and execution, how we simulate the large number of possible futures our choices lead to, and how our brains employ hierarchical structure to make choices more efficiently. While microeconomic theory has proven invaluable for discrete decisions, we propose that engineering control theory may serve as a better foundation for continuous ones. And while the concept of value has proven foundational for discrete decisions, goal states and policies may prove more useful for continuous ones.This article is part of the theme issue ‘Existence and prevalence of economic behaviours among non-human primates’.     

Mechanisms by which neonatal testosterone exposure mediates sex differences in impulsivity in prepubertal rats

Neonatal testosterone, either acting directly or through its conversion to estradiol, can exert organizational effects on the brain and behavior. The goal of the current study was to examine sex differences and determine the role of neonatal testosterone on prefrontal cortex-dependent impulsive choice behavior in prepubertal rats. Male and female prepubertal rats were tested on the delay-based impulsive choice task. Impulsive choice was defined as choosing an immediate small food reward over a delayed large reward. In a first experiment to examine sex differences, males made significantly more impulsive choices than did females. In a second experiment to examine the organizational effects of testosterone, females treated with neonatal testosterone made significantly more impulsive choices than did control females and their performance was indistinguishable from that of control males. In a third experiment to determine if the effect of testosterone on performance is due to the actions of androgens or estrogens through its conversion to estradiol, males treated neonatally with the aromatase inhibitor formestane, which blocks the conversion of testosterone to estradiol, females treated neonatally with the non-aromatizable androgen dihydrotestosterone, and females treated neonatally with estradiol made significantly more impulsive choices than did control females and their performance was indistinguishable from that of control males. Results indicate that male pubertal rats display increased impulsive choice behavior as compared to females, that this sex difference results from organizing actions of testosterone during the neonatal period, and that this effect can result from both androgenic and estrogenic actions.