Altered risk-based decision making following adolescent alcohol use results from an imbalance in reinforcement learning in rats

Alcohol use during adolescence has profound and enduring consequences on decision-making under risk. However, the fundamental psychological processes underlying these changes are unknown. Here, we show that alcohol use produces over-fast learning for better-than-expected, but not worse-than-expected, outcomes without altering subjective reward valuation. We constructed a simple reinforcement learning model to simulate altered decision making using behavioral parameters extracted from rats with a history of adolescent alcohol use. Remarkably, the learning imbalance alone was sufficient to simulate the divergence in choice behavior observed between these groups of animals. These findings identify a selective alteration in reinforcement learning following adolescent alcohol use that can account for a robust change in risk-based decision making persisting into later life.     

A decision-making model based on a spiking neural circuit and synaptic plasticity

To adapt to the environment and survive, most animals can control their behaviors by making decisions. The process of decision-making and responding according to cues in the environment is stable, sustainable, and learnable. Understanding how behaviors are regulated by neural circuits and the encoding and decoding mechanisms from stimuli to responses are important goals in neuroscience. From results observed in Drosophila experiments, the underlying decision-making process is discussed, and a neural circuit that implements a two-choice decision-making model is proposed to explain and reproduce the observations. Compared with previous two-choice decision making models, our model uses synaptic plasticity to explain changes in decision output given the same environment. Moreover, biological meanings of parameters of our decision-making model are discussed. In this paper, we explain at the micro-level (i.e., neurons and synapses) how observable decision-making behavior at the macro-level is acquired and achieved.     

Closed-loop interactions between a shoal of zebrafish and a group of robotic fish in a circular corridor

Collective behavior based on self-organization has been observed in populations of animals from insects to vertebrates. These findings have motivated engineers to investigate approaches to control autonomous multi-robot systems able to reproduce collective animal behaviors, and even to collectively interact with groups of animals. In this article, we show collective decision making by a group of autonomous robots and a group of zebrafish, leading to a shared decision about swimming direction. The robots can also modulate the collective decision-making process in biased and non-biased experimental setups. These results demonstrate the possibility of creating mixed societies of vertebrates and robots in order to study or control animal behavior.     

Neural mechanisms of social learning and decision-making

One of the hallmarks of human society is the ubiquitous interactions among individuals. Indeed, a significant portion of human daily routine decision making is socially related. Normative economic theory, namely game theory, has prescribed the canonical decision strategy when "rational" social agents have full information about the decision environment. In reality, however, social decision is often influenced by the trait and state parameters of selves and others. Therefore, understanding the cognitive and neural processes of inferring the decision parameters is pivotal for social decision making. Recently, both correlational and causal non-invasive neuroimaging studies have started to reveal the critical neural computations underlying social learning and decision-making, and highlighted the unique roles of "social" brain structures such as temporal-parietal junction(TPJ) and dorsomedial prefrontal cortex(dmPFC). Here we review recent advances in social decision neuroscience and maintain the focus on how the inference about others is dynamically acquired during social learning, as well as how the prosocial(altruistic)behavior results from orchestrated interactions of different brain regions specified under the social utility framework. We conclude by emphasizing the importance of combining computational decision theory with the identification of neural mechanisms that represent, evaluate and integrate value related social information and generate decision variables guiding behavioral output in the complex social environment.     

Robustness of learning that is based on covariance-driven synaptic plasticity

It is widely believed that learning is due, at least in part, to long-lasting modifications of the strengths of synapses in the brain. Theoretical studies have shown that a family of synaptic plasticity rules, in which synaptic changes are driven by covariance, is particularly useful for many forms of learning, including associative memory, gradient estimation, and operant conditioning. Covariance-based plasticity is inherently sensitive. Even a slight mistuning of the parameters of a covariance-based plasticity rule is likely to result in substantial changes in synaptic efficacies. Therefore, the biological relevance of covariance-based plasticity models is questionable. Here, we study the effects of mistuning parameters of the plasticity rule in a decision making model in which synaptic plasticity is driven by the covariance of reward and neural activity. An exact covariance plasticity rule yields Herrnstein's matching law. We show that although the effect of slight mistuning of the plasticity rule on the synaptic efficacies is large, the behavioral effect is small. Thus, matching behavior is robust to mistuning of the parameters of the covariance-based plasticity rule. Furthermore, the mistuned covariance rule results in undermatching, which is consistent with experimentally observed behavior. These results substantiate the hypothesis that approximate covariance-based synaptic plasticity underlies operant conditioning. However, we show that the mistuning of the mean subtraction makes behavior sensitive to the mistuning of the properties of the decision making network. Thus, there is a tradeoff between the robustness of matching behavior to changes in the plasticity rule and its robustness to changes in the properties of the decision making network.     

Sensorimotor Learning Biases Choice Behavior: A Learning Neural Field Model for Decision Making

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject''s learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.     

Probabilistic decision making by slow reverberation in cortical circuits

Recent physiological studies of alert primates have revealed cortical neural correlates of key steps in a perceptual decision-making process. To elucidate synaptic mechanisms of decision making, I investigated a biophysically realistic cortical network model for a visual discrimination experiment. In the model, slow recurrent excitation and feedback inhibition produce attractor dynamics that amplify the difference between conflicting inputs and generates a binary choice. The model is shown to account for salient characteristics of the observed decision-correlated neural activity, as well as the animal's psychometric function and reaction times. These results suggest that recurrent excitation mediated by NMDA receptors provides a candidate cellular mechanism for the slow time integration of sensory stimuli and the formation of categorical choices in a decision-making neocortical network.     

Accuracy and response-time distributions for decision-making: linear perfect integrators versus nonlinear attractor-based neural circuits

Animals choose actions based on imperfect, ambiguous data. “Noise” inherent in neural processing adds further variability to this already-noisy input signal. Mathematical analysis has suggested that the optimal apparatus (in terms of the speed/accuracy trade-off) for reaching decisions about such noisy inputs is perfect accumulation of the inputs by a temporal integrator. Thus, most highly cited models of neural circuitry underlying decision-making have been instantiations of a perfect integrator. Here, in accordance with a growing mathematical and empirical literature, we describe circumstances in which perfect integration is rendered suboptimal. In particular we highlight the impact of three biological constraints: (1) significant noise arising within the decision-making circuitry itself; (2) bounding of integration by maximal neural firing rates; and (3) time limitations on making a decision. Under conditions (1) and (2), an attractor system with stable attractor states can easily best an integrator when accuracy is more important than speed. Moreover, under conditions in which such stable attractor networks do not best the perfect integrator, a system with unstable initial states can do so if readout of the system’s final state is imperfect. Ubiquitously, an attractor system with a nonselective time-dependent input current is both more accurate and more robust to imprecise tuning of parameters than an integrator with such input. Given that neural responses that switch stochastically between discrete states can “masquerade” as integration in single-neuron and trial-averaged data, our results suggest that such networks should be considered as plausible alternatives to the integrator model.     

The neural basis of financial risk taking

Investors systematically deviate from rationality when making financial decisions, yet the mechanisms responsible for these deviations have not been identified. Using event-related fMRI, we examined whether anticipatory neural activity would predict optimal and suboptimal choices in a financial decision-making task. We characterized two types of deviations from the optimal investment strategy of a rational risk-neutral agent as risk-seeking mistakes and risk-aversion mistakes. Nucleus accumbens activation preceded risky choices as well as risk-seeking mistakes, while anterior insula activation preceded riskless choices as well as risk-aversion mistakes. These findings suggest that distinct neural circuits linked to anticipatory affect promote different types of financial choices and indicate that excessive activation of these circuits may lead to investing mistakes. Thus, consideration of anticipatory neural mechanisms may add predictive power to the rational actor model of economic decision making.     

Music and Video Gaming during Breaks: Influence on Habitual versus Goal-Directed Decision Making

Different systems for habitual versus goal-directed control are thought to underlie human decision-making. Working memory is known to shape these decision-making systems and their interplay, and is known to support goal-directed decision making even under stress. Here, we investigated if and how decision systems are differentially influenced by breaks filled with diverse everyday life activities known to modulate working memory performance. We used a within-subject design where young adults listened to music and played a video game during breaks interleaved with trials of a sequential two-step Markov decision task, designed to assess habitual as well as goal-directed decision making. Based on a neurocomputational model of task performance, we observed that for individuals with a rather limited working memory capacity video gaming as compared to music reduced reliance on the goal-directed decision-making system, while a rather large working memory capacity prevented such a decline. Our findings suggest differential effects of everyday activities on key decision-making processes.     

Performance-gated deliberation: A context-adapted strategy in which urgency is opportunity cost

Finding the right amount of deliberation, between insufficient and excessive, is a hard decision making problem that depends on the value we place on our time. Average-reward, putatively encoded by tonic dopamine, serves in existing reinforcement learning theory as the opportunity cost of time, including deliberation time. Importantly, this cost can itself vary with the environmental context and is not trivial to estimate. Here, we propose how the opportunity cost of deliberation can be estimated adaptively on multiple timescales to account for non-stationary contextual factors. We use it in a simple decision-making heuristic based on average-reward reinforcement learning (AR-RL) that we call Performance-Gated Deliberation (PGD). We propose PGD as a strategy used by animals wherein deliberation cost is implemented directly as urgency, a previously characterized neural signal effectively controlling the speed of the decision-making process. We show PGD outperforms AR-RL solutions in explaining behaviour and urgency of non-human primates in a context-varying random walk prediction task and is consistent with relative performance and urgency in a context-varying random dot motion task. We make readily testable predictions for both neural activity and behaviour.     

Majority-rule opinion dynamics with differential latency: a mechanism for self-organized collective decision-making

Collective decision-making is a process whereby the members of a group decide on a course of action by consensus. In this paper, we propose a collective decision-making mechanism for robot swarms deployed in scenarios in which robots can choose between two actions that have the same effects but that have different execution times. The proposed mechanism allows a swarm composed of robots with no explicit knowledge about the difference in execution times between the two actions to choose the one with the shorter execution time. We use an opinion formation model that captures important elements of the scenarios in which the proposed mechanism can be used in order to predict the system??s behavior. The model predicts that when the two actions have different average execution times, the swarm chooses with high probability the action with the shorter average execution time. We validate the model??s predictions through a swarm robotics experiment in which robot teams must choose one of two paths of different length that connect two locations. Thanks to the proposed mechanism, a swarm made of robot teams that do not measure time or distance is able to choose the shorter path.     

Ballast Blockade: Stopping Aquatic Immigrants

This article emphasizes not only an important environmental issue for the Great Lakes but also the importance of decision-making skills in scientific thinking. The activity allows students to acquaint themselves with current Great Lakes topics while simultaneously partaking in decision-making processes that could affect them. As students work through the outlined steps of making a decision, they also participate in cooperative learning, scientific reasoning, and interdisciplinary processing. Students are presented with six ballast water treatment methods that they must evaluate on the basis of provided criteria. They assess the treatments using the decision-making skills of rating, weighing, and discussing.     

Humans and insects decide in similar ways

Behavioral ecologists assume that animals use a motivational mechanism for decisions such as action selection and time allocation, allowing the maximization of their fitness. They consider both the proximate and ultimate causes of behavior in order to understand this type of decision-making in animals. Experimental psychologists and neuroeconomists also study how agents make decisions but they consider the proximate causes of the behavior. In the case of patch-leaving, motivation-based decision-making remains simple speculation. In contrast to other animals, human beings can assess and evaluate their own motivation by an introspection process. It is then possible to study the declared motivation of humans during decision-making and discuss the mechanism used as well as its evolutionary significance. In this study, we combine both the proximate and ultimate causes of behavior for a better understanding of the human decision-making process. We show for the first time ever that human subjects use a motivational mechanism similar to small insects such as parasitoids and bumblebees to decide when to leave a patch. This result is relevant for behavioral ecologists as it supports the biological realism of this mechanism. Humans seem to use a motivational mechanism of decision making known to be adaptive to a heterogeneously distributed resource. As hypothesized by Hutchinson et al. and Wilke and Todd, our results are consistent with the evolutionary shaping of decision making because hominoids were hunters and gatherers on food patches for more than two million years. We discuss the plausibility of a neural basis for the motivation mechanism highlighted here, bridging the gap between behavioral ecology and neuroeconomy. Thus, both the motivational mechanism observed here and the neuroeconomy findings are most likely adaptations that were selected for during ancestral times.     

Dynamic Excitatory and Inhibitory Gain Modulation Can Produce Flexible,Robust and Optimal Decision-making

Behavioural and neurophysiological studies in primates have increasingly shown the involvement of urgency signals during the temporal integration of sensory evidence in perceptual decision-making. Neuronal correlates of such signals have been found in the parietal cortex, and in separate studies, demonstrated attention-induced gain modulation of both excitatory and inhibitory neurons. Although previous computational models of decision-making have incorporated gain modulation, their abstract forms do not permit an understanding of the contribution of inhibitory gain modulation. Thus, the effects of co-modulating both excitatory and inhibitory neuronal gains on decision-making dynamics and behavioural performance remain unclear. In this work, we incorporate time-dependent co-modulation of the gains of both excitatory and inhibitory neurons into our previous biologically based decision circuit model. We base our computational study in the context of two classic motion-discrimination tasks performed in animals. Our model shows that by simultaneously increasing the gains of both excitatory and inhibitory neurons, a variety of the observed dynamic neuronal firing activities can be replicated. In particular, the model can exhibit winner-take-all decision-making behaviour with higher firing rates and within a significantly more robust model parameter range. It also exhibits short-tailed reaction time distributions even when operating near a dynamical bifurcation point. The model further shows that neuronal gain modulation can compensate for weaker recurrent excitation in a decision neural circuit, and support decision formation and storage. Higher neuronal gain is also suggested in the more cognitively demanding reaction time than in the fixed delay version of the task. Using the exact temporal delays from the animal experiments, fast recruitment of gain co-modulation is shown to maximize reward rate, with a timescale that is surprisingly near the experimentally fitted value. Our work provides insights into the simultaneous and rapid modulation of excitatory and inhibitory neuronal gains, which enables flexible, robust, and optimal decision-making.     

Robot control and the evolution of modular neurodynamics

Summary A modular approach to neural behavior control of autonomous robots is presented. It is based on the assumption that complex internal dynamics of recurrent neural networks can efficiently solve complex behavior tasks. For the development of appropriate neural control structures an evolutionary algorithm is introduced, which is able to generate neuromodules with specific functional properties, as well as the connectivity structure for a modular synthesis of such modules. This so called ENS ³-algorithm does not use genetic coding. It is primarily designed to develop size and connectivity structure of neuro-controllers. But at the same time it optimizes also parameters of individual networks like synaptic weights and bias terms. For demonstration, evolved networks for the control of miniature Khepera robots are presented. The aim is to develop robust controllers in the sense that neuro-controllers evolved in a simulator show comparably good behavior when loaded to a real robot acting in a physical environment. Discussed examples of such controllers generate obstacle avoidance and phototropic behaviors in non-trivial environments.     

On the Rationality of Decision-Making Studies: Part 2: Divergent Rationalities

A study of treatment decision making in an Anishinaabe community in Manitoba, Canada was designed to be comparable with an earlier project carried out in a Mexican town. One objective was to compare the resulting decision models. For both communities, a decision-making perspective was compatible with how individuals talked about actions taken in response to illness, and it proved to be a useful means for learning about the process of seeking care. At the same time, a decision-modeling approach is better suited to explaining treatment actions taken in the Mexican community than in the Anishinaabe community. I suggest that this finding reflects the variable potentiality, in the Anishinaabe community, for affliction and its treatment to be constructed within a cultural framework in which the underlying assumptions differ from those implicit in studies of decision modeling, [care seeking, decision models, decision making, Anishinaabe]     

Structure learning in human sequential decision-making

Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner.     

A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales

Volitional behavior relies on the brain's ability to remap sensory flow to motor programs whenever demanded by a changed behavioral context. To investigate the circuit basis of such flexible behavior, we have developed a biophysically based decision-making network model of spiking neurons for arbitrary sensorimotor mapping. The model quantitatively reproduces behavioral and prefrontal single-cell data from an experiment in which monkeys learn visuomotor associations that are reversed unpredictably from time to time. We show that when synaptic modifications occur on multiple timescales, the model behavior becomes flexible only when needed: slow components of learning usually dominate the decision process. However, if behavioral contexts change frequently enough, fast components of plasticity take over, and the behavior exhibits a quick forget-and-learn pattern. This model prediction is confirmed by monkey data. Therefore, our work reveals a scenario for conditional associative learning that is distinct from instant switching between sets of well-established sensorimotor associations.