Sensorimotor Learning Biases Choice Behavior: A Learning Neural Field Model for Decision Making

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject''s learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.     

A molecular diffusion based utility model for Drosophila larval phototaxis

Background

Generally, utility based decision making models focus on experimental outcomes. In this paper we propose a utility model based on molecular diffusion to simulate the choice behavior of Drosophila larvae exposed to different light conditions.

Methods

In this paper, light/dark choice-based Drosophila larval phototaxis is analyzed with our molecular diffusion based model. An ISCEM algorithm is developed to estimate the model parameters.

Results

By applying this behavioral utility model to light intensity and phototaxis data, we show that this model fits the experimental data very well.

Conclusions

Our model provides new insights into decision making mechanisms in general. From an engineering viewpoint, we propose that the model could be applied to a wider range of decision making practices.     

Lateralized Readiness Potentials Reveal Properties of a Neural Mechanism for Implementing a Decision Threshold

Many perceptual decision making models posit that participants accumulate noisy evidence over time to improve the accuracy of their decisions, and that in free response tasks, participants respond when the accumulated evidence reaches a decision threshold. Research on the neural correlates of these models'' components focuses primarily on evidence accumulation. Far less attention has been paid to the neural correlates of decision thresholds, reflecting the final commitment to a decision. Inspired by a model of bistable neural activity that implements a decision threshold, we reinterpret human lateralized readiness potentials (LRPs) as reflecting the crossing of a decision threshold. Interestingly, this threshold crossing preserves signatures of a drift-diffusion process of evidence accumulation that feeds in to the threshold mechanism. We show that, as our model predicts, LRP amplitudes and growth rates recorded while participants performed a motion discrimination task correlate with individual differences in behaviorally-estimated prior beliefs, decision thresholds and evidence accumulation rates. As such LRPs provide a useful measure to test dynamical models of both evidence accumulation and decision commitment processes non-invasively.     

Thermodynamic view on decision-making process: emotions as a potential power vector of realization of the choice

This research is devoted to possible mechanisms of decision-making in frames of thermodynamic principles. It is also shown that the decision-making system in reply to emotion includes vector component which seems to be often a necessary condition to transfer system from one state to another. The phases of decision-making system can be described as supposed to be nonequilibrium and irreversible to which thermodynamics laws are applied. The mathematical model of a decision choice, proceeding from principles of the nonlinear dynamics considering instability of movement and bifurcation is offered. The thermodynamic component of decision-making process on the basis of vector transfer of energy induced by emotion at the given time is surveyed. It is proposed a three-modular model of decision making based on principles of thermodynamics. Here it is suggested that at entropy impact due to effect of emotion, on the closed system—the human brain,—initially arises chaos, then after fluctuations of possible alternatives which were going on—reactions of brain zones in reply to external influence, an order is forming and there is choice of alternatives, according to primary entrance conditions and a state of the closed system. Entropy calculation of a choice expectation of negative and positive emotion shows judgment possibility of existence of “the law of emotion conservation” in accordance with several experimental data.     

Action and Valence Modulate Choice and Choice-Induced Preference Change

Choices are not only communicated via explicit actions but also passively through inaction. In this study we investigated how active or passive choice impacts upon the choice process itself as well as a preference change induced by choice. Subjects were tasked to select a preference for unfamiliar photographs by action or inaction, before and after they gave valuation ratings for all photographs. We replicate a finding that valuation increases for chosen items and decreases for unchosen items compared to a control condition in which the choice was made post re-evaluation. Whether choice was expressed actively or passively affected the dynamics of revaluation differently for positive and negatively valenced items. Additionally, the choice itself was biased towards action such that subjects tended to choose a photograph obtained by action more often than a photographed obtained through inaction. These results highlight intrinsic biases consistent with a tight coupling of action and reward and add to an emerging understanding of how the mode of action itself, and not just an associated outcome, modulates the decision making process.     

Dancing for a decision: a matrix model for nest-site choice by honeybees

A mathematical model is formulated for decision making by honeybees during nest-site choice, using a population matrix model. This model explains how the observed dynamics of the nest-site scouts' dancing can reliably lead to a choice of the best nest site available.     

Probabilistic decision making by slow reverberation in cortical circuits

Recent physiological studies of alert primates have revealed cortical neural correlates of key steps in a perceptual decision-making process. To elucidate synaptic mechanisms of decision making, I investigated a biophysically realistic cortical network model for a visual discrimination experiment. In the model, slow recurrent excitation and feedback inhibition produce attractor dynamics that amplify the difference between conflicting inputs and generates a binary choice. The model is shown to account for salient characteristics of the observed decision-correlated neural activity, as well as the animal's psychometric function and reaction times. These results suggest that recurrent excitation mediated by NMDA receptors provides a candidate cellular mechanism for the slow time integration of sensory stimuli and the formation of categorical choices in a decision-making neocortical network.     

Active inference and agency: optimal control without cost functions

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.     

The role of single-cell analyses in understanding cell lineage commitment

The study of cell lineage commitment is critical for improving our understanding of tissue development and regeneration, and for realizing stem cell-based therapies and engineered tissue replacements. Recently, the discovery of an unanticipated degree of variability in fundamental biological processes, including divergent responses of genetically identical cells to various stimuli, has provided mechanistic insight into cellular decision making and the collective behavior of cell populations. Therefore, the study of lineage commitment with single-cell resolution could provide greater knowledge of cellular differentiation mechanisms and the influence of noise on cellular processes. This will require the adoption of new technologies for single-cell analysis as traditional methods typically measure average values of bulk population behavior. This review discusses the recent developments in methods for analyzing the behavior of individual cells, and how these approaches are leading to a deeper understanding and better control of cellular decision making.     

The anatomy of choice: dopamine and decision-making

This paper considers goal-directed decision-making in terms of embodied or active inference. We associate bounded rationality with approximate Bayesian inference that optimizes a free energy bound on model evidence. Several constructs such as expected utility, exploration or novelty bonuses, softmax choice rules and optimism bias emerge as natural consequences of free energy minimization. Previous accounts of active inference have focused on predictive coding. In this paper, we consider variational Bayes as a scheme that the brain might use for approximate Bayesian inference. This scheme provides formal constraints on the computational anatomy of inference and action, which appear to be remarkably consistent with neuroanatomy. Active inference contextualizes optimal decision theory within embodied inference, where goals become prior beliefs. For example, expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature (associated with softmax functions and quantal response equilibria) has a unique and Bayes-optimal solution. Crucially, this sensitivity corresponds to the precision of beliefs about behaviour. The changes in precision during variational updates are remarkably reminiscent of empirical dopaminergic responses—and they may provide a new perspective on the role of dopamine in assimilating reward prediction errors to optimize decision-making.     

A gaze bias with coarse spatial indexing during a gambling task

Researchers have used eye-tracking methods to infer cognitive processes during decision making in choice tasks involving visual materials. Gaze likelihood analysis has shown a cascading effect, suggestive of a causal role for the gaze in preference formation during evaluative decision making. According to the gaze bias hypothesis, the gaze serves to build commitment gradually towards a choice. Here, we applied gaze likelihood analysis in a two-choice version of the well-known Iowa Gambling Task. This task requires active learning of the value of different choice options. As such, it does not involve visual preference formation, but choice optimization through learning. In Experiment 1 we asked subjects to choose between two decks with different payoff structures, and to give their responses using mouse clicks. Two groups of subjects were exposed to stable versus varying outcome contingencies. The analysis revealed a pronounced gaze bias towards the chosen stimuli in both groups of subjects, plateauing at more than 400 ms before the choice. The early plateauing suggested that the gaze effect partially reflected eye-hand coordination. In Experiment 2 we asked subjects to give responses using a key press. The results again showed a clear gaze bias towards the chosen deck, this time without any influence from eye-hand coordination. In both experiments, there was a clear gaze bias towards the choice even though the gaze fixations did not narrowly focus on the spatial positions of choice options. Taken together, the data suggested a role for gaze in coarse spatial indexing during non-perceptual decision making.     

Discrimination of Complex Human Behavior by Pigeons (Columba livia) and Humans

The cognitive and neural mechanisms for recognizing and categorizing behavior are not well understood in non-human animals. In the current experiments, pigeons and humans learned to categorize two non-repeating, complex human behaviors (“martial arts” vs. “Indian dance”). Using multiple video exemplars of a digital human model, pigeons discriminated these behaviors in a go/no-go task and humans in a choice task. Experiment 1 found that pigeons already experienced with discriminating the locomotive actions of digital animals acquired the discrimination more rapidly when action information was available than when only pose information was available. Experiments 2 and 3 found this same dynamic superiority effect with naïve pigeons and human participants. Both species used the same combination of immediately available static pose information and more slowly perceived dynamic action cues to discriminate the behavioral categories. Theories based on generalized visual mechanisms, as opposed to embodied, species-specific action networks, offer a parsimonious account of how these different animals recognize behavior across and within species.     

Proximate structural mechanisms for variation in food-chain length

Food-chain length is a central characteristic of ecological communities because of its strong influence on community structure and ecosystem function. While recent studies have started to better clarify the relationship between food-chain length and environmental gradients such as resource availability and ecosystem size, much less progress has been made in isolating the ultimate and proximate mechanisms that determine food-chain length. Progress has been slow, in part, because research has paid little attention to the proximate changes in food web structure that must link variation in food-chain length to the ultimate dynamic mechanism. Here we outline the structural mechanisms that determine variation in food-chain length. We explore the implications of these mechanisms for understanding how changes in food-web structure influence food-chain length using both an intraguild predation community model and data from natural ecosystems. The resulting framework provides the mechanisms for linking ultimate dynamic mechanisms to variation in food-chain length. It also suggests that simple linear food-chain models may make misleading predictions about patterns of variation in food-chain length because they are unable to incorporate important structural mechanisms that alter food-web dynamics and cause non-linear shifts in food-web structure. Intraguild predation models provide a more appropriate theoretical framework for understanding food-chain length in most natural ecosystems because they accommodate all of the proximate structural mechanisms identified here.     

Spatial learning and action planning in a prefrontal cortical network model

The interplay between hippocampus and prefrontal cortex (PFC) is fundamental to spatial cognition. Complementing hippocampal place coding, prefrontal representations provide more abstract and hierarchically organized memories suitable for decision making. We model a prefrontal network mediating distributed information processing for spatial learning and action planning. Specific connectivity and synaptic adaptation principles shape the recurrent dynamics of the network arranged in cortical minicolumns. We show how the PFC columnar organization is suitable for learning sparse topological-metrical representations from redundant hippocampal inputs. The recurrent nature of the network supports multilevel spatial processing, allowing structural features of the environment to be encoded. An activation diffusion mechanism spreads the neural activity through the column population leading to trajectory planning. The model provides a functional framework for interpreting the activity of PFC neurons recorded during navigation tasks. We illustrate the link from single unit activity to behavioral responses. The results suggest plausible neural mechanisms subserving the cognitive "insight" capability originally attributed to rodents by Tolman & Honzik. Our time course analysis of neural responses shows how the interaction between hippocampus and PFC can yield the encoding of manifold information pertinent to spatial planning, including prospective coding and distance-to-goal correlates.     

The equilibrium assumption in estimating the parameters of metapopulation models

1.  The construction of a predictive metapopulation model includes three steps: the choice of factors affecting metapopulation dynamics, the choice of model structure, and finally parameter estimation and model testing.
2.  Unless the assumption is made that the metapopulation is at stochastic quasi-equilibrium and unless the method of parameter estimation of model parameters uses that assumption, estimates from a limited amount of data will usually predict a trend in metapopulation size.
3.  This implicit estimation of a trend occurs because extinction-colonization stochasticity, possibly amplified by regional stochasticity, leads to unequal numbers of observed extinction and colonization events during a short study period.
4.  Metapopulation models, such as those based on the logistic regression model, that rely on observed population turnover events in parameter estimation are sensitive to the implicit estimation of a trend.
5.  A new parameter estimation method, based on Monte Carlo inference for statistically implicit models, allows an explicit decision about whether metapopulation quasi-stability is assumed or not.
6. Our confidence in metapopulation model parameter estimates that have been produced from only a few years of data is decreased by the need to know before parameter estimation whether the metapopulation is in quasi-stable state or not.
7. The choice of whether metapopulation stability is assumed or not in parameter estimation should be done consciously. Typical data sets cover only a few years and rarely allow a statistical test of a possible trend. While making the decision about stability one should consider any information about the landscape history and species and metapopulation characteristics.     

Conflict and competition between model-based and model-free control

A large literature has accumulated suggesting that human and animal decision making is driven by at least two systems, and that important functions of these systems can be captured by reinforcement learning algorithms. The “model-free” system caches and uses stimulus–value or stimulus–response associations, and the “model-based” system implements more flexible planning using a model of the world. However, it is not clear how the two systems interact during deliberation and how a single decision emerges from this process, especially when they disagree. Most previous work has assumed that while the systems operate in parallel, they do so independently, and they combine linearly to influence decisions. Using an integrated reinforcement learning/drift-diffusion model, we tested the hypothesis that the two systems interact in a non-linear fashion similar to other situations with cognitive conflict. We differentiated two forms of conflict: action conflict, a binary state representing whether the systems disagreed on the best action, and value conflict, a continuous measure of the extent to which the two systems disagreed on the difference in value between the available options. We found that decisions with greater value conflict were characterized by reduced model-based control and increased caution both with and without action conflict. Action conflict itself (the binary state) acted in the opposite direction, although its effects were less prominent. We also found that between-system conflict was highly correlated with within-system conflict, and although it is less clear a priori why the latter might influence the strength of each system above its standard linear contribution, we could not rule it out. Our work highlights the importance of non-linear conflict effects, and provides new constraints for more detailed process models of decision making. It also presents new avenues to explore with relation to disorders of compulsivity, where an imbalance between systems has been implicated.     

Sharing death and dying: advance directives, autonomy and the family

This paper critically examines the liberal model of decision making for the terminally ill and contrasts it with the familial model that can be found in some Asian cultures. The contrast between the two models shows that the liberal model is excessively patient-centered, and misconceives and marginalises the role of the family in the decision making process. The paper argues that the familial model is correct in conceiving the last journey of one's life as a sharing process rather than a process of exercising one's prior or counterfactual choice, and concludes by suggesting a policy framework for the practice of familialism that can answer the liberal challenge that familialism cannot safeguard the patient from abuse and neglect.