Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system

The prefrontal cortex (PFC) has long been thought to serve as an 'executive' that controls the selection of actions and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified often amounting to a homunculus. This paper reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a 'Go' versus 'NoGo' modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.     

Behavioral planning in the prefrontal cortex

Recent studies have presented evidence that the prefrontal cortex plays a crucial role in every aspect of the cognitive processes necessary for behavioral planning: processing and integration of perceived or memorized information, associative learning, reward-based behavioral control, behavioral selection/decision-making and behavioral guidance. We propose that the creation of novel information is the means by which the prefrontal cortex operates to achieve executive control over behavioral planning. The prefrontal cortex is the site of operation of nodal points, where neural circuits integrate currently available or memorized information to generate the information that is necessary to perform an action. The prefrontal cortex also regulates the flow of information through multiple nodes to meet behavioral demands.     

An evolutionary computational theory of prefrontal executive function in decision-making

The prefrontal cortex subserves executive control and decision-making, that is, the coordination and selection of thoughts and actions in the service of adaptive behaviour. We present here a computational theory describing the evolution of the prefrontal cortex from rodents to humans as gradually adding new inferential Bayesian capabilities for dealing with a computationally intractable decision problem: exploring and learning new behavioural strategies versus exploiting and adjusting previously learned ones through reinforcement learning (RL). We provide a principled account identifying three inferential steps optimizing this arbitration through the emergence of (i) factual reactive inferences in paralimbic prefrontal regions in rodents; (ii) factual proactive inferences in lateral prefrontal regions in primates and (iii) counterfactual reactive and proactive inferences in human frontopolar regions. The theory clarifies the integration of model-free and model-based RL through the notion of strategy creation. The theory also shows that counterfactual inferences in humans yield to the notion of hypothesis testing, a critical reasoning ability for approximating optimal adaptive processes and presumably endowing humans with a qualitative evolutionary advantage in adaptive behaviour.     

Frontal cortex and reward-guided learning and decision-making

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.     

Path planning versus cue responding: a bio-inspired model of switching between navigation strategies

In this article, we describe a new computational model of switching between path-planning and cue-guided navigation strategies. It is based on three main assumptions: (i) the strategies are mediated by separate memory systems that learn independently and in parallel; (ii) the learning algorithms are different in the two memory systems—the cue-guided strategy uses a temporal-difference (TD) learning rule to approach a visible goal, whereas the path-planning strategy relies on a place-cell-based graph-search algorithm to learn the location of a hidden goal; (iii) a strategy selection mechanism uses TD-learning rule to choose the most successful strategy based on past experience. We propose a novel criterion for strategy selection based on the directions of goal-oriented movements suggested by the different strategies. We show that the selection criterion based on this “common currency” is capable of choosing the best among TD-learning and planning strategies and can be used to solve navigational tasks in continuous state and action spaces. The model has been successfully applied to reproduce rat behavior in two water-maze tasks in which the two strategies were shown to interact. The model was used to analyze competitive and cooperative interactions between different strategies during these tasks as well as relative influence of different types of sensory cues.     

When in Rome,do as the Romans do: the coevolution of altruistic punishment,conformist learning,and cooperation

We model the coevolution of behavioral strategies and social learning rules in the context of a cooperative dilemma, a situation in which individuals must decide whether or not to subordinate their own interests to those of the group. There are two learning rules in our model, conformism and payoff-dependent imitation, which evolve by natural selection, and three behavioral strategies, cooperate, defect, and cooperate, plus punish defectors, which evolve under the influence of the prevailing learning rules. Group and individual level selective pressures drive evolution.We also simulate our model for conditions that approximate those in which early hominids lived. We find that conformism can evolve when the only problem that individuals face is a cooperative dilemma, in which prosocial behavior is always costly to the individual. Furthermore, the presence of conformists dramatically increases the group size for which cooperation can be sustained. The results of our model are robust: they hold even when migration rates are high, and when conflict among groups is infrequent.     

Frontal Lobe Contusion in Mice Chronically Impairs Prefrontal-Dependent Behavior

Traumatic brain injury (TBI) is a major cause of chronic disability in the world. Moderate to severe TBI often results in damage to the frontal lobe region and leads to cognitive, emotional, and social behavioral sequelae that negatively affect quality of life. More specifically, TBI patients often develop persistent deficits in social behavior, anxiety, and executive functions such as attention, mental flexibility, and task switching. These deficits are intrinsically associated with prefrontal cortex (PFC) functionality. Currently, there is a lack of analogous, behaviorally characterized TBI models for investigating frontal lobe injuries despite the prevalence of focal contusions to the frontal lobe in TBI patients. We used the controlled cortical impact (CCI) model in mice to generate a frontal lobe contusion and studied behavioral changes associated with PFC function. We found that unilateral frontal lobe contusion in mice produced long-term impairments to social recognition and reversal learning while having only a minor effect on anxiety and completely sparing rule shifting and hippocampal-dependent behavior.     

The Anterior Insula Tracks Behavioral Entropy during an Interpersonal Competitive Game

In competitive situations, individuals need to adjust their behavioral strategy dynamically in response to their opponent’s behavior. In the present study, we investigated the neural basis of how individuals adjust their strategy during a simple, competitive game of matching pennies. We used entropy as a behavioral index of randomness in decision-making, because maximizing randomness is thought to be an optimal strategy in the game, according to game theory. While undergoing functional magnetic resonance imaging (fMRI), subjects played matching pennies with either a human or computer opponent in each block, although in reality they played the game with the same computer algorithm under both conditions. The winning rate of each block was also manipulated. Both the opponent (human or computer), and the winning rate, independently affected subjects’ block-wise entropy during the game. The fMRI results revealed that activity in the bilateral anterior insula was positively correlated with subjects’ (not opponent’s) behavioral entropy during the game, which indicates that during an interpersonal competitive game, the anterior insula tracked how uncertain subjects’ behavior was, rather than how uncertain subjects felt their opponent''s behavior was. Our results suggest that intuitive or automatic processes based on somatic markers may be a key to optimally adjusting behavioral strategies in competitive situations.     

The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning

While there is no doubt that social signals affect human reinforcement learning, there is still no consensus about how this process is computationally implemented. To address this issue, we compared three psychologically plausible hypotheses about the algorithmic implementation of imitation in reinforcement learning. The first hypothesis, decision biasing (DB), postulates that imitation consists in transiently biasing the learner’s action selection without affecting their value function. According to the second hypothesis, model-based imitation (MB), the learner infers the demonstrator’s value function through inverse reinforcement learning and uses it to bias action selection. Finally, according to the third hypothesis, value shaping (VS), the demonstrator’s actions directly affect the learner’s value function. We tested these three hypotheses in 2 experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task. We show through model comparison and model simulation that VS provides the best explanation of learner’s behavior. Results replicated in a third independent experiment featuring a larger cohort and a different design (N = 302). In our experiments, we also manipulated the quality of the demonstrators’ choices and found that learners were able to adapt their imitation rate, so that only skilled demonstrators were imitated. We proposed and tested an efficient meta-learning process to account for this effect, where imitation is regulated by the agreement between the learner and the demonstrator. In sum, our findings provide new insights and perspectives on the computational mechanisms underlying adaptive imitation in human reinforcement learning.

This study investigates imitation from a computational perspective; three experiments show that, in the context of reinforcement learning, imitation operates via a durable modification of the learner''s values, shedding new light on how imitation is computationally implemented and shapes learning and decision-making.     

Neural Correlates of Successful and Unsuccessful Strategical Mechanisms Involved in Uncertain Decision-Making

The ability to develop successful long-term strategies in uncertain situations relies on complex neural mechanisms. Although lesion studies have shown some of the mechanisms involved, it is still unknown why some healthy subjects are able to make the right decision whereas others are not. The aim of our study was to investigate neurophysiological differences underlying this ability to develop a successful strategy in a group of healthy subjects playing a monetary card game called the Iowa Gambling Task (IGT). In this task, subjects have to win and earn money by choosing between four decks of cards, two were advantageous in the long term and two disadvantageous. Twenty healthy right-handed subjects performed the IGT while their cerebral activity was recorded by electroencephalography. Based on their behavioral performances, two groups of subjects could clearly be distinguished: one who selected the good decks and thus succeeded in developing a Favorable strategy (9 subjects) and one who remained Undecided (11 subjects). No neural difference was found between each group before the selection of a deck, but in both groups a greater negativity was found emerging from the right superior frontal gyrus 600 ms before a disadvantageous selection. During the processing of the feedback, an attenuation of the P200 and P300 waveforms was found for the Undecided group, and a P300 originating from the medial frontal gyrus was found in response to a loss only in the Favorable group. Our results suggest that undecided subjects are hyposensitive to the valence of the cards during gambling, which affects the feedback processing.     

Operant Procedures for Assessing Behavioral Flexibility in Rats

Executive functions consist of multiple high-level cognitive processes that drive rule generation and behavioral selection. An emergent property of these processes is the ability to adjust behavior in response to changes in one’s environment (i.e., behavioral flexibility). These processes are essential to normal human behavior, and may be disrupted in diverse neuropsychiatric conditions, including schizophrenia, alcoholism, depression, stroke, and Alzheimer’s disease. Understanding of the neurobiology of executive functions has been greatly advanced by the availability of animal tasks for assessing discrete components of behavioral flexibility, particularly strategy shifting and reversal learning. While several types of tasks have been developed, most are non-automated, labor intensive, and allow testing of only one animal at a time. The recent development of automated, operant-based tasks for assessing behavioral flexibility streamlines testing, standardizes stimulus presentation and data recording, and dramatically improves throughput. Here, we describe automated strategy shifting and reversal tasks, using operant chambers controlled by custom written software programs. Using these tasks, we have shown that the medial prefrontal cortex governs strategy shifting but not reversal learning in the rat, similar to the dissociation observed in humans. Moreover, animals with a neonatal hippocampal lesion, a neurodevelopmental model of schizophrenia, are selectively impaired on the strategy shifting task but not the reversal task. The strategy shifting task also allows the identification of separate types of performance errors, each of which is attributable to distinct neural substrates. The availability of these automated tasks, and the evidence supporting the dissociable contributions of separate prefrontal areas, makes them particularly well-suited assays for the investigation of basic neurobiological processes as well as drug discovery and screening in disease models.     

Probability Matching as a Computational Strategy Used in Perception

The question of which strategy is employed in human decision making has been studied extensively in the context of cognitive tasks; however, this question has not been investigated systematically in the context of perceptual tasks. The goal of this study was to gain insight into the decision-making strategy used by human observers in a low-level perceptual task. Data from more than 100 individuals who participated in an auditory-visual spatial localization task was evaluated to examine which of three plausible strategies could account for each observer''s behavior the best. This task is very suitable for exploring this question because it involves an implicit inference about whether the auditory and visual stimuli were caused by the same object or independent objects, and provides different strategies of how using the inference about causes can lead to distinctly different spatial estimates and response patterns. For example, employing the commonly used cost function of minimizing the mean squared error of spatial estimates would result in a weighted averaging of estimates corresponding to different causal structures. A strategy that would minimize the error in the inferred causal structure would result in the selection of the most likely causal structure and sticking with it in the subsequent inference of location—“model selection.” A third strategy is one that selects a causal structure in proportion to its probability, thus attempting to match the probability of the inferred causal structure. This type of probability matching strategy has been reported to be used by participants predominantly in cognitive tasks. Comparing these three strategies, the behavior of the vast majority of observers in this perceptual task was most consistent with probability matching. While this appears to be a suboptimal strategy and hence a surprising choice for the perceptual system to adopt, we discuss potential advantages of such a strategy for perception.     

Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum

Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the “win-stay, lose-switch” strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.     

Understanding decision-making deficits in neurological conditions: insights from models of natural action selection

Models of natural action selection implicate fronto-striatal circuits in both motor and cognitive 'actions'. Dysfunction of these circuits leads to decision-making deficits in various populations. We review how computational models provide insights into the mechanistic basis for these deficits in Parkinson's patients and those with ventromedial frontal damage. We then consider implications of the models for understanding behaviour and cognition in attention-deficit/hyperactivity disorder (ADHD). Incorporation of cortical noradrenaline function into the model improves action selection in noisy environments and accounts for response variability in ADHD. We close with more general clinical implications.     

Elucidating Poor Decision-Making in a Rat Gambling Task

Although poor decision-making is a hallmark of psychiatric conditions such as attention deficit/hyperactivity disorder, pathological gambling or substance abuse, a fraction of healthy individuals exhibit similar poor decision-making performances in everyday life and specific laboratory tasks such as the Iowa Gambling Task. These particular individuals may provide information on risk factors or common endophenotypes of these mental disorders. In a rodent version of the Iowa gambling task – the Rat Gambling Task (RGT), we identified a population of poor decision makers, and assessed how these rats scored for several behavioral traits relevant to executive disorders: risk taking, reward seeking, behavioral inflexibility, and several aspects of impulsivity. First, we found that poor decision-making could not be well predicted by single behavioral and cognitive characteristics when considered separately. By contrast, a combination of independent traits in the same individual, namely risk taking, reward seeking, behavioral inflexibility, as well as motor impulsivity, was highly predictive of poor decision-making. Second, using a reinforcement-learning model of the RGT, we confirmed that only the combination of extreme scores on these traits could induce maladaptive decision-making. Third, the model suggested that a combination of these behavioral traits results in an inaccurate representation of rewards and penalties and inefficient learning of the environment. Poor decision-making appears as a consequence of the over-valuation of high-reward-high-risk options in the task. Such a specific psychological profile could greatly impair clinically healthy individuals in decision-making tasks and may predispose to mental disorders with similar symptoms.     

Executive dysfunction and the relation with behavioral problems in children with 47,XXY and 47,XXX

Neuroimaging studies have shown that having an extra X chromosome is associated with abnormal structure and function of brain areas in the frontal lobe, which is crucially involved in executive functioning. However, there is little of knowledge of the type and severity of executive dysfunction, and the impact on emotional and behavioral problems. The present study aims to provide in this. In total, 40 children (23 boys with 47,XXY and 17 girls with 47,XXX) with an extra X chromosome and 100 non‐clinical controls (47 boys and 53 girls) participated in the study. The participants were 9–18 years old. Processing speed and executive functioning were assessed using the Amsterdam Neuropsychological Testbattery (ANT) and the Dysexecutive Questionnaire (DEX). Problems in emotional and behavioral functioning were assessed with the Childhood Behavior Checklist (CBCL). Children with an extra X chromosome showed deficits in inhibition, mental flexibility, sustained attention and visual working memory. Parental report showed high levels of everyday manifestations of executive dysfunction. More severe inhibition difficulties were associated with higher levels of thought problems, aggression and rule breaking behavior. Boys and girls with an extra X chromosome could not be differentiated based on severity of executive dysfunction, however, girls had lower information processing speed than boys. These findings suggest that executive dysfunction may be part of the phenotype of children with an extra X chromosome, impacting the ability to function adequately in everyday life. Furthermore, children with impairments in inhibition may have more problems in regulating their thinking, emotions and behavior.     

Cortical mechanisms for reinforcement learning in competitive games

Game theory analyses optimal strategies for multiple decision makers interacting in a social group. However, the behaviours of individual humans and animals often deviate systematically from the optimal strategies described by game theory. The behaviours of rhesus monkeys (Macaca mulatta) in simple zero-sum games showed similar patterns, but their departures from the optimal strategies were well accounted for by a simple reinforcement-learning algorithm. During a computer-simulated zero-sum game, neurons in the dorsolateral prefrontal cortex often encoded the previous choices of the animal and its opponent as well as the animal's reward history. By contrast, the neurons in the anterior cingulate cortex predominantly encoded the animal's reward history. Using simple competitive games, therefore, we have demonstrated functional specialization between different areas of the primate frontal cortex involved in outcome monitoring and action selection. Temporally extended signals related to the animal's previous choices might facilitate the association between choices and their delayed outcomes, whereas information about the choices of the opponent might be used to estimate the reward expected from a particular action. Finally, signals related to the reward history might be used to monitor the overall success of the animal's current decision-making strategy.     

Spike-based Decision Learning of Nash Equilibria in Two-Player Games

Humans and animals face decision tasks in an uncertain multi-agent environment where an agent''s strategy may change in time due to the co-adaptation of others strategies. The neuronal substrate and the computational algorithms underlying such adaptive decision making, however, is largely unknown. We propose a population coding model of spiking neurons with a policy gradient procedure that successfully acquires optimal strategies for classical game-theoretical tasks. The suggested population reinforcement learning reproduces data from human behavioral experiments for the blackjack and the inspector game. It performs optimally according to a pure (deterministic) and mixed (stochastic) Nash equilibrium, respectively. In contrast, temporal-difference(TD)-learning, covariance-learning, and basic reinforcement learning fail to perform optimally for the stochastic strategy. Spike-based population reinforcement learning, shown to follow the stochastic reward gradient, is therefore a viable candidate to explain automated decision learning of a Nash equilibrium in two-player games.     

Direct reciprocity between individuals that use different strategy spaces

In repeated interactions, players can use strategies that respond to the outcome of previous rounds. Much of the existing literature on direct reciprocity assumes that all competing individuals use the same strategy space. Here, we study both learning and evolutionary dynamics of players that differ in the strategy space they explore. We focus on the infinitely repeated donation game and compare three natural strategy spaces: memory-1 strategies, which consider the last moves of both players, reactive strategies, which respond to the last move of the co-player, and unconditional strategies. These three strategy spaces differ in the memory capacity that is needed. We compute the long term average payoff that is achieved in a pairwise learning process. We find that smaller strategy spaces can dominate larger ones. For weak selection, unconditional players dominate both reactive and memory-1 players. For intermediate selection, reactive players dominate memory-1 players. Only for strong selection and low cost-to-benefit ratio, memory-1 players dominate the others. We observe that the supergame between strategy spaces can be a social dilemma: maximum payoff is achieved if both players explore a larger strategy space, but smaller strategy spaces dominate.     

Majority-biased transmission in chimpanzees and human children, but not orangutans

Cultural transmission is a key component of human evolution. Two of humans' closest living relatives, chimpanzees and orangutans, have also been argued to transmit behavioral traditions across generations culturally [1-3], but how much the process might resemble the human process is still in large part unknown. One key phenomenon of human cultural transmission is majority-biased transmission: the increased likelihood for learners to end up not with the most frequent behavior but rather with the behavior demonstrated by most individuals. Here we show that chimpanzees and human children as young as 2 years of age, but not orangutans, are more likely to copy an action performed by three individuals, once each, than an action performed by one individual three times. The tendency to acquire the behaviors of the majority has been posited as key to the transmission of relatively safe, reliable, and productive behavioral strategies [4-7] but has not previously been demonstrated in primates.