Is there a brainstem substrate for action selection?

The search for the neural substrate of vertebrate action selection has focused on structures in the forebrain and midbrain, and particularly on the group of sub-cortical nuclei known as the basal ganglia. Yet, the behavioural repertoire of decerebrate and neonatal animals suggests the existence of a relatively self-contained neural substrate for action selection in the brainstem. We propose that the medial reticular formation (mRF) is the substrate's main component and review evidence showing that the mRF's inputs, outputs and intrinsic organization are consistent with the requirements of an action-selection system. The internal architecture of the mRF is composed of interconnected neuron clusters. We present an anatomical model which suggests that the mRF's intrinsic circuitry constitutes a small-world network and extend this result to show that it may have evolved to reduce axonal wiring. Potential configurations of action representation within the internal circuitry of the mRF are then assessed by computational modelling. We present new results demonstrating that each cluster's output is most likely to represent activation of a component action; thus, coactivation of a set of these clusters would lead to the coordinated behavioural response observed in the animal. Finally, we consider the potential integration of the basal ganglia and mRF substrates for selection and suggest that they may collectively form a layered/hierarchical control system.     

Multisensory Causal Inference in the Brain

At any given moment, our brain processes multiple inputs from its different sensory modalities (vision, hearing, touch, etc.). In deciphering this array of sensory information, the brain has to solve two problems: (1) which of the inputs originate from the same object and should be integrated and (2) for the sensations originating from the same object, how best to integrate them. Recent behavioural studies suggest that the human brain solves these problems using optimal probabilistic inference, known as Bayesian causal inference. However, how and where the underlying computations are carried out in the brain have remained unknown. By combining neuroimaging-based decoding techniques and computational modelling of behavioural data, a new study now sheds light on how multisensory causal inference maps onto specific brain areas. The results suggest that the complexity of neural computations increases along the visual hierarchy and link specific components of the causal inference process with specific visual and parietal regions.     

The neural systems that mediate human perceptual decision making

Perceptual decision making is the act of choosing one option or course of action from a set of alternatives on the basis of available sensory evidence. Thus, when we make such decisions, sensory information must be interpreted and translated into behaviour. Neurophysiological work in monkeys performing sensory discriminations, combined with computational modelling, has paved the way for neuroimaging studies that are aimed at understanding decision-related processes in the human brain. Here we review findings from human neuroimaging studies in conjunction with data analysis methods that can directly link decisions and signals in the human brain on a trial-by-trial basis. This leads to a new view about the neural basis of human perceptual decision-making processes.     

Integrative neuroscience: linking levels of analyses

If we are to understand how the brain performs different integrated functions in cellular terms, we need both to understand all relevant levels of analysis from the molecular to the behavioural and cognitive levels and to realize an integration of such levels. This is currently a major challenge for neuroscience. Most research, whether dealing with perception, action or learning, focuses on a few levels of organization, for instance the molecular level and brain imaging, and leaves other crucial areas practically untouched. To reach the level of understanding that we desire, a multi-level approach is required in which the different levels link into each other. It is possible to bridge across the different levels for one system, and this has been demonstrated, for example, in the lamprey in generation of goal-directed locomotion. It can be argued that an integrated analysis of any neural system cannot be performed without the aid of a close interaction between experiments and modelling. The dynamic processing within any neural system is such that an intuitive interpretation is rarely sufficient.     

On the mathematical modelling of pain

In this review a case is presented for the use of mathematical modelling in the study of pain. The philosophy of mathematical modelling is outlined and a recommendation is made for the use of modern nonlinear techniques and computational neuroscience in the modelling of pain. Classic and more recent examples of modelling in neurobiology in general and pain in particular, at three different levels—molecular, cellular and neural networks—are described and evaluated. Directions for further progress are indicated, particularly in plasticity and in modelling brain mechanisms. Major advantages of mathematical modelling are that it can handle extremely complex theories and it is non-invasive, and so is particularly valuable in the investigation of chronic pain. Special issue dedicated to Dr. Herman Bachelard     

Neural networks for perceptual processing: from simulation tools to theories

Neural networks are modelling tools that are, in principle, able to capture the input-output behaviour of arbitrary systems that may include the dynamics of animal populations or brain circuits. While a neural network model is useful if it captures phenomenologically the behaviour of the target system in this way, its utility is amplified if key mechanisms of the model can be discovered, and identified with those of the underlying system. In this review, we first describe, at a fairly high level with minimal mathematics, some of the tools used in constructing neural network models. We then go on to discuss the implications of network models for our understanding of the system they are supposed to describe, paying special attention to those models that deal with neural circuits and brain systems. We propose that neural nets are useful for brain modelling if they are viewed in a wider computational framework originally devised by Marr. Here, neural networks are viewed as an intermediate mechanistic abstraction between 'algorithm' and 'implementation', which can provide insights into biological neural representations and their putative supporting architectures.     

A computational model of action selection in the basal ganglia. I. A new functional anatomy

 We present a biologically plausible model of processing intrinsic to the basal ganglia based on the computational premise that action selection is a primary role of these central brain structures. By encoding the propensity for selecting a given action in a scalar value (the salience), it is shown that action selection may be re-cast in terms of signal selection. The generic properties of signal selection are defined and neural networks for this type of computation examined. A comparison between these networks and basal ganglia anatomy leads to a novel functional decomposition of the basal ganglia architecture into `selection' and `control' pathways. The former pathway performs the selection per se via a feedforward off-centre on-surround network. The control pathway regulates the action of the selection pathway to ensure its effective operation, and synergistically complements its dopaminergic modulation. The model contrasts with the prevailing functional segregation of basal ganglia into `direct' and `indirect' pathways. Received: 16 February 2000 / Accepted in revised form: 30 October 2000     

Cortical mechanisms of action selection: the affordance competition hypothesis

At every moment, the natural world presents animals with two fundamental pragmatic problems: selection between actions that are currently possible and specification of the parameters or metrics of those actions. It is commonly suggested that the brain addresses these by first constructing representations of the world on which to build knowledge and make a decision, and then by computing and executing an action plan. However, neurophysiological data argue against this serial viewpoint. In contrast, it is proposed here that the brain processes sensory information to specify, in parallel, several potential actions that are currently available. These potential actions compete against each other for further processing, while information is collected to bias this competition until a single response is selected. The hypothesis suggests that the dorsal visual system specifies actions which compete against each other within the fronto-parietal cortex, while a variety of biasing influences are provided by prefrontal regions and the basal ganglia. A computational model is described, which illustrates how this competition may take place in the cerebral cortex. Simulations of the model capture qualitative features of neurophysiological data and reproduce various behavioural phenomena.     

The computational neurobiology of learning and reward

Following the suggestion that midbrain dopaminergic neurons encode a signal, known as a 'reward prediction error', used by artificial intelligence algorithms for learning to choose advantageous actions, the study of the neural substrates for reward-based learning has been strongly influenced by computational theories. In recent work, such theories have been increasingly integrated into experimental design and analysis. Such hybrid approaches have offered detailed new insights into the function of a number of brain areas, especially the cortex and basal ganglia. In part this is because these approaches enable the study of neural correlates of subjective factors (such as a participant's beliefs about the reward to be received for performing some action) that the computational theories purport to quantify.     

The hippocampus, spatial memory and food hoarding: a puzzle revisited

Behavioural ecology assumes that cognitive traits and their underlying neural substrates are shaped by natural selection in much the same way as morphological traits are, resulting in adaptation to the natural environment of the species concerned. Recently, however, the 'neuroecology' approach of attempting to gain insight into brain structure and function by testing predictions about variation in brain structure based on knowledge of the lifestyle of the animal has been criticized on the grounds that such an adaptationist view cannot provide insight into the underlying mechanisms. Furthermore, the criticism has focussed on attempts to use variation in demand for spatial memory and in hippocampal size as a basis for predicting variation in cognitive abilities. Here, we revisit this critique against the field of so-called 'neuroecology' and argue that using knowledge of the natural history of animals has lead to a better understanding of the interspecific variation in spatial abilities and hippocampal size, and to the generation of novel hypotheses and predictions.     

Neural mechanisms in Williams syndrome: a unique window to genetic influences on cognition and behaviour

Williams syndrome, a rare disorder caused by hemizygous microdeletion of about 28 genes on chromosome 7q11.23, has long intrigued neuroscientists with its unique combination of striking behavioural abnormalities, such as hypersociability, and characteristic neurocognitive profile. Williams syndrome, therefore, raises fundamental questions about the neural mechanisms of social behaviour, the modularity of mind and brain development, and provides a privileged setting to understand genetic influences on complex brain functions in a 'bottom-up' way. We review recent advances in uncovering the functional and structural neural substrates of Williams syndrome that provide an emerging understanding of how these are related to dissociable genetic contributions characterized both in special participant populations and animal models.     

sOAR: a tool for modelling optimal animal life‐history strategies in cyclic environments

Periodic environments determine the life cycle of many animals across the globe and the timing of important life history events, such as reproduction and migration. These adaptive behavioural strategies are complex and can only be fully understood (and predicted) within the framework of natural selection in which species adopt evolutionary stable strategies. We present sOAR, a powerful and user‐friendly implementation of the well‐established framework of optimal annual routine modelling. It allows determining optimal animal life history strategies under cyclic environmental conditions using stochastic dynamic programming. It further includes the simulation of population dynamics under the optimal strategy. sOAR provides an important tool for theoretical studies on the behavioural and evolutionary ecology of animals. It is especially suited for studying bird migration. In particular, we integrated options to differentiate between costs of active and passive flight into the optimal annual routine modelling framework, as well as options to consider periodic wind conditions affecting flight energetics. We provide an illustrative example of sOAR where food supply in the wintering habitat of migratory birds significantly alters the optimal timing of migration. sOAR helps improving our understanding of how complex behaviours evolve and how behavioural decisions are constrained by internal and external factors experienced by the animal. Such knowledge is crucial for anticipating potential species’ response to global environmental change.     

Identifying the Basal Ganglia Network Model Markers for Medication-Induced Impulsivity in Parkinson's Disease Patients

Impulsivity, i.e. irresistibility in the execution of actions, may be prominent in Parkinson''s disease (PD) patients who are treated with dopamine precursors or dopamine receptor agonists. In this study, we combine clinical investigations with computational modeling to explore whether impulsivity in PD patients on medication may arise as a result of abnormalities in risk, reward and punishment learning. In order to empirically assess learning outcomes involving risk, reward and punishment, four subject groups were examined: healthy controls, ON medication PD patients with impulse control disorder (PD-ON ICD) or without ICD (PD-ON non-ICD), and OFF medication PD patients (PD-OFF). A neural network model of the Basal Ganglia (BG) that has the capacity to predict the dysfunction of both the dopaminergic (DA) and the serotonergic (5HT) neuromodulator systems was developed and used to facilitate the interpretation of experimental results. In the model, the BG action selection dynamics were mimicked using a utility function based decision making framework, with DA controlling reward prediction and 5HT controlling punishment and risk predictions. The striatal model included three pools of Medium Spiny Neurons (MSNs), with D1 receptor (R) alone, D2R alone and co-expressing D1R-D2R. Empirical studies showed that reward optimality was increased in PD-ON ICD patients while punishment optimality was increased in PD-OFF patients. Empirical studies also revealed that PD-ON ICD subjects had lower reaction times (RT) compared to that of the PD-ON non-ICD patients. Computational modeling suggested that PD-OFF patients have higher punishment sensitivity, while healthy controls showed comparatively higher risk sensitivity. A significant decrease in sensitivity to punishment and risk was crucial for explaining behavioral changes observed in PD-ON ICD patients. Our results highlight the power of computational modelling for identifying neuronal circuitry implicated in learning, and its impairment in PD. The results presented here not only show that computational modelling can be used as a valuable tool for understanding and interpreting clinical data, but they also show that computational modeling has the potential to become an invaluable tool to predict the onset of behavioral changes during disease progression.     

Neuroethology of reward and decision making

Ethology, the evolutionary science of behaviour, assumes that natural selection shapes behaviour and its neural substrates in humans and other animals. In this view, the nervous system of any animal comprises a suite of morphological and behavioural adaptations for solving specific information processing problems posed by the physical or social environment. Since the allocation of behaviour often reflects economic optimization of evolutionary fitness subject to physical and cognitive constraints, neurobiological studies of reward, punishment, motivation and decision making will profit from an appreciation of the information processing problems confronted by animals in their natural physical and social environments.     

Inside the brain of a neuron

For many decades, neurons were considered to be the elementary computational units of the brain and were assumed to summate incoming signals and elicit action potentials only in response to suprathreshold stimuli. Although modelling studies predicted that single neurons constitute a much more powerful computational entity, able to perform an array of nonlinear calculations, this possibility was not explored experimentally until the discovery of active mechanisms in the dendrites of most neuron types. Here, we review several modelling studies that have addressed information processing in single neurons, starting with those characterizing the arithmetic of different dendritic components, to those tackling neuronal integration at the cell body and, finally, those analysing the computational abilities of the axon. We present modelling predictions along with supporting experimental data in an effort to highlight the significant contribution of modelling work to enhancing our understanding of single-neuron arithmetic.     

A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem—action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.     

A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour

 In a companion paper a new functional architecture was proposed for the basal ganglia based on the premise that these brain structures play a central role in behavioural action selection. The current paper quantitatively describes the properties of the model using analysis and simulation. The decomposition of the basal ganglia into selection and control pathways is supported in several ways. First, several elegant features are exposed – capacity scaling, enhanced selectivity and synergistic dopamine modulation – which might be expected to exist in a well designed action selection mechanism. The discovery of these features also lends support to the computational premise of selection that underpins our model. Second, good matches between model globus pallidus external segment output and globus pallidus internal segment and substantia nigra reticulata area output, and neurophysiological data, have been found which are indicative of common architectural features in the model and biological basal ganglia. Third, the behaviour of the model as a signal selection mechanism has parallels with some kinds of action selection observed in animals under various levels of dopaminergic modulation. Received: 16 July 2000 / Accepted in revised form: 30 October 2000