Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit

Accumulating evidence shows that the neural network of the cerebral cortex and the basal ganglia is critically involved in reinforcement learning. Recent studies found functional heterogeneity within the cortico-basal ganglia circuit, especially in its ventromedial to dorsolateral axis. Here we review computational issues in reinforcement learning and propose a working hypothesis on how multiple reinforcement learning algorithms are implemented in the cortico-basal ganglia circuit using different representations of states, values, and actions.     

A dynamical model for the basal ganglia-thalamo-cortical oscillatory activity and its implications in Parkinson’s disease

We propose to investigate brain electrophysiological alterations associated with Parkinson’s disease through a novel adaptive dynamical model of the network of the basal ganglia, the cortex and the thalamus. The model uniquely unifies the influence of dopamine in the regulation of the activity of all basal ganglia nuclei, the self-organised neuronal interdependent activity of basal ganglia-thalamo-cortical circuits and the generation of subcortical background oscillations. Variations in the amount of dopamine produced in the neurons of the substantia nigra pars compacta are key both in the onset of Parkinson’s disease and in the basal ganglia action selection. We model these dopamine-induced relationships, and Parkinsonian states are interpreted as spontaneous emergent behaviours associated with different rhythms of oscillatory activity patterns of the basal ganglia-thalamo-cortical network. These results are significant because: (1) the neural populations are built upon single-neuron models that have been robustly designed to have eletrophysiologically-realistic responses, and (2) our model distinctively links changes in the oscillatory activity in subcortical structures, dopamine levels in the basal ganglia and pathological synchronisation neuronal patterns compatible with Parkinsonian states, this still remains an open problem and is crucial to better understand the progression of the disease.Electronic supplementary materialThe online version of this article (10.1007/s11571-020-09653-y) contains supplementary material, which is available to authorized users.     

The role of the basal ganglia in exploration in a neural model based on reinforcement learning

We present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.     

Default Mode Network,Motor Network,Dorsal and Ventral Basal Ganglia Networks in the Rat Brain: Comparison to Human Networks Using Resting State-fMRI

Rodent models are developed to enhance understanding of the underlying biology of different brain disorders. However, before interpreting findings from animal models in a translational aspect to understand human disease, a fundamental step is to first have knowledge of similarities and differences of the biological systems studied. In this study, we analyzed and verified four known networks termed: default mode network, motor network, dorsal basal ganglia network, and ventral basal ganglia network using resting state functional MRI (rsfMRI) in humans and rats. Our work supports the notion that humans and rats have common robust resting state brain networks and that rsfMRI can be used as a translational tool when validating animal models of brain disorders. In the future, rsfMRI may be used, in addition to short-term interventions, to characterize longitudinal effects on functional brain networks after long-term intervention in humans and rats.     

A biologically constrained model of the whole basal ganglia addressing the paradoxes of connections and selection

The basal ganglia nuclei form a complex network of nuclei often assumed to perform selection, yet their individual roles and how they influence each other is still largely unclear. In particular, the ties between the external and internal parts of the globus pallidus are paradoxical, as anatomical data suggest a potent inhibitory projection between them while electrophysiological recordings indicate that they have similar activities. Here we introduce a theoretical study that reconciles both views on the intra-pallidal projection, by providing a plausible characterization of the relationship between the external and internal globus pallidus. Specifically, we developed a mean-field model of the whole basal ganglia, whose parameterization is optimized to respect best a collection of numerous anatomical and electrophysiological data. We first obtained models respecting all our constraints, hence anatomical and electrophysiological data on the intrapallidal projection are globally consistent. This model furthermore predicts that both aforementioned views about the intra-pallidal projection may be reconciled when this projection is weakly inhibitory, thus making it possible to support similar neural activity in both nuclei and for the entire basal ganglia to select between actions. Second, we predicts that afferent projections are substantially unbalanced towards the external segment, as it receives the strongest excitation from STN and the weakest inhibition from the striatum. Finally, our study strongly suggests that the intrapallidal connection pattern is not focused but diffuse, as this latter pattern is more efficient for the overall selection performed in the basal ganglia.     

Do Basal Ganglia Amplify Willed Action by Stochastic Resonance? A Model

Basal ganglia are usually attributed a role in facilitating willed action, which is found to be impaired in Parkinson''s disease, a pathology of basal ganglia. We hypothesize that basal ganglia possess the machinery to amplify will signals, presumably weak, by stochastic resonance. Recently we proposed a computational model of Parkinsonian reaching, in which the contributions from basal ganglia aid the motor cortex in learning to reach. The model was cast in reinforcement learning framework. We now show that the above basal ganglia computational model has all the ingredients of stochastic resonance process. In the proposed computational model, we consider the problem of moving an arm from a rest position to a target position: the two positions correspond to two extrema of the value function. A single kick (a half-wave of sinusoid, of sufficiently low amplitude) given to the system in resting position, succeeds in taking the system to the target position, with high probability, only at a critical noise level. But for suboptimal noise levels, the model arm''s movements resemble Parkinsonian movement symptoms like akinetic rigidity (low noise) and dyskinesias (high noise).     

Brain-like Intelligent Decision-making Based on Basal Ganglia and Its Application in Automatic Car-following

The anthropomorphic intelligence of autonomous driving has been a research hotspot in the world.However,current stud-ies have not been able to reveal the mechanism of drivers'natural driving behaviors.Therefore,this thesis starts from the perspective of cognitive decision-making in the human brain,which is inspired by the regulation of dopamine feedback in the basal ganglia,and a reinforcement learning model is established to solve the brain-like intelligent decision-making problems in the process of interacting with the environment.In this thesis,first,a detailed bionic mechanism architecture based on basal ganglia was proposed by the consideration and analysis of its feedback regulation mechanism;second,the above mechanism was transformed into a reinforcement Q-learning model,so as to implement the learning and adaptation abilities of an intelligent vehicle for brain-like intelligent decision-making during car-following;finally,the feasibility and effectiveness of the proposed method were verified by the simulations and real vehicle tests.     

A model for altered neural network dynamics related to prehension movements in Parkinson disease

In this paper, we present a neural network model of the interactions between cortex and the basal ganglia during prehensile movements. Computational neuroscience methods are used to explore the hypothesis that the altered kinematic patterns observed in Parkinson’s disease patients performing prehensile movements is mainly due to an altered neuronal activity located in the networks of cholinergic (ACh) interneurons of the striatum. These striatal cells, under a strong influence of the dopaminergic system, significantly contribute to the neural processing within the striatum and in the cortico-basal ganglia loops. In order to test this hypothesis, a large-scale model of neural interactions in the basal ganglia has been integrated with previous models accounting for the cortical organization of goal directed reaching and grasping movements in normal and perturbed conditions. We carry out a discussion of the model hypothesis validation by providing a control engineering analysis and by comparing results of real experiments with our simulation results in conditions resembling these original experiments.     

PET imaging of the basal ganglia

The present chapter reviews PET imaging in basal ganglia disorders; Parkinson's disease is used as a model of these disorders because the neurochemical pathobiology of this disease is well known and great advances in the imaging area have been achieved. Other basal ganglia disorders including Tourette's syndrome, dystonia, Huntington's chorea and Wilson's disease are also dealt with. With PET and SPECT techniques, the whole integrative dopaminergic network of neurons can be studied, which plays an important role in differential diagnostics. Furthermore, pharmacological effects of medication can be visualized and the role of stereotaxic neurosurgery can be evaluated. Finally, functional imaging gives clues about the prognosis and rehabilitation aspects of the basal ganglia disorders.     

Complementary roles of basal ganglia and cerebellum in learning and motor control

The classical notion that the basal ganglia and the cerebellum are dedicated to motor control has been challenged by the accumulation of evidence revealing their involvement in non-motor, cognitive functions. From a computational viewpoint, it has been suggested that the cerebellum, the basal ganglia, and the cerebral cortex are specialized for different types of learning: namely, supervised learning, reinforcement learning and unsupervised learning, respectively. This idea of learning-oriented specialization is helpful in understanding the complementary roles of the basal ganglia and the cerebellum in motor control and cognitive functions.     

Two-phase model of the basal ganglia: implications for discontinuous control of the motor system

In this article, I point out that simple one-phase models of the role of the basal ganglia in action selection have a problem. Furthermore, I suggest a solution with major implications for the organization of the action-selection and motor systems. In current models, the striatum evaluates multiple potential actions by adding biases based on previous conditioning. These biases may arise in both the direct (bias for) and indirect (bias against) pathways. Together, these biases influence which action is ultimately chosen. For efficient conditioning to occur, a positive outcome must selectively strengthen the striatal bias for the chosen action (via a dopaminergic mechanism). This is problematic, however, because all potential action choices have influenced firing patterns in striatal cells during the selection process; it is therefore unclear how the synapses that represent the chosen plan could be selectively strengthened. I suggest a simple solution in which the striatum has two functional phases. In the first phase, the basal ganglia provide biases for multiple potential actions (using both the direct and indirect pathways), leading to the choice of a single action in the cortex. In the second phase, an efference copy of the chosen action is sent to the striatum, where it contributes to the establishment of the eligibility trace for that action. This trace, when acted on by subsequent dopaminergic reinforcement, leads to specific strengthening of the bias only for the chosen action. Consistent with this model, recordings show post-choice imposition onto the striatum of signals corresponding to the chosen action. The existence of dual phases of basal ganglia function implies that decisions about action choice are sent to the motor system in a discontinuous manner. This would not be problematic if the motor system also operated discontinuously. I will review evidence suggesting that this is the case, notably that action is organized by approximately 10 Hz oscillations.     

A computational model of action selection in the basal ganglia. I. A new functional anatomy

 We present a biologically plausible model of processing intrinsic to the basal ganglia based on the computational premise that action selection is a primary role of these central brain structures. By encoding the propensity for selecting a given action in a scalar value (the salience), it is shown that action selection may be re-cast in terms of signal selection. The generic properties of signal selection are defined and neural networks for this type of computation examined. A comparison between these networks and basal ganglia anatomy leads to a novel functional decomposition of the basal ganglia architecture into `selection' and `control' pathways. The former pathway performs the selection per se via a feedforward off-centre on-surround network. The control pathway regulates the action of the selection pathway to ensure its effective operation, and synergistically complements its dopaminergic modulation. The model contrasts with the prevailing functional segregation of basal ganglia into `direct' and `indirect' pathways. Received: 16 February 2000 / Accepted in revised form: 30 October 2000     

运动过程的网络逻辑——从离子通道到动物行为

为了揭示神经网络在脊椎动物运动中所行使的内在功能,作者开发了七鳃鳗这种低等脊椎动物模型。在这套系统中,不仅可以了解到运动模式生成网络以及激活此网络的命令系统,同时还可以在运动中研究方向控制系统和变向控制系统。七鳃鳗的神经系统有较少的神经元,而且运动行为中的不同运动模式可以由分离的神经系统所引发。模式生成神经网络包括同侧的谷氨酸能中间神经元和对侧的抑制性甘氨酸能中间神经元。网络中的突触连接、细胞膜特性和神经递质都也已经被鉴定。运动是由脑干区域的网状脊髓神经元所引起,而这些神经元又是被问脑和中脑分离的一些运动命令神经元群所控制。因此,运动行为最初是由这两个“运动核心”所启动。而这两个运动核心被基底神经节调控,基底神经节即时地做出判断是否允许下游的运动程序启动。在静止情况下基底神经节的输出核团维持对下游不同运动核心的抑制作用,反之则去除抑制活化运动核心。纹状体和苍白球被认为是这个运动抉择系统的主要部件。根据“霍奇金一贺胥黎”模型神经元开发了这套网络模型,不同的细胞具有各自相应的不同亚型的钠、钾、钙离子通道和钙依赖的钾通道。每个模型神经元拥有86个不同区域模块以及其对应的生物学功能,例如频率控制、超极化等等。然后根据已有实验证据,利用突触将不同的模型神经元相连。而系统中的10000个神经元大致和生物学网络上的细胞数量相当。突触数量为760000。突触类型有AMPA、NMDA、glycine型。有了这样大规模的模型,不仅可以模拟肌节与肌节之间的神经网络,还可以模拟到由基底神经节开始的行为起始部分。此外,这些网络模拟还被用于一个神经机械学模型来模拟包含有推进和方向控制部分的真实运动。     

A neural network model of basal ganglia’s decision-making circuitry

The basal ganglia have been increasingly recognized as an important structure involved in decision making. Neurons in the basal ganglia were found to reflect the evidence accumulation process during decision making. However, it is not well understood how the direct and indirect pathways of the basal ganglia work together for decision making. Here, we create a recurrent neural network model that is composed of the direct and indirect pathways and test it with the classic random dot motion discrimination task. The direct pathway drives the outputs, which are modulated through a gating mechanism controlled by the indirect pathway. We train the network to learn the task and find that the network reproduces the accuracy and reaction time patterns of previous animal studies. Units in the model exhibit ramping activities that reflect evidence accumulation. Finally, we simulate manipulations of the direct and indirect pathways and find that the manipulations of the direct pathway mainly affect the choice while the manipulations of the indirect pathway affect the model’s reaction time. These results suggest a potential circuitry mechanism of the basal ganglia’s role in decision making with predictions that can be tested experimentally in the future.     

Anatomical funneling, sparse connectivity and redundancy reduction in the neural networks of the basal ganglia

The major anatomical characteristics of the main axis of the basal ganglia are: (1) Numerical reduction in the number of neurons across layers of the feed-forward network, (2) lateral inhibitory connections within the layers, and (3) neuro-modulatory effects of dopamine and acetylcholine, both on the basal ganglia neurons and on the efficacy of information transmission along the basal ganglia axis. We recorded the simultaneous activity of neurons in the output stages of the basal ganglia as well as the activity of dopaminergic and cholinergic neurons during the performance of a probability decision-making task. We found that the functional messages of the cholinergic and dopaminergic neurons differ, and that the cholinergic message is less specific than that of the dopaminergic neurons. The output stage of the basal ganglia showed uncorrelated neuronal activity. We conclude that despite the huge numerical reduction from the cortex to the output nuclei of the basal ganglia, the activity of these nuclei represents an optimally compressed (uncorrelated) version of distinctive features of cortical information.     

Dopamine modulation in the basal ganglia locks the gate to working memory

The prefrontal cortex and basal ganglia are deeply implicated in working memory. Both structures are subject to dopaminergic neuromodulation in a way that exerts a critical influence on the proper operation of working memory. We present a novel network model to elucidate the role of phasic dopamine in the interaction of these two structures in initiating and maintaining mnemonic activity. We argue that neuromodulation plays a critical role in protecting memories against both internal and external sources of noise. Increases in cortical gain engendered by prefrontal dopamine release help make memories robust against external distraction, but do not offer protection against internal noise accompanying recurrent cortical activity. Rather, the output of the basal ganglia provides the gating function of stabilization against noise and distraction by enhancing select memories through targeted disinhibition of cortex. Dopamine in the basal ganglia effectively locks this gate by influencing the stability of up and down states in the striatum. Dopamine's involvement in affective processing endows this gating with specificity to motivational salience. We model a spatial working memory task and show that these combined effects of dopamine lead to superior performance. Action editor: Misha V. Tsodyks     

Action,time and the basal ganglia

The ability to control the speed of movement is compromised in neurological disorders involving the basal ganglia, a set of subcortical cerebral nuclei that receive prominent dopaminergic projections from the midbrain. For example, bradykinesia, slowness of movement, is a major symptom of Parkinson''s disease, whereas rapid tics are observed in patients with Tourette syndrome. Recent experimental work has also implicated dopamine (DA) and the basal ganglia in action timing. Here, I advance the hypothesis that the basal ganglia control the rate of change in kinaesthetic perceptual variables. In particular, the sensorimotor cortico-basal ganglia network implements a feedback circuit for the control of movement velocity. By modulating activity in this network, DA can change the gain of velocity reference signals. The lack of DA thus reduces the output of the velocity control system which specifies the rate of change in body configurations, slowing the transition from one body configuration to another.     

Properties of Neurons in External Globus Pallidus Can Support Optimal Action Selection

The external globus pallidus (GPe) is a key nucleus within basal ganglia circuits that are thought to be involved in action selection. A class of computational models assumes that, during action selection, the basal ganglia compute for all actions available in a given context the probabilities that they should be selected. These models suggest that a network of GPe and subthalamic nucleus (STN) neurons computes the normalization term in Bayes’ equation. In order to perform such computation, the GPe needs to send feedback to the STN equal to a particular function of the activity of STN neurons. However, the complex form of this function makes it unlikely that individual GPe neurons, or even a single GPe cell type, could compute it. Here, we demonstrate how this function could be computed within a network containing two types of GABAergic GPe projection neuron, so-called ‘prototypic’ and ‘arkypallidal’ neurons, that have different response properties in vivo and distinct connections. We compare our model predictions with the experimentally-reported connectivity and input-output functions (f-I curves) of the two populations of GPe neurons. We show that, together, these dichotomous cell types fulfil the requirements necessary to compute the function needed for optimal action selection. We conclude that, by virtue of their distinct response properties and connectivities, a network of arkypallidal and prototypic GPe neurons comprises a neural substrate capable of supporting the computation of the posterior probabilities of actions.     

A neural model of basal ganglia-thalamocortical relations in normal and parkinsonian movement

 Anatomical, neurophysiological, and neurochemical evidence supports the notion of parallel basal ganglia–thalamocortical motor systems. We developed a neural network model for the functioning of these systems during normal and parkinsonian movement. Parkinson’s disease (PD), which results predominantly from nigrostriatal pathway damage, is used as a window to examine basal ganglia function. Simulations of dopamine depletion produce motor impairments consistent with motor deficits observed in PD that suggest the basal ganglia play a role in motor initiation and execution, and sequencing of motor programs. Stereotaxic lesions in the model’s globus pallidus and subthalamic nucleus suggest that these lesions, although reducing some PD symptoms, may constrain the repertoire of available movements. It is proposed that paradoxical observations of basal ganglia responses reported in the literature may result from regional functional neuronal specialization, and the non-uniform distributions of neurochemicals in the basal ganglia. It is hypothesized that dopamine depletion produces smaller-than-normal pallidothalamic gating signals that prevent rescalability of these signals to control variable movement speed, and that in PD can produce smaller-than-normal movement amplitudes. Received: 1 September 1994/Accepted in revised form: 16 May 1995     

Effects on hypothalamus when CPG is fed back to basal ganglia based on KIV model

The KIV model approximates the operation of the basic vertebrate forebrain together with the basal ganglia and motor systems. In KIV model, the hypothalamus and the basal ganglia which are two important parts in the midline forebrain are closely associated with the locomotion. The CPG model with time delay is established in this paper and the stability of this CPG model is discussed. The CPG output is treated as the proprioception and fed back to the basal ganglia. We focus on the effects on the hypothalamus and the basal ganglia when the time delay parameter a_d, the CPG amplitude parameter e and the CPG frequency parameter T_r are changed. Through analysis, we find that there exists optimum value of the parameters a_d or T_r which can make the synchronization of the hypothalamus optimum when the CPG is added into the basal ganglia. The results could have important implications for biological processes which are about interaction between the neural network and the CPG.