Dissociation between Active and Observational Learning from Positive and Negative Feedback in Parkinsonism

Feedback to both actively performed and observed behaviour allows adaptation of future actions. Positive feedback leads to increased activity of dopamine neurons in the substantia nigra, whereas dopamine neuron activity is decreased following negative feedback. Dopamine level reduction in unmedicated Parkinson’s Disease patients has been shown to lead to a negative learning bias, i.e. enhanced learning from negative feedback. Recent findings suggest that the neural mechanisms of active and observational learning from feedback might differ, with the striatum playing a less prominent role in observational learning. Therefore, it was hypothesized that unmedicated Parkinson’s Disease patients would show a negative learning bias only in active but not in observational learning. In a between-group design, 19 Parkinson’s Disease patients and 40 healthy controls engaged in either an active or an observational probabilistic feedback-learning task. For both tasks, transfer phases aimed to assess the bias to learn better from positive or negative feedback. As expected, actively learning patients showed a negative learning bias, whereas controls learned better from positive feedback. In contrast, no difference between patients and controls emerged for observational learning, with both groups showing better learning from positive feedback. These findings add to neural models of reinforcement-learning by suggesting that dopamine-modulated input to the striatum plays a minor role in observational learning from feedback. Future research will have to elucidate the specific neural underpinnings of observational learning.     

Ventral striatum: a critical look at models of learning and evaluation

Extensive evidence implicates the ventral striatum in multiple distinct facets of action selection. Early work established a role in modulating ongoing behavior, as engaged by the energizing and directing influences of motivationally relevant cues and the willingness to expend effort in order to obtain reward. More recently, reinforcement learning models have suggested the notion of ventral striatum primarily as an evaluation step during learning, which serves as a critic to update a separate actor. Recent computational and experimental work may provide a resolution to the differences between these two theories through a careful parsing of behavior and the instrinsic heterogeneity that characterizes this complex structure.     

Action selection and action value in frontal-striatal circuits

The role that frontal-striatal circuits play in normal behavior remains unclear. Two of the leading hypotheses suggest that these circuits are important for action selection or reinforcement learning. To examine these hypotheses, we carried out an experiment in which monkeys had to select actions in two different task conditions. In the first (random) condition, actions were selected on the basis of perceptual inference. In the second (fixed) condition, the animals used reinforcement from previous trials to select actions. Examination of neural activity showed that the representation of the selected action was stronger in lateral prefrontal cortex (lPFC), and occurred earlier in the lPFC than it did in the dorsal striatum (dSTR). In contrast to this, the representation of action values, in both the random and fixed conditions, was stronger in the dSTR. Thus, the dSTR contains an enriched representation of action value, but it followed frontal cortex in action selection.     

The role of the striatum in aversive learning and aversive prediction errors

Neuroeconomic studies of decision making have emphasized reward learning as critical in the representation of value-driven choice behaviour. However, it is readily apparent that punishment and aversive learning are also significant factors in motivating decisions and actions. In this paper, we review the role of the striatum and amygdala in affective learning and the coding of aversive prediction errors (PEs). We present neuroimaging results showing aversive PE-related signals in the striatum in fear conditioning paradigms with both primary (shock) and secondary (monetary loss) reinforcers. These results and others point to the general role for the striatum in coding PEs across a broad range of learning paradigms and reinforcer types.     

The Striatum: Where Skills and Habits Meet

After more than a century of work concentrating on the motor functions of the basal ganglia, new ideas have emerged, suggesting that the basal ganglia also have major functions in relation to learning habits and acquiring motor skills. We review the evidence supporting the role of the striatum in optimizing behavior by refining action selection and in shaping habits and skills as a modulator of motor repertoires. These findings challenge the notion that striatal learning processes are limited to the motor domain. The learning mechanisms supported by striatal circuitry generalize to other domains, including cognitive skills and emotion-related patterns of action.The nuclei and interconnections of the basal ganglia are widely recognized for modulating motor behavior. Whether measured at the neuronal or regional level, the activities of neurons in the basal ganglia correlate with many movement parameters, particularly those that influence the vigor of an action, such as force and velocity. Pathology within different basal ganglia circuits predictably leads to either hypokinetic or hyperkinetic movement disorders. In parallel, however, the basal ganglia, and especially the striatum, are now widely recognized as being engaged in activity related to learning. Interactions between the dopamine-containing neurons of the midbrain and their targets in the striatum are critical to this function. A fundamental question is how these two capacities—(motor behavior and reinforcement-based learning)—relate to each other and what role the striatum and other basal ganglia nuclei have in forming new behavioral repertoires. Here, we consider relevant physiological properties of the striatum by contrasting two common forms of adaptation found in all mammals: the acquisition of behavioral habits and physical skills.Without resorting to technical definitions, we all have an intuition of what habits and skills are. Tying one’s shoes after putting them on is something we consider a habit—part of a behavioral routine. The capacity to tie the laces properly is a skill. Habits and skills have many common features. Habits are consistent behaviors triggered by appropriate events (typically, but not always, external stimuli) occurring within particular contexts. Physical skills are changes in a physical repertoire: new combinations of movements that lead to new capacities for goal-directed action. Both habits and skills can leverage reward-based learning, particularly during their initial acquisition. In either instance, after sufficient experience, the need for reward becomes lower and lower. With sufficient practice, both lead to “automaticity” and a resilience against competing actions that might lead to unlearning.     

5HT2A receptor blockade in dorsomedial striatum reduces repetitive behaviors in BTBR mice

Restricted and repetitive behaviors are a defining feature of autism, which can be expressed as a cognitive flexibility deficit or stereotyped, motor behaviors. There is limited knowledge about the underlying neuropathophysiology contributing to these behaviors. Previous findings suggest that central 5HT_2A receptor activity is altered in autism, while recent work indicates that systemic 5HT_2A receptor antagonist treatment reduces repetitive behaviors in an idiopathic model of autism. 5HT_2A receptors are expressed in the orbitofrontal cortex and striatum. These two regions have been shown to be altered in autism. The present study investigated whether 5HT_2A receptor blockade in the dorsomedial striatum or orbitofrontal cortex in the BTBR mouse strain, an idiopathic model of autism, affects the phenotype related to restricted and repetitive behaviors. Microinfusion of the 5HT_2A receptor antagonist, M100907 into the dorsomedial striatum alleviated a reversal learning impairment and attenuated grooming behavior. M100907 infusion into the orbitofrontal cortex increased perseveration during reversal learning and potentiated grooming. These findings suggest that increased 5HT_2A receptor activity in the dorsomedial striatum may contribute to behavioral inflexibility and stereotyped behaviors in the BTBR mouse. 5HT_2A receptor signaling in the orbitofrontal cortex may be critical for inhibiting a previously learned response during reversal learning and expression of stereotyped behavior. The present results suggest which brain areas exhibit abnormalities underlying repetitive behaviors in an idiopathic mouse model of autism, as well as which brain areas systemic treatment with M100907 may principally act on in BTBR mice to attenuate repetitive behaviors.     

A non-selective serotonin antagonist promotes rapid habituation in the terrestrial hermit crab

Research has indicated that serotonin (5-HT) modulates non-associative learning in a variety of invertebrate species. Recent work has demonstrated that the terrestrial hermit crab is a suitable animal model for non-associative learning phenomena, including habituation, sensitization, and dishabituation. We examined the potential role of a non-selective 5-HT antagonist, methysergide, in non-associative learning in the hermit crab. We administered methysergide prior to delivering repeated stimulus presentations of a looming visual predator. We found evidence for more rapid habituation relative to a control condition in which crabs did not receive the drug. These results indicate a role for 5-HT in the defensive behavior of the hermit crab and importantly, suggest a conserved role for 5-HT in modulating basic learning processes in invertebrates.     

Modular features of motor control and learning

The study of complex motor behaviours has highlighted the role of modular representations both in the planning and in the execution of actions. Recent findings suggest the presence of functional modules within a variety of neural structures. Computational investigations are now addressing the issue of how these modules may act concurrently to generate a wide repertoire of behaviours.     

GPR55, a G-Protein Coupled Receptor for Lysophosphatidylinositol,Plays a Role in Motor Coordination

The G-protein coupled receptor 55 (GPR55) is activated by lysophosphatidylinositols and some cannabinoids. Recent studies found prominent roles for GPR55 in neuropathic/inflammatory pain, cancer and bone physiology. However, little is known about the role of GPR55 in CNS development and function. To address this question, we performed a detailed characterization of GPR55 knockout mice using molecular, anatomical, electrophysiological, and behavioral assays. Quantitative PCR studies found that GPR55 mRNA was expressed (in order of decreasing abundance) in the striatum, hippocampus, forebrain, cortex, and cerebellum. GPR55 deficiency did not affect the concentrations of endocannabinoids and related lipids or mRNA levels for several components of the endocannabinoid system in the hippocampus. Normal synaptic transmission and short-term as well as long-term synaptic plasticity were found in GPR55 knockout CA1 pyramidal neurons. Deleting GPR55 function did not affect behavioral assays assessing muscle strength, gross motor skills, sensory-motor integration, motor learning, anxiety or depressive behaviors. In addition, GPR55 null mutant mice exhibited normal contextual and auditory-cue conditioned fear learning and memory in a Pavlovian conditioned fear test. In contrast, when presented with tasks requiring more challenging motor responses, GPR55 knockout mice showed impaired movement coordination. Taken together, these results suggest that GPR55 plays a role in motor coordination, but does not strongly regulate CNS development, gross motor movement or several types of learned behavior.     

Impaired working memory,cognitive flexibility and reward processing in mice genetically lacking Gpr88: Evidence for a key role for Gpr88 in multiple cortico-striatal-thalamic circuits

The GPR88 orphan G protein-coupled receptor is expressed throughout the striatum, being preferentially localised in medium spiny neurons. It is also present in lower densities in frontal cortex and thalamus. Rare mutations in humans suggest a role in cognition and motor function, while common variants are associated with psychosis. Here we evaluate the influence of genetic deletion of GPR88 upon performance in translational tasks interrogating motivation, reward evaluation and cognitive function. In an automated radial arm maze ‘N-back’ working memory task, Gpr88 KO mice showed impaired correct responding, suggesting a role for GPR88 receptors in working memory circuitry. Associative learning performance was similar to wild-type controls in a touchscreen task but performance was impaired at the reversal learning stage, suggesting cognitive inflexibility. Gpr88 KO mice showed higher breakpoints, reduced latencies and lengthened session time in a progressive ratio task consistent with enhanced motivation. Simultaneously, locomotor hyperactivity was apparent in this task, supporting previous findings of actions of GPR88 in a cortico-striatal-thalamic motor loop. Evidence for a role of GPR88 in reward processing was demonstrated in a touchscreen-based equivalent of the Iowa gambling task. Although both Gpr88 KO and wild-type mice showed a preference for an optimum contingency choice, Gpr88 KO mice selected more risky choices at the expense of more advantageous lower risk options. Together these novel data suggest that striatal GPR88 receptors influence activity in a range of procedures integrated by prefrontal, orbitofrontal and anterior cingulate cortico-striatal-thalamic loops leading to altered cognitive, motivational and reward evaluation processes.     

Caspr3-Deficient Mice Exhibit Low Motor Learning during the Early Phase of the Accelerated Rotarod Task

Caspr3 (Contactin-associated protein-like 3, Cntnap3) is a neural cell adhesion molecule belonging to the Caspr family. We have recently shown that Caspr3 is expressed abundantly between the first and second postnatal weeks in the mouse basal ganglia, including the striatum, external segment of the globus pallidus, subthalamic nucleus, and substantia nigra. However, its physiological role remains largely unknown. In this study, we conducted a series of behavioral analyses on Capsr3-knockout (KO) mice and equivalent wild-type (WT) mice to investigate the role of Caspr3 in brain function. No significant differences were observed in most behavioral traits between Caspr3-KO and WT mice, but we found that Caspr3-KO mice performed poorly during the early phase of the accelerated rotarod task in which latency to falling off a rod rotating with increasing velocity was examined. In the late phase, the performance of the Caspr3-KO mice caught up to the level of WT mice, suggesting that the deletion of Caspr3 caused a delay in motor learning. We then examined changes in neural activity after training on the accelerated rotarod by conducting immunohistochemistry using antibody to c-Fos, an indirect marker for neuronal activity. Experience of the accelerated rotarod task caused increases in the number of c-Fos-positive cells in the dorsal striatum, cerebellum, and motor cortex in both Caspr3-KO and WT mice, but the number of c-Fos-positive cells was significantly lower in the dorsal striatum of Caspr3-KO mice than in that of WT mice. The expression of c-Fos in the ventral striatum of Caspr3-KO and WT mice was not altered by the training. Our findings suggest that reduced activation of neural cells in the dorsal striatum in Caspr3-KO mice leads to a decline in motor learning in the accelerated rotarod task.     

Cdk5: a novel role in learning and memory

Learning and memory are processes by which organisms acquire, retain and retrieve information. They result in modifications of behavior in response to new or previously encountered stimuli thereby enabling adaptation to a permanently changing environment. Protein phosphorylation has long been known to play a key role in triggering synaptic changes underlying learning and memory. Although intracellular phosphorylation and dephosphorylation is orchestrated by a complex network of interactions between a number of protein kinases and phosphatases, significant advances in the understanding of neuronal mechanisms underlying learning and memory have been achieved by investigating the actions of individual molecules under defined experimental conditions, brain areas, neuronal cells and their subcellular compartments. On the basis of these approaches, the cyclic AMP protein kinase (PKA), protein kinase C (PKC) and extracellularly regulated protein kinases 1 and 2 (Erk-1/2) have been identified as the core signaling pathways in memory consolidation. Here we review recent findings demonstrating an important novel role for Cdk5 in learning and memory. We suggest that some of the well-characterized roles of Cdk5 during neurodevelopmental processes, such as interactions with distinct cytoplasmic and synaptic target molecules, may be also involved in synaptic plasticity underlying memory consolidation within the adult central nervous system.     

Progressive plasticity of auditory cortex during appetitive operant conditioning

In stimulus-response-outcome learning, different regions in the cortico-basal ganglia network are progressively involved according to the stage of learning. However, the involvement of sensory cortex remains ellusive even though massive cortical projections to the striatum imply its significant role in this learning. Here we show that the global tonotopic representation in the auditory cortex changed progressively depending on the stage of training in auditory operant conditioning. At the early stage, tone-responsive areas mainly in the core cortex expanded, while both the core and belt cortices shrank at the late stage as behavior became conditioned. Taken together with previous findings, this progressive global plasticity from the core to belt cortices suggests differentiated roles in these areas: the core cortex serves as a filter to better identify auditory objects for hierarchical computation within the belt cortex, while the belt stores auditory objects and affects decision making through direct projections to limbic system and higher association cortex. Thus, the progressive plasticity in the present study reflects a shift from identification to storage of a behaviorally relevant auditory object, which is potentially associated with a habitual behavior.     

Adenosine signaling in striatal circuits and alcohol use disorders

Adenosine signaling has been implicated in the pathophysiology of alcohol use disorders and other psychiatric disorders such as anxiety and depression. Numerous studies have indicated a role for A1 receptors (A1R) in acute ethanol-induced motor incoordination, while A2A receptors (A2AR) mainly regulate the rewarding effect of ethanol in mice. Recent findings have demonstrated that dampened A2AR-mediated signaling in the dorsomedial striatum (DMS) promotes ethanol-seeking behaviors. Moreover, decreased A2AR function is associated with decreased CREB activity in the DMS, which enhances goal-oriented behaviors and contributes to excessive ethanol drinking in mice. Interestingly, caffeine, the most commonly used psychoactive substance, is known to inhibit both the A1R and A2AR. This dampened adenosine receptor function may mask some of the acute intoxicating effects of ethanol. Furthermore, based on the fact that A2AR activity plays a role in goal-directed behavior, caffeine may also promote ethanol-seeking behavior. The A2AR is enriched in the striatum and exclusively expressed in striatopallidal neurons, which may be responsible for the regulation of inhibitory behavioral control over drug rewarding processes through the indirect pathway of the basal ganglia circuit. Furthermore, the antagonistic interactions between adenosine and dopamine receptors in the striatum also play an integral role in alcoholism and addiction-related disorders. This review focuses on regulation of adenosine signaling in striatal circuits and the possible implication of caffeine in goal-directed behaviors and addiction.     

Beetles, boxes and brain cells: neural mechanisms underlying valuation and learning

Sensory cues in the environment can predict the availability of reward. Through experience, humans and animals learn these predictions and use them to guide their actions. For example, we can learn to discriminate chanterelles from ordinary champignons through experience. Assuming the development of a taste for the complex and lingering flavors of chanterelles, we therefore learn to value the same action--picking mushrooms--differentially depending upon the appearance of a mushroom. One major goal of cognitive neuroscience is to understand the neural mechanisms that underlie this sort of learning. Because the acquisition of rewards motivates much behavior, recent efforts have focused on describing the neural signals related to learning the value of stimuli and actions. Neurons in the basal ganglia, in midbrain dopamine areas, in frontal and parietal cortices and in other brain areas, all modulate their activity in relation to aspects of learning. By training monkeys on various behavioral tasks, recent studies have begun to characterize how neural signals represent distinct processes, such as the timing of events, motivation, absolute (objective) and relative (subjective) valuation, and the formation of associative links between stimuli and potential actions. In addition, a number of studies have either further characterized dopamine signals or sought to determine how such signaling might interact with target structures, such as the striatum and rhinal cortex, to underlie learning.     

Getting formal with dopamine and reward

Recent neurophysiological studies reveal that neurons in certain brain structures carry specific signals about past and future rewards. Dopamine neurons display a short-latency, phasic reward signal indicating the difference between actual and predicted rewards. The signal is useful for enhancing neuronal processing and learning behavioral reactions. It is distinctly different from dopamine's tonic enabling of numerous behavioral processes. Neurons in the striatum, frontal cortex, and amygdala also process reward information but provide more differentiated information for identifying and anticipating rewards and organizing goal-directed behavior. The different reward signals have complementary functions, and the optimal use of rewards in voluntary behavior would benefit from interactions between the signals. Addictive psychostimulant drugs may exert their action by amplifying the dopamine reward signal.     

Striatal level of regulation of learned forepaw movements in rats

The role of the striatal adenylyl cyclase (AC) and cholinergic systems in the learning and expression of new forepaw movements (reaching with prolonged pushing on a fixed piston) was studied in male Wistar rats. Motor learning processes, prenatal hypoxia, and cholinergic drugs changed the properties of the AC system in the striatum. After learning, the striatal basal AC activity was decreased compared to untrained control rats. In addition, the AC activity was more decreased in animals with a good ability to learn compared to poor learners (up to 31 % and 51 %, correspondingly; p<0.01). Rats subjected to prenatal hypoxia (13-14th days of embryogenesis) had a lower ability to learn the new movements requiring tactile control and the striatal AC activity in these rats was 1.8 times higher (p<0.001) than controls. In vitro application of the cholinergic agonist carbachol (CARB) 10-5 M (corresponding to approximately 0.3 microg), as well as the antagonist scopolomine (SCOP) 10(-5) M (approximately 0.3 microg) decreased AC activity in the synaptosomal fraction of the striatum. In vivo injections of CARB (0.3-3 microg/1microl) or SCOP (0.3-3 microg/1microl) into the ventral striatum (nucleus accumbens) modified the newly learned sensorimotor skill. After CARB injections the rats performed slower movements with more prolonged pushing. After SCOP the rats could not retain the learned pushing movement. These in vivo and in vitro data suggest that the cholinergic mediator system of the striatum is involved in learning sensory-controlled forepaw movements as well as the regulation of new motor skills by modulating the AC signal transduction process in the striatum. The data confirmed that modification of the striatal AC system resulted in the modulation of reaching behavior and better expression of the learned reflex.     

Model-based influences on humans' choices and striatal prediction errors

The mesostriatal dopamine system is prominently implicated in model-free reinforcement learning, with fMRI BOLD signals in ventral striatum notably covarying with model-free prediction errors. However, latent learning and devaluation studies show that behavior also shows hallmarks of model-based planning, and the interaction between model-based and model-free values, prediction errors, and preferences is underexplored. We designed a multistep decision task in which model-based and model-free influences on human choice behavior could be distinguished. By showing that choices reflected both influences we could then test the purity of the ventral striatal BOLD signal as a model-free report. Contrary to expectations, the signal reflected both model-free and model-based predictions in proportions matching those that best explained choice behavior. These results challenge the notion of a separate model-free learner and suggest a more integrated computational architecture for high-level human decision-making.     

The contribution of oxytocin and vasopressin to mammalian social behavior: potential role in autism spectrum disorder

Oxytocin (OT) and arginine-vasopressin (AVP) are 2 peptides that are produced in the brain and released via the pituitary gland to the peripheral blood, where they have diverse physiological functions. In the last 2 decades it has become clear that these peptides also play a central role in the modulation of mammalian social behavior by their actions within the brain. Several lines of evidence suggest their involvement in autism spectrum disorder (ASD), which is known to be associated with impaired social cognition and behavior. Recent clinical trials using OT administration to autistic patients have reported promising results. Here, we aim to describe the main data that suggest a connection between these peptides and ASD. Following a short illustration of several major topics in ASD biology we will (a) briefly describe the oxytocinergic and vasopressinergic systems in the brain, (b) discuss a few compelling cases manifesting the involvement of OT and AVP in mammalian social behavior, (c) describe data supporting the role of these peptides in human social cognition and behavior, and (d) discuss the possibility of the involvement of OT and AVP in ASD etiology, as well as the prospect of using these peptides as a treatment for ASD patients.     

Motor Skill Learning Is Associated with Phase-Dependent Modifications in the Striatal cAMP/PKA/DARPP-32 Signaling Pathway in Rodents

Abundant evidence points to a key role of dopamine in motor skill learning, although the underlying cellular and molecular mechanisms are still poorly understood. Here, we used a skilled-reaching paradigm to first examine changes in the expression of the plasticity-related gene Arc to map activity in cortico-striatal circuitry during different phases of motor skill learning in young animals. In the early phase, Arc mRNA was significantly induced in the medial prefrontal cortex (mPFC), cingulate cortex, primary motor cortex, and striatum. In the late phase, expression of Arc did not change in most regions, except in the mPFC and dorsal striatum. In the second series of experiments, we studied the learning-induced changes in the phosphorylation state of dopamine and cAMP-regulated phosphoprotein, 32k Da (DARPP-32). Western blot analysis of the phosphorylation state of DARPP-32 and its downstream target cAMP response element-binding protein (CREB) in the striatum revealed that the early, but not late, phase of motor skill learning was associated with increased levels of phospho-Thr34-DARPP-32 and phospho-Ser133-CREB. Finally, we used the DARPP-32 knock-in mice with a point mutation in the Thr34 regulatory site (i.e., protein kinase A site) to test the significance of this pathway in motor skill learning. In accordance with our hypothesis, inhibition of DARPP-32 activity at the Thr34 regulatory site strongly attenuated the motor learning rate and skilled reaching performance of mice. These findings suggest that the cAMP/PKA/DARPP-32 signaling pathway is critically involved in the acquisition of novel motor skills, and also demonstrate a dynamic shift in the contribution of cortico-striatal circuitry during different phases of motor skill learning.