The Effect of Ratio and Interval Training on Pavlovian-Instrumental Transfer in Mice

Conditional stimuli (CS) that are paired with reward can be used to motivate instrumental responses. This process is called Pavlovian-instrumental transfer (PIT). A recent study in rats suggested that habitual responses are particularly sensitive to the motivational effects of reward cues. The current experiments examined this idea using ratio and interval training in mice. Two groups of animals were trained to lever press for food pellets that were delivered on random ratio or random interval schedules. Devaluation tests revealed that interval training led to habitual responding while ratio training produced goal-directed actions. The presentation of CSs paired with reward led to positive transfer in both groups, however, the size of this effect was much larger in mice that were trained on interval schedules. This result suggests that habitual responses are more sensitive to the motivational influence of reward cues than goal-directed actions. The implications for neurobiological models of motivation and drug seeking behaviors are discussed.     

Actions,Action Sequences and Habits: Evidence That Goal-Directed and Habitual Action Control Are Hierarchically Organized

Behavioral evidence suggests that instrumental conditioning is governed by two forms of action control: a goal-directed and a habit learning process. Model-based reinforcement learning (RL) has been argued to underlie the goal-directed process; however, the way in which it interacts with habits and the structure of the habitual process has remained unclear. According to a flat architecture, the habitual process corresponds to model-free RL, and its interaction with the goal-directed process is coordinated by an external arbitration mechanism. Alternatively, the interaction between these systems has recently been argued to be hierarchical, such that the formation of action sequences underlies habit learning and a goal-directed process selects between goal-directed actions and habitual sequences of actions to reach the goal. Here we used a two-stage decision-making task to test predictions from these accounts. The hierarchical account predicts that, because they are tied to each other as an action sequence, selecting a habitual action in the first stage will be followed by a habitual action in the second stage, whereas the flat account predicts that the statuses of the first and second stage actions are independent of each other. We found, based on subjects'' choices and reaction times, that human subjects combined single actions to build action sequences and that the formation of such action sequences was sufficient to explain habitual actions. Furthermore, based on Bayesian model comparison, a family of hierarchical RL models, assuming a hierarchical interaction between habit and goal-directed processes, provided a better fit of the subjects'' behavior than a family of flat models. Although these findings do not rule out all possible model-free accounts of instrumental conditioning, they do show such accounts are not necessary to explain habitual actions and provide a new basis for understanding how goal-directed and habitual action control interact.     

Goal-directed and habitual control in the basal ganglia: implications for Parkinson's disease

Progressive loss of the ascending dopaminergic projection in the basal ganglia is a fundamental pathological feature of Parkinson's disease. Studies in animals and humans have identified spatially segregated functional territories in the basal ganglia for the control of goal-directed and habitual actions. In patients with Parkinson's disease the loss of dopamine is predominantly in the posterior putamen, a region of the basal ganglia associated with the control of habitual behaviour. These patients may therefore be forced into a progressive reliance on the goal-directed mode of action control that is mediated by comparatively preserved processing in the rostromedial striatum. Thus, many of their behavioural difficulties may reflect a loss of normal automatic control owing to distorting output signals from habitual control circuits, which impede the expression of goal-directed action.     

Impairments in Goal-Directed Actions Predict Treatment Response to Cognitive-Behavioral Therapy in Social Anxiety Disorder

Social anxiety disorder is characterized by excessive fear and habitual avoidance of social situations. Decision-making models suggest that patients with anxiety disorders may fail to exhibit goal-directed control over actions. We therefore investigated whether such biases may also be associated with social anxiety and to examine the relationship between such behavior with outcomes from cognitive-behavioral therapy. Patients diagnosed with social anxiety and controls completed an instrumental learning task in which two actions were performed to earn food outcomes. After outcome devaluation, where one outcome was consumed to satiety, participants were re-tested in extinction. Results indicated that, as expected, controls were goal-directed, selectively reducing responding on the action that previously delivered the devalued outcome. Patients with social anxiety, however, exhibited no difference in responding on either action. This loss of a devaluation effect was associated with greater symptom severity and poorer response to therapy. These findings indicate that variations in goal-directed control in social anxiety may represent both a behavioral endophenotype and may be used to predict individuals who will respond to learning-based therapies.     

Mushroom Bodies Regulate Habit Formation in Drosophila

To make good decisions, we evaluate past choices to guide later decisions. In most situations, we have the opportunity to simultaneously learn about both the consequences of our choice (i.e., operantly) and the stimuli associated with correct or incorrect choices (i.e., classically) [1]. Interestingly, in many species, including humans, these learning processes occasionally lead to irrational decisions [2]. An extreme case is the habitual drug user consistently administering the drug despite the negative consequences, but we all have experience with our own, less severe habits. The standard animal model employs a combination of operant and classical learning components to bring about habit formation in rodents [3] and [4]. After extended training, these animals will press a lever even if the outcome associated with lever-pressing is no longer desired [5]. In this study, experiments with wild-type and transgenic flies revealed that a prominent insect neuropil, the mushroom bodies (MBs), regulates habit formation in flies by inhibiting the operant learning system when a predictive stimulus is present. This inhibition enables generalization of the classical memory and prevents premature habit formation. Extended training in wild-type flies produced a phenocopy of MB-impaired flies, such that generalization was abolished and goal-directed actions were transformed into habitual responses.     

Speed/accuracy trade-off between the habitual and the goal-directed processes

Instrumental responses are hypothesized to be of two kinds: habitual and goal-directed, mediated by the sensorimotor and the associative cortico-basal ganglia circuits, respectively. The existence of the two heterogeneous associative learning mechanisms can be hypothesized to arise from the comparative advantages that they have at different stages of learning. In this paper, we assume that the goal-directed system is behaviourally flexible, but slow in choice selection. The habitual system, in contrast, is fast in responding, but inflexible in adapting its behavioural strategy to new conditions. Based on these assumptions and using the computational theory of reinforcement learning, we propose a normative model for arbitration between the two processes that makes an approximately optimal balance between search-time and accuracy in decision making. Behaviourally, the model can explain experimental evidence on behavioural sensitivity to outcome at the early stages of learning, but insensitivity at the later stages. It also explains that when two choices with equal incentive values are available concurrently, the behaviour remains outcome-sensitive, even after extensive training. Moreover, the model can explain choice reaction time variations during the course of learning, as well as the experimental observation that as the number of choices increases, the reaction time also increases. Neurobiologically, by assuming that phasic and tonic activities of midbrain dopamine neurons carry the reward prediction error and the average reward signals used by the model, respectively, the model predicts that whereas phasic dopamine indirectly affects behaviour through reinforcing stimulus-response associations, tonic dopamine can directly affect behaviour through manipulating the competition between the habitual and the goal-directed systems and thus, affect reaction time.     

Evidence for Habitual and Goal-Directed Behavior Following Devaluation of Cocaine: A Multifaceted Interpretation of Relapse

Background

Cocaine addiction is characterized as a chronically relapsing disorder. It is believed that cues present during self-administration become learned and increase the probability that relapse will occur when they are confronted during abstinence. However, the way in which relapse-inducing cues are interpreted by the user has remained elusive. Recent theories of addiction posit that relapse-inducing cues cause relapse habitually or automatically, bypassing processing information related to the consequences of relapse. Alternatively, other theories hypothesize that relapse-inducing cues produce an expectation of the drug''s consequences, designated as goal-directed relapse. Discrete discriminative stimuli signaling the availability of cocaine produce robust cue-induced responding after thirty days of abstinence. However, it is not known whether cue-induced responding is a goal-directed action or habit.

Methodology/Principal Findings

We tested whether cue-induced responding is a goal-directed action or habit by explicitly pairing or unpairing cocaine with LiCl-induced sickness (n = 7/group), thereby decreasing or not altering the value of cocaine, respectively. Following thirty days of abstinence, no difference in responding between groups was found when animals were reintroduced to the self-administration environment alone, indicating habitual behavior. However, upon discriminative stimulus presentations, cocaine-sickness paired animals exhibited decreased cue-induced responding relative to unpaired controls, indicating goal-directed behavior. In spite of the difference between groups revealed during abstinent testing, no differences were found between groups when animals were under the influence of cocaine.

Conclusions/Significance

Unexpectedly, both habitual and goal-directed responding occurred during abstinent testing. Furthermore, habitual or goal-directed responding may have been induced by cues that differed in their correlation with the cocaine infusion. Non-discriminative stimulus cues were weak correlates of the infusion, which failed to evoke a representation of the value of cocaine and led to habitual behavior. However, the discriminative stimulus–nearly perfectly correlated with the infusion–likely evoked a representation of the value of the infusion and led to goal-directed behavior. These data indicate that abstinent cue-induced responding is multifaceted, dynamically engendering habitual or goal-directed behavior. Moreover, since goal-directed behavior terminated habitual behavior during testing, therapeutic approaches aimed at reducing the perceived value of cocaine in addicted individuals may reduce the capacity of cues to induce relapse.     

Music and Video Gaming during Breaks: Influence on Habitual versus Goal-Directed Decision Making

Different systems for habitual versus goal-directed control are thought to underlie human decision-making. Working memory is known to shape these decision-making systems and their interplay, and is known to support goal-directed decision making even under stress. Here, we investigated if and how decision systems are differentially influenced by breaks filled with diverse everyday life activities known to modulate working memory performance. We used a within-subject design where young adults listened to music and played a video game during breaks interleaved with trials of a sequential two-step Markov decision task, designed to assess habitual as well as goal-directed decision making. Based on a neurocomputational model of task performance, we observed that for individuals with a rather limited working memory capacity video gaming as compared to music reduced reliance on the goal-directed decision-making system, while a rather large working memory capacity prevented such a decline. Our findings suggest differential effects of everyday activities on key decision-making processes.     

Optimal Behavioral Hierarchy

Human behavior has long been recognized to display hierarchical structure: actions fit together into subtasks, which cohere into extended goal-directed activities. Arranging actions hierarchically has well established benefits, allowing behaviors to be represented efficiently by the brain, and allowing solutions to new tasks to be discovered easily. However, these payoffs depend on the particular way in which actions are organized into a hierarchy, the specific way in which tasks are carved up into subtasks. We provide a mathematical account for what makes some hierarchies better than others, an account that allows an optimal hierarchy to be identified for any set of tasks. We then present results from four behavioral experiments, suggesting that human learners spontaneously discover optimal action hierarchies.     

Ethanol seeking by long evans rats is not always a goal-directed behavior

Background

Two parallel and interacting processes are said to underlie animal behavior, whereby learning and performance of a behavior is at first via conscious and deliberate (goal-directed) processes, but after initial acquisition, the behavior can become automatic and stimulus-elicited (habitual). With respect to instrumental behaviors, animal learning studies suggest that the duration of training and the action-outcome contingency are two factors involved in the emergence of habitual seeking of “natural” reinforcers (e.g., sweet solutions, food or sucrose pellets). To rigorously test whether behaviors reinforced by abused substances such as ethanol, in particular, similarly become habitual was the primary aim of this study.

Methodology/Principal Findings

Male Long Evans rats underwent extended or limited operant lever press training with 10% sucrose/10% ethanol (10S10E) reinforcement (variable interval (VI) or (VR) ratio schedule of reinforcement), or with 10% sucrose (10S) reinforcement (VI schedule only). Once training and pretesting were complete, the impact of outcome devaluation on operant behavior was evaluated after lithium chloride injections were paired with the reinforcer, or unpaired 24 hours later. After limited, but not extended instrumental training, lever pressing by groups trained under VR with 10S10E and under VI with 10S was sensitive to outcome devaluation. In contrast, responding by both the extended and limited training 10S10E VI groups was not sensitive to ethanol devaluation during the test for habitual behavior.

Conclusions/Significance

Operant behavior by rats trained to self-administer an ethanol-sucrose solution showed variable sensitivity to a change in the value of ethanol, with relative insensitivity developing sooner in animals that received time-variable ethanol reinforcement during training sessions. One important implication, with respect to substance abuse in humans, is that initial learning about the relationship between instrumental actions and the opportunity to consume ethanol-containing drinks can influence the time course for the development or expression of habitual ethanol seeking behavior.     

Dopamine Enhances Model-Based over Model-Free Choice Behavior

Decision making is often considered to arise out of contributions from a model-free habitual system and a model-based goal-directed system. Here, we investigated the effect of a dopamine manipulation on the degree to which either system contributes to instrumental behavior in a two-stage Markov decision task, which has been shown to discriminate model-free from model-based control. We found increased dopamine levels promote model-based over model-free choice.     

Effects of subclinical depression on prefrontal–striatal model-based and model-free learning

Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal–striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.     

Habitual vs Non-Habitual Manual Actions: An ERP Study on Overt Movement Execution

This study explored the neurophysiological mechanisms underlying the planning and execution of an overt goal-related handle rotation task. More specifically, we studied the neural basis of motor actions concerning the influence of the grasp choice. The aim of the present study was to differentiate cerebral activity between grips executed in a habitual and a non-habitual mode, and between specified and free grip choices. To our knowledge, this is the first study to differentiate cerebral activity underlying overt goal-related actions executed with a focus on the habitual mode. In a handle rotation task, participants had to use thumb-toward (habitual) or thumb-away (non-habitual) grips to rotate a handle to a given target position. Reaction and reach times were shorter for the habitual compared to the non-habitual mode indicating that the habitual mode requires less cognitive processing effort than the non-habitual mode. Neural processes for action execution (measured by event-related potentials (ERPs)) differed between habitual and non-habitual conditions. We found differential activity between habitual and non-habitual conditions in left and right frontal areas from −600 to 200 ms time-locked to reaching the target position. No differential neural activity could be traced for the specification of the grip. The results suggested that the frontal negativity reflected increased difficulty in movement precision control in the non-habitual mode compared to the habitual mode during the homing in phase of grasp and rotation actions.     

Humans but Not Chimpanzees Vary Face-Scanning Patterns Depending on Contexts during Action Observation

Human and nonhuman primates comprehend the actions of other individuals by detecting social cues, including others’ goal-directed motor actions and faces. However, little is known about how this information is integrated with action understanding. Here, we present the ontogenetic and evolutionary foundations of this capacity by comparing face-scanning patterns of chimpanzees and humans as they viewed goal-directed human actions within contexts that differ in whether or not the predicted goal is achieved. Human adults and children attend to the actor’s face during action sequences, and this tendency is particularly pronounced in adults when observing that the predicted goal is not achieved. Chimpanzees rarely attend to the actor’s face during the goal-directed action, regardless of whether the predicted action goal is achieved or not. These results suggest that in humans, but not chimpanzees, attention to actor’s faces conveying referential information toward the target object indicates the process of observers making inferences about the intentionality of an action. Furthermore, this remarkable predisposition to observe others’ actions by integrating the prediction of action goals and the actor’s intention is developmentally acquired.     

Dopamine receptors in the learning,memory and drug reward circuitry

As primary targets of a variety of abused drugs G-protein-coupled dopamine receptors in the brain play an important role in mediating the various drug-induced alterations in neural and psychological processes thought to underlie the transition from voluntary drug use to habitual and progressively compulsive drug-taking. This review considers the functional involvement of the five major dopamine receptor subtypes in drug reinforcement and reward and discusses the development of addiction as a series of learning transitions from initial goal-directed behaviour to pathological stimulus–response habits in which drug-seeking behaviours are automatically elicited and maintained by cues and stimuli associated with drug rewards.     

Young children with autism spectrum disorder use predictive eye movements in action observation

Does a dysfunction in the mirror neuron system (MNS) underlie the social symptoms defining autism spectrum disorder (ASD)? Research suggests that the MNS matches observed actions to motor plans for similar actions, and that these motor plans include directions for predictive eye movements when observing goal-directed actions. Thus, one important question is whether children with ASD use predictive eye movements in action observation. Young children with ASD as well as typically developing children and adults were shown videos in which an actor performed object-directed actions (human agent condition). Children with ASD were also shown control videos showing objects moving by themselves (self-propelled condition). Gaze was measured using a corneal reflection technique. Children with ASD and typically developing individuals used strikingly similar goal-directed eye movements when observing others’ actions in the human agent condition. Gaze was reactive in the self-propelled condition, suggesting that prediction is linked to seeing a hand–object interaction. This study does not support the view that ASD is characterized by a global dysfunction in the MNS.     

Do Domestic Dogs Understand Human Actions as Goal-Directed?

Understanding of other’s actions as goal-directed is considered a fundamental ability underlying cognitive and social development in human infants. A number of studies using the habituation-dishabituation paradigm have shown that the ability to discern intentional relations, in terms of goal-directedness of an action towards an object, appears around 5 months of age. The question of whether non-human species can perceive other’s actions as goal-directed has been more controversial, however there is mounting evidence that at least some primates species do. Recently domestic dogs have been shown to be particularly sensitive to human communicative cues and more so in cooperative and intentional contexts. Furthermore, they have been shown to imitate selectively. Taken together these results suggest that dogs may perceive others'' actions as goal-directed, however no study has investigated this issue directly. In the current study, adopting an infant habituation-dishabituation paradigm, we investigated whether dogs attribute intentions to an animate (a human) but not an inanimate (a black box) agent interacting with an object. Following an habituation phase in which the agent interacted always with one of two objects, two sets of 3 trials were presented: new side trials (in which the agent interacted with the same object as in the habituation trial but placed in a novel location) and new goal trials (in which the agent interacted with the other object placed in the old location). Dogs showed a similar pattern of response to that shown in infants, looking longer in the new goal than new side trials when they saw the human agent interact with the object. No such difference emerging with the inanimate agent (the black box). Results provide the first evidence that a non-primate species can perceive another individual’s actions as goal-directed. We discuss results in terms of the prevailing mentalisitic and non-mentalistic hypotheses regarding goal-attribution.     

The dopamine circuit as a reward-taxis navigation system

Studying the brain circuits that control behavior is challenging, since in addition to their structural complexity there are continuous feedback interactions between actions and sensed inputs from the environment. It is therefore important to identify mathematical principles that can be used to develop testable hypotheses. In this study, we use ideas and concepts from systems biology to study the dopamine system, which controls learning, motivation, and movement. Using data from neuronal recordings in behavioral experiments, we developed a mathematical model for dopamine responses and the effect of dopamine on movement. We show that the dopamine system shares core functional analogies with bacterial chemotaxis. Just as chemotaxis robustly climbs chemical attractant gradients, the dopamine circuit performs ‘reward-taxis’ where the attractant is the expected value of reward. The reward-taxis mechanism provides a simple explanation for scale-invariant dopaminergic responses and for matching in free operant settings, and makes testable quantitative predictions. We propose that reward-taxis is a simple and robust navigation strategy that complements other, more goal-directed navigation mechanisms.     

Associative and sensorimotor cortico‐basal ganglia circuit roles in effects of abused drugs

The mammalian forebrain is characterized by the presence of several parallel cortico‐basal ganglia circuits that shape the learning and control of actions. Among these are the associative, limbic and sensorimotor circuits. The function of all of these circuits has now been implicated in responses to drugs of abuse, as well as drug seeking and drug taking. While the limbic circuit has been most widely examined, key roles for the other two circuits in control of goal‐directed and habitual instrumental actions related to drugs of abuse have been shown. In this review we describe the three circuits and effects of acute and chronic drug exposure on circuit physiology. Our main emphasis is on drug actions in dorsal striatal components of the associative and sensorimotor circuits. We then review key findings that have implicated these circuits in drug seeking and taking behaviors, as well as drug use disorders. Finally, we consider different models describing how the three cortico‐basal ganglia circuits become involved in drug‐related behaviors. This topic has implications for drug use disorders and addiction, as treatments that target the balance between the different circuits may be useful for reducing excessive substance use.