Relapse processes after the extinction of instrumental learning: renewal, resurgence, and reacquisition

It is widely recognized that extinction (the procedure in which a Pavlovian conditioned stimulus or an instrumental action is repeatedly presented without its reinforcer) weakens behavior without erasing the original learning. Most of the experiments that support this claim have focused on several "relapse" effects that occur after Pavlovian extinction, which collectively suggest that the original learning is saved through extinction. However, although such effects do occur after instrumental extinction, they have not been explored there in as much detail. This article reviews recent research in our laboratory that has investigated three relapse effects that occur after the extinction of instrumental (operant) learning. In renewal, responding returns after extinction when the behavior is tested in a different context; in resurgence, responding recovers when a second response that has been reinforced during extinction of the first is itself put on extinction; and in rapid reacquisition, extinguished responding returns rapidly when the response is reinforced again. The results provide new insights into extinction and relapse, and are consistent with principles that have been developed to explain extinction and relapse as they occur after Pavlovian conditioning. Extinction of instrumental learning, like Pavlovian learning, involves new learning that is relatively dependent on the context for expression.     

Whisking as a "voluntary" response: operant control of whisking parameters and effects of whisker denervation

The rat's ability to vary its whisking "strategies" to meet the functional demands of a discriminative task suggests that whisking may be characterized as a "voluntary" behavior--an operant--and like other operants, should be modifiable by appropriate manipulations of response-reinforcer contingencies. To test this hypothesis we have used high-resolution, optoelectronic "real-time" recording procedures to monitor the movements of individual whiskers and reinforce specific movement parameters (amplitude, frequency). In one operant paradigm (N = 9) whisks with protractions above a specified amplitude were reinforced (Variable Interval 30 s) in the presence of a tone, but extinguished (EXT) in its absence. In a second paradigm (N = 3), rats were reinforced on two different VI schedules (VI-20s/VI-120s) signaled, respectively, by the presence or absence of the tone. Selective reinforcement of whisking movements maintained the behavior over many weeks of testing and brought it under stimulus and schedule control. Subjects in the first paradigm learned to increase responding in the presence of the tone and inhibit responding in its absence. In the second paradigm, subjects whisked at significantly different rates in the two stimulus conditions. Bilateral deafferentation of the whisker pad did not impair conditioned whisking or disrupt discrimination behavior. Our results confirm the hypothesis that rodent whisking has many of the properties of an operant response. The ability to bring whisking movement parameters under operant control should facilitate electrophysiological and lesion/behavioral studies of this widely used "model" sensorimotor system.     

Stimulus change dis-habituates operant responding supported by water reinforcers

The present experiment examined whether habituation contributes to within-session decreases in operant responding for water reinforcers. The experiment asked if this responding can be dishabituated, a fundamental property of habituated behavior. During baseline, rats’ lever pressing was reinforced by water on a variable interval 15-s schedule. During experimental conditions, rats responded on the same schedule and a new stimulus was introduced for 5 min at 15, 30 or 45 min into the 60-min session. The new stimulus was extinction, continuous reinforcement or flashing lights in different conditions. Rate of responding primarily decreased within the session during baseline. Introducing a new stimulus sometimes suppressed (extinction, continuous reinforcement) and sometimes increased (flashing lights) responding while it was in effect. The new stimulus increased responding after it ended and before it was presented in the session. The results are incompatible with the idea that non-habituation satiety factors (e.g., cellular hydration and blood volume) contributed to within-session changes in responding. These satiety factors should increase with increases in consumption, decrease with decreases in consumption and remain constant with constant consumption of water. Nevertheless, all stimulus changes increased operant responding for water. These results support the idea that habituation contributes to within-session decreases in responding for water reinforcers.     

Extinguished operant responding shows stimulus specificity

The experiment tested for stimulus specificity in extinguished operant responding. Eight pigeons pecked keys for food reinforcers delivered by a variable interval (VI) 60-s schedule. The key was illuminated with red light during some sessions and white light during others. Then, responding was placed on extinction. During some sessions of extinction, the color of the key light remained constant throughout the session (red or white). During other sessions the color changed at 30 min into the session (red to white or white to red). Response rate increased after the change of key color in extinction. If it is assumed that key color is part of the stimulus to which subjects habituate, then these results are consistent with McSweeney and Swindell's [J. Gen. Psychol. 129 (2002) 364] suggestion that responding declines in extinction partly because subjects habituate to the stimuli that support conditioned responding. Habituation is relatively specific to the exact nature of the stimulus presented. Therefore, changes in the stimulus violate stimulus specificity and restore habituated responding. The results are also consistent with other theories that attribute extinction to a reduction of stimulus control [e.g., Psychol. Bull. 114 (1993) 80; J. Exp. Psychol.: Anim. Behav. Process. 16 (1990) 235], but considerations such as parsimony and testability favor the habituation hypothesis over these theories.     

Reinstatement of a food-maintained operant produced by compounding discriminative stimuli

After a tone and a light were established as discriminative stimuli for food-reinforced responding in rats, presenting these stimuli simultaneously produced over three times as many responses as either the tone or light alone. Following this stimulus compounding test, responses during the tone and during the light were not reinforced (extinction) for 20 sessions, essentially eliminating responding. On stimulus compounding tests administered after the 10th and 20th extinction sessions, tone-plus-light continued to produce significantly more responding than the tone or light alone. The compound even produced responses when the individual stimuli no longer did. These results suggest that the simultaneous presentation of multiple extinguished discriminative stimuli may also contribute to the reinstatement of other positively-reinforced behaviors, such as drug taking.     

A general method for evaluating incubation of sucrose craving in rats

For someone on a food-restricted diet, food craving in response to food-paired cues may serve as a key behavioral transition point between abstinence and relapse to food taking. Food craving conceptualized in this way is akin to drug craving in response to drug-paired cues. A rich literature has been developed around understanding the behavioral and neurobiological determinants of drug craving; we and others have been focusing recently on translating techniques from basic addiction research to better understand addiction-like behaviors related to food. As done in previous studies of drug craving, we examine sucrose craving behavior by utilizing a rat model of relapse. In this model, rats self-administer either drug or food in sessions over several days. In a session, lever responding delivers the reward along with a tone+light stimulus. Craving behavior is then operationally defined as responding in a subsequent session where the reward is not available. Rats will reliably respond for the tone+light stimulus, likely due to its acquired conditioned reinforcing properties. This behavior is sometimes referred to as sucrose seeking or cue reactivity. In the present discussion we will use the term "sucrose craving" to subsume both of these constructs. In the past decade, we have focused on how the length of time following reward self-administration influences reward craving. Interestingly, rats increase responding for the reward-paired cue over the course of several weeks of a period of forced-abstinence. This "incubation of craving" is observed in rats that have self-administered either food or drugs of abuse. This time-dependent increase in craving we have identified in the animal model may have great potential relevance to human drug and food addiction behaviors. Here we present a protocol for assessing incubation of sucrose craving in rats. Variants of the procedure will be indicated where craving is assessed as responding for a discrete sucrose-paired cue following extinction of lever pressing within the sucrose self-administration context (Extinction without cues) or as responding for sucrose-paired cues in a general extinction context (Extinction with cues).     

Contrasting the overexpectation and extinction effects

After many target stimulus (X)-unconditioned stimulus (US) pairings, further conditioning of X in the presence of another well-established signal for the US (A) disrupts X's behavioral control. Some researchers have argued that the mechanism underlying this so-called overexpectation effect is similar to that underlying extinction (a reduction in X's behavioral control due to X-alone presentations). Three conditioned suppression experiments with rats as subjects compared overexpectation and extinction. Experiment 1 replicated the basic overexpectation effect by showing that A disrupts responding to X more than does a previously neutral stimulus. Experiment 2 found that posttraining context exposure disrupts extinction but not overexpectation. Experiment 3 suggested that overexpectation and extinction are differentially sensitive to the effects of overtraining (compound reinforced or nonreinforced, respectively), such that extinction is enhanced by increases in the amount of nonreinforced trials and overexpectation is unaffected. These results are inconsistent with the view that overexpectation and extinction are driven by a common mechanism.     

Whisking as a “voluntary” response: operant control of whisking parameters and effects of whisker denervation

The rat’s ability to vary its whisking “strategies” to meet the functional demands of a discriminative task suggests that whisking may be characterized as a “voluntary” behavior—an operant—and like other operants, should be modifiable by appropriate manipulations of response–reinforcer contingencies. To test this hypothesis we have used high-resolution, optoelectronic “real-time” recording procedures to monitor the movements of individual whiskers and reinforce specific movement parameters (amplitude, frequency). In one operant paradigm (N = 9) whisks with protractions above a specified amplitude were reinforced (Variable Interval 30?s) in the presence of a tone, but extinguished (EXT) in its absence. In a second paradigm (N = 3), rats were reinforced on two different VI schedules (VI-20s/VI-120s) signaled, respectively, by the presence or absence of the tone. Selective reinforcement of whisking movements maintained the behavior over many weeks of testing and brought it under stimulus and schedule control. Subjects in the first paradigm learned to increase responding in the presence of the tone and inhibit responding in its absence. In the second paradigm, subjects whisked at significantly different rates in the two stimulus conditions. Bilateral deafferentation of the whisker pad did not impair conditioned whisking or disrupt discrimination behavior. Our results confirm the hypothesis that rodent whisking has many of the properties of an operant response. The ability to bring whisking movement parameters under operant control should facilitate electrophysiological and lesion/behavioral studies of this widely used “model” sensorimotor system.     

The role of contextual conditioning in the effect of reinforcer devaluation on instrumental performance by rats

Different groups of rats received different amounts of training to lever press for a food reinforcer before an aversion was conditioned to the food. This devaluation of the reinforcer reduced responding in both subsequent extinction and reinforced tests of responding to a degree that was independent of the amount of instrumental training. Moreover, interpolating context extinction between aversion conditioning and the extinction test reduced the magnitude of the devaluation effect, thereby indicating that Pavlovian contextual conditioning may play a role in the instrumental devaluation effect.     

Effects of an extinguished CS on competition with another CS

Three experiments were conducted using a conditioned taste aversion procedure with rats to examine the effect of nonreinforced presentations of a conditioned stimulus (CS) on its ability to compete with a target stimulus for manifest conditioned responding. Two CSs (A and B) were presented in a serial compound and then paired with the unconditioned stimulus. CS A was first paired with the US and then presented without the US (i.e., extinction) prior to reinforced presentation of the AB compound. Experiment 1 showed that A was poor at competing with B for conditioned responding when given conditioning and extinction prior to reinforcement of AB relative to a group that received both A and B for the first time during compound conditioning. That is, an extinguished A stimulus allowed greater manifest acquisition to B. Experiment 2 found that extinction treatment produced a poor CR to the pretrained and extinguished CS itself following compound conditioning. Experiment 3 found that interposing a retention interval after extinction of A and prior to compound conditioning enhanced A's ability to compete with B. The results of these experiments are discussed with regard to different theories of extinction and associative competition.     

Contextual fear conditioning differs for infant, adolescent, and adult rats

Contextual fear conditioning was tested in infant, adolescent, and adult rats in terms of Pavlovian-conditioned suppression. When a discrete auditory-conditioned stimulus (CS) was paired with footshock (unconditioned stimulus, US) within the largely olfactory context, infants and adolescents conditioned to the context with substantial effectiveness, but adult rats did not. When unpaired presentations of the CS and US occurred within the context, contextual fear conditioning was strong for adults, weak for infants, but about as strong for adolescents as when pairings of CS and US occurred in the context. Nonreinforced presentations of either the CS or context markedly reduced contextual fear conditioning in infants, but, in adolescents, CS extinction had no effect on contextual fear conditioning, although context extinction significantly reduced it. Neither CS extinction nor context extinction affected responding to the CS–context compound in infants, suggesting striking discrimination between the compound and its components. Female adolescents showed the same lack of effect of component extinction on response to the compound as infants, but CS extinction reduced responding to the compound in adolescent males, a sex difference seen also in adults. Theoretical implications are discussed for the development of perceptual-cognitive processing and hippocampus role.     

A hippocampal insulin-growth factor 2 pathway regulates the extinction of fear memories

Extinction learning refers to the phenomenon that a previously learned response to an environmental stimulus, for example, the expression of an aversive behaviour upon exposure to a specific context, is reduced when the stimulus is repeatedly presented in the absence of a previously paired aversive event. Extinction of fear memories has been implicated with the treatment of anxiety disease but the molecular processes that underlie fear extinction are only beginning to emerge. Here, we show that fear extinction initiates upregulation of hippocampal insulin-growth factor 2 (Igf2) and downregulation of insulin-growth factor binding protein 7 (Igfbp7). In line with this observation, we demonstrate that IGF2 facilitates fear extinction, while IGFBP7 impairs fear extinction in an IGF2-dependent manner. Furthermore, we identify one cellular substrate of altered IGF2 signalling during fear extinction. To this end, we show that fear extinction-induced IGF2/IGFBP7 signalling promotes the survival of 17-19-day-old newborn hippocampal neurons. In conclusion, our data suggest that therapeutic strategies that enhance IGF2 signalling and adult neurogenesis might be suitable to treat disease linked to excessive fear memory.     

Simple discrete-time metapopulation models of patch occupancy

Simple models in theoretical ecology have a long-standing history of being used to understand how specific processes influence population dynamics as well as providing a foundation for future endeavors. The Levins model is the seminal example of this for continuous-time metapopulation dynamics. However, many natural populations have a distinct separation between processes and data is not collected continuously leading to the need for using a discrete-time model. Our goal is to develop a simple discrete-time metapopulation model of patch occupancy using difference equations. In our formulation, we consider the two fundamental processes of colonization and extinction that will be treated as sequential events and will only consider patch occupancy. To achieve this, we use a composition of two functions where one will reflect the extinction process and the other for the colonization process. Under some mild assumptions, we are able determine the dynamic behavior of the metapopulation. In addition, we provide numerous examples for the functions used to emulate the colonization and extinction processes. Our results illustrate that the dynamics of the model are tied to properties such as convexity and monotonicity of the colonization and extinction functions. In particular, if the model is non-monotone, then complex dynamics can arise such as cyclic and even chaotic behavior. Overall, our approach shows how certain properties of the colonization and extinction functions can influence metapopulation dynamics.     

60 Hz electric fields and incandescent light as aversive stimuli controlling the behavior of rats responding under concurrent schedules of reinforcement

Several reports have shown that animals will sometimes engage in behaviors that reduce their exposure to a 60 Hz electric field (E-field). The field, therefore, can function as an aversive stimulus. In other studies, the E-field at equivalent strengths failed to function as an aversive stimulus. The present experiment, using rats, demonstrates how factors other than field strength can influence whether a subject engages in behavior that reduces field exposure. The general design consisted of giving the rat a choice between two alternatives, one of which sometimes included an added stimulus. Each subject was trained to press each of two levers to obtain food. Pressing one lever was reinforced intermittently under a variable interval 2 min schedule (VI 2); pressing the other lever was reinforced by a second VI 2 schedule operating independently of the first. Under this concurrent schedule the rat spent 50% of the daily 50 min session responding to each of the levers, indicating that they were equally “valued.” Next, while the schedules remained in effect, the first response to one of the levers turned on a 100 kV/m E-field which remained on until the rat pressed the other lever. The time spent responding under the schedule associated with the field was reduced by about 5–10%. When the procedure was changed so that no lever presses produced food, i.e., extinction, but the added stimulus contingency remained, the rats spent even less time in the presence of the field. Similar outcomes were observed during both the concurrent food or extinction schedules when incandescent light was used. Thus, both an E-field and incandescent light functioned as aversive stimuli, but the magnitude of the aversiveness was small. Aversiveness depended not only on stimulus intensity, but also on behavioral factors. Bioelectromagnetics 19:210–221, 1998. © 1998 Wiley-Liss, Inc.     

Effects of peripherally injected vasopressin and des-glycinamide vasopressin on the extinction of a spatial learning task in rats

An elevated eight-arm radial maze was employed to study the effects of neuropeptide administration on the spatial learning abilities of food-deprived rats. Following 18 days of reinforced training, each animal was briefly exposed to the maze with no food available in any of the eight food-cups. Immediately after this preliminary trial, animals were injected with a single subcutaneous dose of either saline, arginine vasopressin (AVP: 1.0 or 5.0 micrograms/kg), or an AVP analog with only weak endocrinological activity, des-gly-arginine vasopressin (DG-AVP: 1.0, 5.0 or 10.0 micrograms/kg). Additional extinction trials were conducted at 2, 4, 6 and 8 h post-injection. These tests consisted of individually placing an animal on the empty maze and recording the number of arms chosen in a 5-min period. In this situation, animals learn that food is no longer present in the maze and, consequently, extinguish responding. Vasopressin potentiated this radial maze extinction behavior while DG-AVP produced behavioral results directionally opposite to those predicted by a memory facilitation hypothesis. In a subsequent experiment, vasopressin had no effects on unconditioned locomotor activity measured 2 and 4 h post-injection. These results suggest that: vasopressin improved the learning that occurred during extinction of conditioned appetitive behaviors, these vasopressin effects on conditioned behavior were independent of any unconditioned, sedative or non-specific actions of the peptide, and peripheral endocrinological responses may be necessary to demonstrate memory-enhancing effects following peripherally administered AVP.     

The effects of naloxone,diprenorphine, and diazepam on responding suppressed by pre-shock and pre-food stimuli

The lever-pressing of rats was reinforced with food according to a variable-interval 1-min schedule. In one group, occasional illumination of cue lights for 30-sec periods was followed by a brief electric shock; responding was suppressed during these periods. Naloxone (0.01–10 mg/kg) did not change the degree to which responding was suppressed during the pre-shock stimulus. Diprenorphine (0.1–10 mg/kg) slightly attenuated suppression, and diazepam (1.0–3.0 mg/kg) increased responding during the stimulus to normal levels. These results confirm that opiate antagonists do not always enhance the effects of shock on behavior. In a second group, occasional illumination of the cue lights for 20-sec periods was followed by delivery of free food pellets. Responding was also suppressed during the pre-food stimulus. Neither naloxone nor diprenorphine had any effect on response rate during this stimulus. In contrast to the results of earlier studies using benzodiazepines, diazepam (1.0–3.0 mg/kg) produced a marked attenuation of response suppression during the pre-food stimulus.     

The extinction context enables extinction performance after a change in context

One experiment with human participants determined the extent to which recovery of extinguished responding with a context switch was due to a failure to retrieve contextually controlled learning, or some other process such as participants learning that context changes signal reversals in the meaning of stimulus-outcome relationships. In a video game, participants learned to suppress mouse clicking in the presence of a stimulus that predicted an attack. Then, that stimulus underwent extinction in a different context (environment within the game). Following extinction, suppression was recovered and then extinguished again during testing in the conditioning context. In a final test, participants that were tested in the context where extinction first took place showed less of a recovery than those tested in a neutral context, but they showed a recovery of suppression nevertheless. A change in context tended to cause a change in the meaning of the stimulus, leading to recovery in both the neutral and extinction contexts. The extinction context attenuated that recovery, perhaps by enabling retrieval of the learning that took place in extinction. Recovery outside an extinction context is due to a failure of the context to enable the learning acquired during extinction, but only in part.     

Differential resurgence and response elimination

Resurgence refers to the transient recovery of previously reinforced, but presently not reinforced, responding when more recently reinforced responding is extinguished. The primary purpose of our research was to determine how differential resurgence results from the procedures used to eliminate that responding. There were three conditions in each of five experiments. In Condition 1, key pecking by pigeons was maintained under a two-component multiple variable-interval (VI) 30-s VI 30-s schedule. In Condition 2, this pecking was eliminated in different ways across components. In Condition 3, extinction was in effect for all responses, and resurgence of key pecking was compared across components. These three conditions were repeated for most pigeons, and the procedures used to eliminate responding in Condition 2 varied across experiments. In Experiment 1, there was greater resurgence, and an earlier onset of it, after a differential-reinforcement-of-other-behavior (DRO) schedule than after a VI schedule was correlated with pecking an alternative key. Experiments 2 and 3 showed that the differential resurgence in Experiment 1 probably was not due to conditional stimulus control or the periodicity of food delivery, respectively. In Experiment 4, there was no systematic difference in resurgence after either a DRO schedule or a VI schedule correlated with treadle pressing. In Experiment 5, there was greater resurgence, and/or an earlier onset of it, after a VI schedule correlated with treadle pressing than after a VI schedule correlated with pecking an alternative key. Taken together, the results showed that the reinforcement of an alternative key-peck response was the most effective means of reducing subsequent key-peck resurgence. The relation of these results to an understanding of resurgence is discussed.     

Intertrial interval as a contextual stimulus

Four experiments with rats investigated whether the time between appetitive conditioning trials can serve as a discriminative cue for responding during the next conditional stimulus (CS). In Experiment 1, rats that received extinction trials with a 4-min intertrial interval (ITI) showed spontaneous recovery after a retention interval of 16 min, whereas rats that received extinction with a 16-min ITI did not. Experiments 2 and 3 investigated more explicit discriminations between the 4- and 16-min ITIs. When a 16-min ITI signaled that the CS would be reinforced and a 4-min ITI signaled that it would not, the ITIs modulated responding to the CS. But when the 4-min ITI signaled reinforcement and the 16-min ITI did not, there was less evidence of modulation by the ITIs. This asymmetry was due at least partly to a difficulty in performance rather than learning. Experiment 4 investigated similar discriminations with 1- and 4-min ITIs. Here the results took a different form: time in the reinforced ITI elicited responding directly, but did not modulate responding to the CS. ITI can function as a contextual cue, and the results suggest new similarities between the processes behind interval timing and associative learning.     

Relative temporal representations in Pavlovian conditioning

The transfer of relative temporal representations was assessed in a series of three experiments. In each experiment, rats (Rattus norvegicus) received one set of conditioned stimulus (CS) and intertrial interval (ITI) durations in Phase 1 and another set in Phase 2. The ratio between the CS and ITI intervals was either changed or maintained across phases. On the hypothesis that relative temporal representations are learned, groups receiving maintained temporal ratios across phases were expected to display greater change in responding upon encountering the new intervals. When the CS duration decreased across phases, maintaining the temporal ratio did lead to greater change in Day 1 of Phase 2 towards the final pattern of responding. However, when the CS increased across phases, maintaining the temporal ratio across phases did not facilitate adjustment to the new intervals, suggesting that extinction of previously reinforced times induced new learning. These results provide evidence that under some conditions, relative relationships in temporal maps may survive transformation-of-scale, like relative relationships in spatial maps.