Relapse processes after the extinction of instrumental learning: renewal, resurgence, and reacquisition

It is widely recognized that extinction (the procedure in which a Pavlovian conditioned stimulus or an instrumental action is repeatedly presented without its reinforcer) weakens behavior without erasing the original learning. Most of the experiments that support this claim have focused on several "relapse" effects that occur after Pavlovian extinction, which collectively suggest that the original learning is saved through extinction. However, although such effects do occur after instrumental extinction, they have not been explored there in as much detail. This article reviews recent research in our laboratory that has investigated three relapse effects that occur after the extinction of instrumental (operant) learning. In renewal, responding returns after extinction when the behavior is tested in a different context; in resurgence, responding recovers when a second response that has been reinforced during extinction of the first is itself put on extinction; and in rapid reacquisition, extinguished responding returns rapidly when the response is reinforced again. The results provide new insights into extinction and relapse, and are consistent with principles that have been developed to explain extinction and relapse as they occur after Pavlovian conditioning. Extinction of instrumental learning, like Pavlovian learning, involves new learning that is relatively dependent on the context for expression.     

Is extinction the hallmark of operant discrimination?: Reinforcement and S effects

Using a successive discrimination procedure with rats, three experiments investigated the contribution of reinforcement rate and amount of S(Delta) exposure on the acquisition of an operant discrimination. S(D) components and were always 2 min in length, while S(Delta) (extinction) components were either 1 min or 4 min in length; responses in S(D) were reinforced on one of four schedules. In Experiment 1, each of eight groups were exposed to one possible combination of rate of reinforcement and S(Delta) component length. At every level of reinforcement, the 4 min S(Delta) groups acquired the discrimination more quickly. However, within each level of reinforcement, the proportions of responding in S(D) as a function cumulative S(Delta) exposure were equivalent, regardless of the number of reinforcers earned in S(D), suggesting that extinction is the "hallmark" of discrimination. Experiment 2 sought to replicate these results in a within-subjects design, and although the 4 min S(Delta) conditions always produced superior discriminations, the lack of discriminated responding in some conditions suggested that stimulus disparity was reduced. Experiment 3 clarified those results and extended the finding that the acquisition of operant discrimination closely parallels extinction of responding in S(Delta). In sum, it appears that higher reinforcement rates and longer S(Delta) exposure facilitate the acquisition of discriminated operant responding.     

Pituitary-adrenal influences on avoidance behavior of pigs

Pigs submitted to extinction of a signaled conditioned avoidance response were injected daily with various doses of dexamethasone (DX) or ACTH. Pigs treated with 0.2 mg/kg of DX showed a higher number of intertrial crosses, but the extinction rate was not modified by either treatment. The effects of ACTH and DX were further studied on the reaction to a Pavlovian conditioned fear signal presented to pigs having learned a continuous avoidance response in a shuttle-box. DX treatment before both the fear conditioning and the test sessions enhanced the reaction to the fear signal at intermediate doses (0.2 mg/kg) but had little effect at lower (0.1 mg/kg) and higher doses (0.5 and 1 mg/kg). ACTH 1–24 treatment induced the same behavioral changes as intermediate doses of DX. A behaviorally active ACTH 4–9 analog, Org 2766, did not modify behavioral reaction to fear signal presentations when administered before fear conditioning and/or test sessions. These results demonstrate that, in pigs, avoidance performance changes under fear signal presentations are modulated by corticosteroids.     

Classical conditioning and retention in normal and mutantDrosophila melanogaster

Summary By changing the conditioned discrimination paradigm of Quinn et al. (1974) from an instrumental procedure to a classical (Pavlovian) one, we have demonstrated strong learning in type flies. About 150 flies were sequestered in a closed chamber and trained by explosing them sequentially to two odors in air currents. Flies received twelve electric shock pulses in the presence of the first odor (CS+) but not in the presence of the second odor (CS–). To test for conditioned avoidance responses, flies were transported to a Tmaze choice point, between converging currents of the two odors. Typically, 95% of trained flies avoided the shock-associated odor (CS+).Acquisition of learning was a function of the number of shock pulses received during CS+ presentation and was asymptotic within one training cycle. Conditioned avoidance increased with increasing shock intensity or odor concentration and was very resistant to extinction. Learning was best when CS+ presentations overlap shock (delay conditioning) and then decreased with increasing CS-US interstimulus intervals. Shocking flies immediately before CS+ presentation (backward conditioning) produced no learning. Nonassociative control procedures (CS Alone, US Alone and Explicitly Unpaired) produced slight decreases in avoidance responses, but these affected both odors equally and did not alter our associative learning index (A).Memory in wild-type flies decayed gradually over the first seven hours after training and still was present 24 h later. The mutantsamnesiac, rutabaga anddunce showed appreciable learning acquisition, but their memories decayed very rapidly during the first 30 min. After this, the rates of decay slowed sharply; conditioned avoidance still was measurable at least three hours after training.Abbreviations OCT 3-octanol - MCH 4-methylcyclohexanol - C-S Canton-Special - CS conditioned stimulus - US unconditioned stimulus     

Neural signatures of arbitration between Pavlovian and instrumental action selection

Pavlovian associations drive approach towards reward-predictive cues, and avoidance of punishment-predictive cues. These associations “misbehave” when they conflict with correct instrumental behavior. This raises the question of how Pavlovian and instrumental influences on behavior are arbitrated. We test a computational theory according to which Pavlovian influence will be stronger when inferred controllability of outcomes is low. Using a model-based analysis of a Go/NoGo task with human subjects, we show that theta-band oscillatory power in frontal cortex tracks inferred controllability, and that these inferences predict Pavlovian action biases. Functional MRI data revealed an inferior frontal gyrus correlate of action probability and a ventromedial prefrontal correlate of outcome valence, both of which were modulated by inferred controllability.     

Safety signals from avoidance learning but not from yoked classical conditioning training pass both summation and retardation tests for inhibition

In one experiment half of the animals were trained to avoid a signaled footshock by jumping (30 or 160 trials), whereas the rest of the animals received the same events as yoked. For all of them the termination of the warning signal and of the shock was followed by a safety signal. Several tests were conducted to assess the ability of the stimuli to suppress licking by measuring the latency in completing 25 consecutive licks in the presence of the stimuli. Fear of the warning signal and inhibitory properties of the safety signal (summation and retardation tests) were measured. The results showed that there were no differences in fear to the warning signal, and that the safety signal behaves as a conditioned inhibitor only for animals trained with a long avoidance procedure, but not in the yoked (classical conditioning) procedure. These results highlight the role played by the avoidance response and its consequences in avoidance learning.     

Electrodermal discrimination with visual feedback in two incompatible stressful situations. A yoked control comparison

One group (N = 14) of human volunteers received three sessions of discriminated avoidance and punishment with the skin resistance response (SRR) as the instrumental behavior. Each session consisted of three 7 minute periods of Sidman avoidance (response-stimulus [R-S] and stimulus-stimulus [S-S] = 40 sec) of a 0.5 second, 15 Hz square wave shock mixed with three periods of punishment with the same shock. The avoidance and punishment periods were differentially signaled by red and green lights, and a circle appeared superimposed on the discriminative stimuli during a criterion SRR. A second group (N = 14) was yoked to the first by recording shock events on magnetic tape and shocking the yoked subject in the same temporal sequence. In both groups the visual feedback stimulus was consistently related to the subject's ongoing electrodermal behavior. As in previous research, the contingent shock subjects made significantly more SRRs during avoidance than during punishment in all three sessions. For yoked subjects the discrimination did not appear until the third session. The findings imply that the discrimination is at least partly independent of the avoidance and punishment contingencies, and they raise questions about the role of the feedback stimulus.     

The role of contextual conditioning in the effect of reinforcer devaluation on instrumental performance by rats

Different groups of rats received different amounts of training to lever press for a food reinforcer before an aversion was conditioned to the food. This devaluation of the reinforcer reduced responding in both subsequent extinction and reinforced tests of responding to a degree that was independent of the amount of instrumental training. Moreover, interpolating context extinction between aversion conditioning and the extinction test reduced the magnitude of the devaluation effect, thereby indicating that Pavlovian contextual conditioning may play a role in the instrumental devaluation effect.     

Instrumental performance following saccharin pre-feeding

The effect of pre-feeding rats with a saccharin solution on instrumental performance, established with the same reinforcer, was studied in two experiments. Experiment 1 demonstrated that such pre-feeding depresses subsequent instrumental performance in an extinction test after training on a variable ratio schedule. In contrast, training on a variable interval schedule in Experiment 2 rendered instrumental performance impervious to pre-feeding. The motivational effects of pre-feeding with a non-nutritive substance appear to act through an incentive rather than drive process.     

Incentive or Habit Learning in Amphibians?

Toads (Rhinella arenarum) received training with a novel incentive procedure involving access to solutions of different NaCl concentrations. In Experiment 1, instrumental behavior and weight variation data confirmed that such solutions yield incentive values ranging from appetitive (deionized water, DW, leading to weight gain), to neutral (300 mM slightly hypertonic solution, leading to no net weight gain or loss), and aversive (800 mM highly hypertonic solution leading to weight loss). In Experiment 2, a downshift from DW to a 300 mM solution or an upshift from a 300 mM solution to DW led to a gradual adjustment in instrumental behavior. In Experiment 3, extinction was similar after acquisition with access to only DW or with a random mixture of DW and 300 mM. In Experiment 4, a downshift from DW to 225, 212, or 200 mM solutions led again to gradual adjustments. These findings add to a growing body of comparative evidence suggesting that amphibians adjust to incentive shifts on the basis of habit formation and reorganization.     

Autoshaping in the rat: conditioned licking response to a stimulus that signals sucrose reinforcement

The present experiments were designed to determine if repeated presentations of an empty sipper tube (the conditioned stimulus or CS) with the response-independent delivery of a sucrose solution (the unconditioned stimulus or US) from a second spout results in the development of Pavlovian conditioned responding. In Experiment 1, rats in the experimental condition received paired CS-US presentations whereas subjects in the control condition were exposed to random presentations of CS and US. In Experiment 2, an additional control condition (CS alone) was included and, to encourage generalized responding between the US and CS, the CS tube was filled with water for all groups. The results of both experiments indicate that the CS-directed responding in the paired CS-US condition was Pavlovian in nature. Thus, the present procedure serves as an autoshaping task in which conditioned licking is generated.     

Hormonal modulation of extinction responses induced by sexual steroid hormones in rats

A functional interrelation between the nervous and endocrine systems has been established. However, few studies have dealt with the effects of sexual steroids on learning and memory. The aim of this work was to determine whether sexual steroid hormones could modulate the extinction response of a passive avoidance conditioning in rats. Male Wistar rats, randomly assigned to five groups, two controls and three experimental groups, were submitted to a one-trial passive avoidance conditioning and tested for their retention 24 hr after and during 10 weeks. One control group received no treatment at all, the other received vegetable oil, and the three experimental received 20 mg of testosterone enanthate, 0.8 mg estradiol valerate, or 4 mg nandrolone decanoate, respectively. All substances were applied in a 0.3 ml volume, 24 hours before training and before testing for retention each week during 10 weeks. Results indicate that the extinction process is modulated by these hormones, since testosterone and estradiol facilitate extinction, whereas the anabolic androgen produced a resistance to the extinction process.     

Passage of time leaves response-outcome associations intact

The effect of the passage of time on the association between an instrumental response and its outcome was examined in three experiments with rats using outcome devaluation. In Experiments 1 and 2, rats were initially trained to make one pair of instrumental responses for different outcomes (e.g., lever press-->pellets and chain pull-->sucrose). Then, after a delay of several weeks, a second pair of responses for those different outcomes (e.g., nose poke-->pellets and handle pull-->sucrose) was trained in Experiment 1 but not in Experiment 2. One of the outcomes was then devalued and the responses were tested in extinction. There was no evidence that the passage of time eliminated the sensitivity of a response to a change in the value of its outcome. Experiment 3 used the same design as Experiment 1 to examine the effect of the passage of time on extinguished instrumental responses. First, one pair of responses was trained with different outcomes and then extinguished prior to the training and subsequent extinction of a second pair of responses. After retraining both pairs of responses with a polycose outcome, one of the original outcomes was devalued. Each pair of responses was then tested in extinction. The results of the extinction tests revealed no evidence that the passage of time removed the sensitivity of an extinguished response to devaluation of its original outcome. These findings are consistent with other reports that the response-outcome association survives various response elimination techniques.     

Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding

Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.     

Reduction of instrumental discrimination performance by post-conditioning devaluation of discriminative stimulus: the effects of novelty in reinforcing outcome and extended training

In three experiments the effects of post-conditioning pairings of a discriminative stimulus (Sd) with an illness-inducing agent (lithium chloride, LiCl) on subsequent discrimination performance in extinction and consumption of reinforcing outcome were investigated. Rats were trained to choose a correct lever to obtain food pellets, with a light presented on a bulb just above the correct lever serving for the Sd on each trial. After achievement of a criterion of the discrimination, animals received paired or unpaired presentations of the Sds and LiCl injection. In Experiment 1, in which a familiar outcome was given throughout the discrimination training, Sd-LiCl pairings did not reduce either lever-press performance during presentation of the Sds or amount of consumption of outcomes. On the other hand, in Experiment 2 where a novel outcome was introduced in the final two sessions of the discrimination training, subsequent Sd devaluation reduced lever-press performance during presentations of the Sds. Similar findings were obtained in Experiment 3, in which animals were given extended discrimination training with introduction of novel outcomes in the final two sessions. These findings suggest that a representation of the outcome, evoked by presentation of the Sd, and illness were associated in the course of Sd-LiCl pairings but only when a novel outcome was used.     

Perceptual learning in an appetitive Pavlovian procedure: Analysis of the effectiveness of the common element

Non-reinforced preexposure to two stimuli often enhances discrimination between them. Analyses of this perceptual learning phenomenon have mainly focused on the role played by the distinctive stimulus features; this study examined the contribution of the non-distinctive common elements. A standard appetitive Pavlovian procedure was used. Rats received two different schedules of exposure - alternated or blocked - to two compound auditory stimuli, AX and BX. In Experiment 1 a generalization test to BX that followed conditioning to AX showed that animals responded less, and hence discriminated better, following alternated exposure, thus extending the generality of this perceptual learning effect to standard appetitive Pavlovian procedures. The degree to which the common element X was mediating this effect was explored in the next three experiments. Experiment 2 assessed the effectiveness of X following conditioning to AX. Experiment 3 explored X's effectiveness throughout extensive conditioning to X. Experiment 4 tested the ability of X to overshadow a novel stimulus Y. The results were consistent with the suggestion that alternated preexposure can reduce the relative effectiveness of the common element.     

Interval duration effects on blocking in appetitive conditioning

Three experiments examined absolute (Experiment 1) and relative (Experiments 2a and 2b) duration effects on blocking. In Experiment 1, rats were pretrained with a short or long conditioned stimulus (CS1) followed by food, after which they were given reinforced short-short or long-long CS1-CS2 simultaneous compounds. Compared to overshadowing control groups, both pretrained groups displayed blocking, and there was no clear effect of absolute stimulus duration on the magnitude of blocking. In Experiments 2a and 2b, the rats received partially overlapping short-long CS1-CS2 compounds. In both experiments, a long CS1 blocked a short CS2, but not vice versa. This was the case when the long CS1 was nine times (Experiment 2a) or only 1.5 times (Experiment 2b) the duration of the short CS2. The pattern of results is most consistent with a real-time model of conditioning, such as the Sutton and Barto [Sutton, R.S., Barto, A.G., 1990. Time derivative models of Pavlovian reinforcement. In: Gabriel, M.R., Moore, J.W. (Eds.), Learning and Computational Neuroscience: Foundations of Adaptive Networks. MIT Press, Cambridge, MA, 497-537] temporal difference model.     

Delay inhibition of the active avoidance reflex and intertrial responses in rats in the shuttle box

A diminished probability of avoidance response in early phases of a warning signal was revealed with salient signals given after short intertrial intervals. The inhibition of the delay in avoidance response is due to an interaction of the safety state conditioning and the excitation elicited by onset of warning signal.     

Retroactive interference after discrimination reversal decreases following temporal and physical context changes in human subjects

One experiment was conducted to test the additive effects of physical context changes and the passage of time on a retroactive interference task in human subjects. Participants learned a discrimination in a symbolic matching to sample situation within a specific context. The discrimination was subsequently reversed. The context in which the reversal occurred was combined factorially with the passage of time before the test. All testing was conducted in the context in which the original discrimination was acquired. Participants had received the discrimination reversal in either a context different from that in which the original discrimination was acquired, or in the same context. Half of each of the groups mentioned above received testing immediately after reversal training and the other half received testing 48 h later. Both manipulations, changing the context after the reversal and the passage of time following the reversal, led to a recovery of the original discrimination performance. Participants that received both a context change and retention interval showed the largest recovery.     

Postnatal development of conditioned reflex behavior: comparison of the periods related to the onset of the different forms of hippocampal plasticity in rats

Fear conditioning, escape and active avoidance reactions in two-way avoidance paradigm were compared in rats of different ages. Fear conditioning, but not escape and active avoidance reactions could be acquired on the 16-17th postnatal days, and the acquisition was more effective than in adults. Escape behavior matured beginning from the 18th postnatal day reaching the adult level within the 3d-4th postnatal weeks. Maturation of the mechanisms of Pavlovian (fear reaction) and instrumental (escape reaction) conditioning did not facilitate the acquisition of two-way avoidance until the 4th postnatal week, young animals displayed low acquisition in this period. The maturation of these memory processes is proposed to be related to developmental stages of different mechanisms of hippocampal plasticity.