Amygdala-prefrontal synchronization underlies resistance to extinction of aversive memories

Emotional memories can persist for a lifetime but can also undergo extinction. Although we know about the mechanisms involved in expression and extinction, we know very little about the mechanisms that determine whether a specific memory would persist or not. Here, we use partial reinforcement extinction effect (PREE) to explore the neural mechanisms that differentiate persistent from labile memories. We recorded the simultaneous activity of neurons in the amygdala and the dorsal anterior cingulate cortex (dACC) while monkeys engaged in tone-odor aversive conditioning. We report that under continuous reinforcement schedule (ConS), activity in the amygdala precedes behavioral response, whereas under partial schedule (ParS), dACC activity precedes it. Moreover, we find that ParS induced cross-regional pairwise correlations throughout the entire acquisition session, and their magnitude and precision predicted the later resistance to extinction. Our results suggest that memory persistence depends on distributed representations and, specifically, resistance to extinction of aversive memories is maintained by correlated amygdala-dACC activity.     

Effects of inter-trial interval length on food-hoarding partial reinforcement of running behaviour in the golden hamster

Food-hoarding provides an adequate motivation in sated hamsters for the acquisition of a two-way running response. This learning was studied using a discrete-trial procedure, in continuous (CR) and partial reinforcement (PR) conditions, with two different inter-trial interval (ITI) lengths. The dependent variables were: the time spent by Ss in the goal section of the runway, and the number of their runs during extinction.The PR training had two effects on goal times: a slow decrease in acquisition on non-reinforced trials, and then a stabilization at this level during the extinction phase, as contrasted with the sudden increase found in CR-trained Ss when they were switched from acquisition to extinction conditions. However, the PR effects on number of runs depended upon ITI length: resistance to extinction of PR-trained Ss was superior to that of CR-trained Ss with spaced, but not with massed, trials. In the latter case, CR-trained Ss persisted as much as did PR-trained Ss. An hypothesis is offered, along the lines of the Frustration theory.     

Effects of an extinguished CS on competition with another CS

Three experiments were conducted using a conditioned taste aversion procedure with rats to examine the effect of nonreinforced presentations of a conditioned stimulus (CS) on its ability to compete with a target stimulus for manifest conditioned responding. Two CSs (A and B) were presented in a serial compound and then paired with the unconditioned stimulus. CS A was first paired with the US and then presented without the US (i.e., extinction) prior to reinforced presentation of the AB compound. Experiment 1 showed that A was poor at competing with B for conditioned responding when given conditioning and extinction prior to reinforcement of AB relative to a group that received both A and B for the first time during compound conditioning. That is, an extinguished A stimulus allowed greater manifest acquisition to B. Experiment 2 found that extinction treatment produced a poor CR to the pretrained and extinguished CS itself following compound conditioning. Experiment 3 found that interposing a retention interval after extinction of A and prior to compound conditioning enhanced A's ability to compete with B. The results of these experiments are discussed with regard to different theories of extinction and associative competition.     

Oscillations following periodic reinforcement

Three experiments examined behavior in extinction following periodic reinforcement. During the first phase of Experiment 1, four groups of pigeons were exposed to fixed interval (FI 16 s or FI 48 s) or variable interval (VI 16 s or VI 48 s) reinforcement schedules. Next, during the second phase, each session started with reinforcement trials and ended with an extinction segment. Experiment 2 was similar except that the extinction segment was considerably longer. Experiment 3 replaced the FI schedules with a peak procedure, with FI trials interspersed with non-food peak interval (PI) trials that were four times longer. One group of pigeons was exposed to FI 20 s PI 80 s trials, and another to FI 40 s PI 160 s trials. Results showed that, during the extinction segment, most pigeons trained with FI schedules, but not with VI schedules, displayed pause-peck oscillations with a period close to, but slightly greater than the FI parameter. These oscillations did not start immediately after the onset of extinction. Comparing the oscillations from Experiments 1 and 2 suggested that the alternation of reconditioning and re-extinction increases the reliability and earlier onset of the oscillations. In Experiment 3 the pigeons exhibited well-defined pause-peck cycles since the onset of extinction. These cycles had periods close to twice the value of the FI and lasted for long intervals of time. We discuss some hypotheses concerning the processes underlying behavioral oscillations following periodic reinforcement.     

Stimulus magnitude effects on the extinction of conditioned consummatory responses to flavored sucrose solution in mice

Paradoxical extinction effects in the conditioned consummatory behavior of rodents have remained largely elusive. Here, appetitive flavor conditioning was studied to determine if a paradoxical magnitude of reinforcement extinction effect (MREE) can occur in the consummatory behavior of mice. During acquisition training of two experiments with factorial design, animals received daily access to either 32% or 4% sucrose solution, and goal tracking time was measured in one-minute bins. In Experiment 1 the solutions were flavored with either 5% or 0.5% almond essence and in Experiment 2 with 2% almond essence, but combined with continuous or partial schedules of reinforcement. During extinction tests of Experiment 1 and 2, water flavored with 0.5% or 2% almond was presented, respectively. Consummatory performance decreased more abruptly during the initial portion of the extinction sessions after training with 32% as compared to 4% sucrose solution. Furthermore, when given a choice test after extinction training (Experiment 2), animals trained with 32% sucrose, preferred the flavored solution, but animals trained with 4% preferred the unflavored solution. These results are interpreted as indicative of the occurrence of a paradoxical MREE in conditioned consummatory behaviors.     

Early-session increases in responding during extinction

After training under a variable-interval 60-s schedule of reinforcement, four rats were exposed to 30-min extinction tests, which occurred either at the start or at the end of the session (each session being 50-min long). Response rate in extinction decreased when the extinction test occurred at the end of the session, but first increased and then decreased when the extinction test occurred at the start of the session. Consistent with other recent results, this finding suggests that some variable, other than reinforcement, contributes to early-session increases in responding.     

Resistance to extinction, generalization decrement, and conditioned reinforcement

This study investigated generalization decrement during an extinction resistance-to-change test for pigeon key pecking using a two-component multiple schedule with equal variable-interval 3-min schedules and different reinforcer amounts (one component presented 2-s access to reinforcement and the other 8s). After establishing baseline responding, subjects were assigned to one of the two extinction conditions: hopper stimuli (hopper and hopper light were activated but no food was available) or Control (inactive hopper and hopper light). Responding in the 8-s component was more resistant to extinction than responding in the 2-s component, the hopper stimuli group was more resistant to extinction compared to the Control group, and an interaction between amount of reinforcement, extinction condition, and session block was present. This finding supports generalization decrement as a factor that influences resistance to extinction. Hopper-time data (the amount of time subjects spent with their heads in the hopper) were compared to resistance-to-change data in an investigation of the role of conditioned reinforcement on resistance to change.     

Resistance to extinction and behavioral momentum

In the metaphor of behavioral momentum, reinforcement is assumed to strengthen discriminated operant behavior in the sense of increasing its resistance to disruption, and extinction is viewed as disruption by contingency termination and reinforcer omission. In multiple schedules of intermittent reinforcement, resistance to extinction is an increasing function of reinforcer rate, consistent with a model based on the momentum metaphor. The partial-reinforcement extinction effect, which opposes the effects of reinforcer rate, can be explained by the large disruptive effect of terminating continuous reinforcement despite its strengthening effect during training. Inclusion of a term for the context of reinforcement during training allows the model to account for a wide range of multiple-schedule extinction data and makes contact with other formulations. The relation between resistance to extinction and reinforcer rate on single schedules of intermittent reinforcement is exactly opposite to that for multiple schedules over the same range of reinforcer rates; however, the momentum model can give an account of resistance to extinction in single as well as multiple schedules. An alternative analysis based on the number of reinforcers omitted to an extinction criterion supports the conclusion that response strength is an increasing function of reinforcer rate during training.     

Superstitious responding and reinforcement rate under concurrent variable-interval extinction schedules

To examine superstitious responding, four pigeons key pecked under multiple concurrent variable-interval 45 s variable-interval 90 s concurrent variable-interval 90 s variable-interval 180 s schedules in the absence of a changeover delay. The two variable-interval 90 s schedules then were replaced by extinction, and key-peck responding during extinction was examined as a function of the prevailing reinforcement rate. During the first several sessions, extinction-key responding was maintained closer to baseline levels in the presence of the higher reinforcement rate, and this effect dissipated or even reversed with continued exposure to extinction. Although extinction-key responding generally decreased to near-zero levels after several sessions, in a few instances, it continued for 30 and 45 sessions. These results demonstrate how concurrent variable-interval extinction schedules can be used to investigate what often has been labeled superstitious responding.     

Isolated effects of number of acquisition trials on extinction of rat conditioned approach behavior

Four conditioned approach experiments with rats assessed for effects of number of acquisition trials on extinction of conditioned responding, when number of acquisition sessions and total acquisition time were held constant. In Experiment 1, 32 trials per acquisition session led to more extinction responding than did 1 or 2 trials per session but less than did 4 trials per session. In Experiment 2, 2 trials per acquisition session led to more spontaneous recovery than did 32 trials per session. These latter findings are reminiscent of the overtraining extinction effect (OEE). Experiment 3 attempted to reduce the OEE with a preconditioning phase of partial reinforcement. Experiment 4 attempted to reduce the beneficial within-subject effects of increasing the number of acquisition trials on extinction observed by Gottlieb and Rescorla (2010) by extinguishing stimuli in different sessions. Overall, results suggest a procedural asymmetry: between-subject, increasing the number of trials between any pair of trials does not lead to greater persistence of responding during extinction; within-subject, it does. Results are discussed from an associative perspective, with a focus on explanations involving either frustration or comparator mechanisms, and from an information processing perspective, with a focus on Rate Estimation Theory.     

The effect of chronic vs. acute injection of vasopressin on animal learning and memory

The effect of chronic and acute treatment with DDAVP, a vasopressin analog, was studied in 2 month old male rats, using an active avoidance test in a shuttle box. The experiment lasted 6 weeks: an acquisition period of 4 weeks and an extinction period of 2 weeks. Rats were treated one hour before behavioral testing 3 times a week for 6 weeks with either DDAVP 20 micrograms/rat/day for the whole period (chronic group) or with DDAVP for the first week and again once only on the first day of the extinction period (acute group) or with saline. Chronic treatment with DDAVP resulted in better acquisition and in a marked retardation of extinction compared with the acute treatment group. These results were obtained both in normal rats and in rats pretreated at age 5 days of life with intracisternal 6-OH dopamine.     

Exposure to estradiol before but not during acquisition of LiCl-induced conditioned taste avoidance accelerates extinction

Estradiol accelerates extinction of LiCl-induced conditioned taste avoidance when it is present continuously before and during acquisition. We have suggested that the effect of estradiol on extinction is due to its illness-associated, rather than learning-associated, properties. If this were the case, then one would expect estradiol to act before but not during acquisition. This expectation is based on previous work showing attenuation of learned taste avoidance when rats are given distal preexposure (greater than 24 h before conditioning) or proximal preexposure (less than 24 h before conditioning) to the illness-inducing agent LiCl before acquisition of a LiCl-induced conditioned taste avoidance. In three separate experiments, estradiol was administered during three different time periods via subcutaneous implantation of a 10-mm estradiol-filled capsule. In each experiment, the extinction of estradiol-treated females was compared to that of females implanted with empty capsules. In the first experiment, female rats were given distal exposure to estradiol before acquisition. Capsules were implanted 11 days before acquisition and were removed 2 days before acquisition. In the second experiment, female rats were given proximal exposure to estradiol before acquisition. Capsules were implanted 2.5 h before LiCl was paired with a sucrose solution and were removed 16.5 h later. In the third experiment, female rats were given exposure to estradiol during acquisition. Capsules were implanted at the same time as LiCl administration and were removed 18 h later. The only estradiol-treated females to show accelerated extinction were those given distal preexposure to estradiol in Experiment 1. These data do not support a learning-associated hypothesis and only partially support an illness-associated hypothesis. The failure to find accelerated extinction following proximal preexposure may reflect an inappropriate choice of the parameters used in the experiment or a difference in the stimulus properties of LiCl and estradiol that allow each to serve as conditioning and preexposure agents in conditioned taste avoidance paradigms [corrected].     

Estradiol accelerates extinction of lithium chloride-induced conditioned taste aversions through its illness-associated properties

Estradiol accelerates extinction of LiCl-induced conditioned taste aversions when it is present during a period that starts 2-3 days after acquisition and extends throughout extinction (before and during extinction). It has been suggested that estradiol acts before, not during, extinction and that its effect on extinction is associated with its illness-inducing properties. This hypothesis is based on previous work which shows an attenuation of conditioned taste aversion learning when rats are exposed to illness-inducing agents during a period that starts 2 days after acquisition and ends 2 days before extinction trials are initiated. Four experiments were designed to test elements of this hypothesis. The first two experiments demonstrated that if an estradiol-filled Silastic capsule is implanted before extinction of a LiCl-induced aversion, when the conditioned taste is not present, it accelerates extinction, but if it is implanted during extinction, when the conditioned taste is present, it prolongs extinction. The third experiment showed that the same dose of estradiol that accelerates extinction of a LiCl-induced aversion was effective in producing a conditioned taste aversion when it was present for 18 h after consumption of a novel sucrose solution. The fourth experiment indicated that serum levels of estradiol were elevated during the 18 h. These results are consistent with the hypothesis that the acceleration of extinction by estradiol is associated with its illness-inducing properties. It is suggested that estradiol acts on neural areas that mediate illness information and that one of these areas, the area postrema is necessary for estradiol to accelerate extinction of a LiCl-induced aversion.     

The operant reserve: a computer simulation in (accelerated) real time

In Skinner's Reflex Reserve theory, reinforced responses added to a reserve depleted by responding. It could not handle the finding that partial reinforcement generated more responding than continuous reinforcement, but it would have worked if its growth had depended not just on the last response but also on earlier responses preceding a reinforcer, each weighted by delay. In that case, partial reinforcement generates steady states in which reserve decrements produced by responding balance increments produced when reinforcers follow responding. A computer simulation arranged schedules for responses produced with probabilities proportional to reserve size. Each response subtracted a fixed amount from the reserve and added an amount weighted by the reciprocal of the time to the next reinforcer. Simulated cumulative records and quantitative data for extinction, random-ratio, random-interval, and other schedules were consistent with those of real performances, including some effects of history. The model also simulated rapid performance transitions with changed contingencies that did not depend on molar variables or on differential reinforcement of inter-response times. The simulation can be extended to inhomogeneous contingencies by way of continua of reserves arrayed along response and time dimensions, and to concurrent performances and stimulus control by way of different reserves created for different response classes.     

Stimulus change dis-habituates operant responding supported by water reinforcers

The present experiment examined whether habituation contributes to within-session decreases in operant responding for water reinforcers. The experiment asked if this responding can be dishabituated, a fundamental property of habituated behavior. During baseline, rats’ lever pressing was reinforced by water on a variable interval 15-s schedule. During experimental conditions, rats responded on the same schedule and a new stimulus was introduced for 5 min at 15, 30 or 45 min into the 60-min session. The new stimulus was extinction, continuous reinforcement or flashing lights in different conditions. Rate of responding primarily decreased within the session during baseline. Introducing a new stimulus sometimes suppressed (extinction, continuous reinforcement) and sometimes increased (flashing lights) responding while it was in effect. The new stimulus increased responding after it ended and before it was presented in the session. The results are incompatible with the idea that non-habituation satiety factors (e.g., cellular hydration and blood volume) contributed to within-session changes in responding. These satiety factors should increase with increases in consumption, decrease with decreases in consumption and remain constant with constant consumption of water. Nevertheless, all stimulus changes increased operant responding for water. These results support the idea that habituation contributes to within-session decreases in responding for water reinforcers.     

Behavioral contrast of a complex operant

Two experiments were conducted in which pigeons were trained to perform a complex operant consisting of a peck to one key followed by a peck to another key. In the first experment this performance was reinforced on a variable-interval schedule and the birds were then subjected to a multiple schedule in which the variable-interval was alternated with extinction. The first two pigeons trained showed some deterioration of the cohesiveness of the response sequence. After measures were taken to correct this the pigeon gave evidence of behavioral contrast in a condition where sessions with discrimination were alternated rapidly with sessions with no discrimination (variable-interval only). Another two birds were trained on variable-interval followed by discrimination. One of these birds showed evidence of contrast but the results of the other were not clear.Experiment II was a three phase experiment in which animals were first trained to make the complex response. In the second phase they were extinguished by providing noncontingent reinforcement in each of two components. In the third phase reinforcement was removed from one component, providing for a stimulus-reinforcer contingency. In the third phase the complex response reappeared temporarily.The results of these two studies indicate that a complex response may show behavioral contrast and that it may be enhanced by stimulus-reinforcer contingencies if it has already been trained.     

Memory of a choice direction in a T maze as measured by spontaneous alternation in mice: Effects of intertrial interval and reward

Spontaneous alternation in a T maze was studied as a one trial learning paradigm in mice of the BALB/c strain. In the first experiment the combined effects of time interval between the first and second trial (intertrial interval: ITI), food deprivation and feeding given during the first trial, were shown to affect performance. Thus, on the one hand, the percentage of spontaneous alternation decreased as ITI increased; on the other hand, food reward dramatically improved spontaneous alternation for the 24-h ITI, but had no significant effect for 30-sec and 1-h ITI. Since the effect of feeding might be due either to an increase of arousal, thus favoring input of informations associated with the first choice, or to an improvement in memory consolidation, a second experiment was aimed at testing the effect of food given after the first trial. It was shown that, as in the first experiment, post-trial feeding improved spontaneous alternation on the second trial given 24 hours later with a temporal gradient of effect less than 30 min. These results clearly showed that the reinforcement of run to one side (first trial) increased the tendency to go to the other side 24 hours later. It is concluded that reinforcement might have two distinct effects: (i) according to SR theory, reinforcement increases conditioned responses and (ii), as shown here, acts on memory processes by preventing memory traces from fading. The fact that this last effect was only observed for long ITI suggests that short-term or transient memory and long-term memory are two relatively independent processes.     

Extinction from a rationalist perspective

The merging of the computational theory of mind and evolutionary thinking leads to a kind of rationalism, in which enduring truths about the world have become implicit in the computations that enable the brain to cope with the experienced world. The dead reckoning computation, for example, is implemented within the brains of animals as one of the mechanisms that enables them to learn where they are (Gallistel, 1990, 1995). It integrates a velocity signal with respect to a time signal. Thus, the manner in which position and velocity relate to one another in the world is reflected in the manner in which signals representing those variables are processed in the brain. I use principles of information theory and Bayesian inference to derive from other simple principles explanations for: (1) the failure of partial reinforcement to increase reinforcements to acquisition; (2) the partial reinforcement extinction effect; (3) spontaneous recovery; (4) renewal; (5) reinstatement; (6) resurgence (aka facilitated reacquisition). Like the principle underlying dead-reckoning, these principles are grounded in analytic considerations. They are the kind of enduring truths about the world that are likely to have shaped the brain's computations.     

Inhibition of Histone Deacetylases Facilitates Extinction and Attenuates Reinstatement of Nicotine Self-Administration in Rats

Chromatin remodelling is integral to the formation of long-term memories. Recent evidence suggests that histone modification may play a role in the persistence of memories associated with drug use. The present series of experiments aimed to examine the effect of histone deacetylase (HDAC) inhibition on the extinction and reinstatement of nicotine self-administration. Rats were trained to intravenously self-administer nicotine for 12 days on a fixed-ratio 1 schedule. In Experiment 1, responding was then extinguished through removal of nicotine and response-contingent cues. After each extinction session, the HDAC inhibitor, sodium butyrate (NaB), was administered immediately, or six hours after each session. In Experiment 2, response-contingent cues remained available across extinction to increase rates of responding during this phase, and NaB was administered immediately after the session. Finally, in Experiment 3, the effect of NaB treatment on extinction of responding for sucrose pellets was assessed. Across all experiments reinstatement to the cue and/or the reward itself was then tested. In the first experiment, treatment with NaB significantly attenuated nicotine and nicotine + cue reinstatement when administered immediately, but not six hours after each extinction session. When administered after cue-extinction (Expt. 2), NaB treatment specifically facilitated the rate of extinction across sessions, indicating that HDAC inhibition enhanced consolidation of the extinction memory. In contrast, there was no effect of NaB on the extinction and reinstatement of sucrose-seeking (Expt. 3), indicating that the observed effects are specific to a drug context. These results provide the first demonstration that HDAC inhibition facilitates the extinction of responding for an intravenously self-administered drug of abuse and further highlight the potential of HDAC inhibitors in the treatment of drug addiction.     

Simulation of rat behavior by a reinforcement learning algorithm in consideration of appearance probabilities of reinforcement signals

Brown and Wanger [Brown, R.T., Wanger, A.R., 1964. Resistance to punishment and extinction following training with shock or nonreinforcement. J. Exp. Psychol. 68, 503-507] investigated rat behaviors with the following features: (1) rats were exposed to reward and punishment at the same time, (2) environment changed and rats relearned, and (3) rats were stochastically exposed to reward and punishment. The results are that exposure to nonreinforcement produces resistance to the decremental effects of behavior after stochastic reward schedule and that exposure to both punishment and reinforcement produces resistance to the decremental effects of behavior after stochastic punishment schedule. This paper aims to simulate the rat behaviors by a reinforcement learning algorithm in consideration of appearance probabilities of reinforcement signals. The former algorithms of reinforcement learning were unable to simulate the behavior of the feature (3). We improve the former reinforcement learning algorithms by controlling learning parameters in consideration of the acquisition probabilities of reinforcement signals. The proposed algorithm qualitatively simulates the result of the animal experiment of Brown and Wanger.