The operant reserve: a computer simulation in (accelerated) real time

In Skinner's Reflex Reserve theory, reinforced responses added to a reserve depleted by responding. It could not handle the finding that partial reinforcement generated more responding than continuous reinforcement, but it would have worked if its growth had depended not just on the last response but also on earlier responses preceding a reinforcer, each weighted by delay. In that case, partial reinforcement generates steady states in which reserve decrements produced by responding balance increments produced when reinforcers follow responding. A computer simulation arranged schedules for responses produced with probabilities proportional to reserve size. Each response subtracted a fixed amount from the reserve and added an amount weighted by the reciprocal of the time to the next reinforcer. Simulated cumulative records and quantitative data for extinction, random-ratio, random-interval, and other schedules were consistent with those of real performances, including some effects of history. The model also simulated rapid performance transitions with changed contingencies that did not depend on molar variables or on differential reinforcement of inter-response times. The simulation can be extended to inhomogeneous contingencies by way of continua of reserves arrayed along response and time dimensions, and to concurrent performances and stimulus control by way of different reserves created for different response classes.     

Performance correlates of social behavior and organization: Social rank and omission of reinforcement in rhesus monkeys (M. mulatta)

The performance of 22 adult male rhesus monkeys on a Fixed Interval 1-min reinforcement schedule was examined under conditions where the reinforcement probabilities were either 1.00 or .80. The results were then related to the social rank of animals at the time that they were taken from their social groups. Both high and low ranked animals reached criterion performance in the same number of trials. In general, high ranking animals responded at lower rates than low ranking animals when the reinforcement probability was 1.00. When the reinforcement probability was shifted to .80, all animals showed an increase in responding after nonreinforced intervals as compared with responses after reinforced intervals. The higher ranked animals tended to have a higher ratio of nonreinforced to reinforced responses than lower ranked animals. Supported by USAMRDC Contract No. DADA 17-73-C-3007.     

Differential resurgence and response elimination

Resurgence refers to the transient recovery of previously reinforced, but presently not reinforced, responding when more recently reinforced responding is extinguished. The primary purpose of our research was to determine how differential resurgence results from the procedures used to eliminate that responding. There were three conditions in each of five experiments. In Condition 1, key pecking by pigeons was maintained under a two-component multiple variable-interval (VI) 30-s VI 30-s schedule. In Condition 2, this pecking was eliminated in different ways across components. In Condition 3, extinction was in effect for all responses, and resurgence of key pecking was compared across components. These three conditions were repeated for most pigeons, and the procedures used to eliminate responding in Condition 2 varied across experiments. In Experiment 1, there was greater resurgence, and an earlier onset of it, after a differential-reinforcement-of-other-behavior (DRO) schedule than after a VI schedule was correlated with pecking an alternative key. Experiments 2 and 3 showed that the differential resurgence in Experiment 1 probably was not due to conditional stimulus control or the periodicity of food delivery, respectively. In Experiment 4, there was no systematic difference in resurgence after either a DRO schedule or a VI schedule correlated with treadle pressing. In Experiment 5, there was greater resurgence, and/or an earlier onset of it, after a VI schedule correlated with treadle pressing than after a VI schedule correlated with pecking an alternative key. Taken together, the results showed that the reinforcement of an alternative key-peck response was the most effective means of reducing subsequent key-peck resurgence. The relation of these results to an understanding of resurgence is discussed.     

Time estimation by pigeons on a fixed interval: the effect of pre-feeding

Pigeons well trained on a fixed interval 10-s schedule of reinforcement were tested on the peak procedure. In a successive conditions design, they were either pre-fed or not in the experimental setting. Pre-feeding decreased the rate of responding. It also led to a maximum rate of responding that occurred 2-3 s later than in the control condition, where the maximum occurred at the usual time of reinforcement. The shift in peak time in response to pre-feeding shows that peak time may not be a pure measure of timing. The results are also interpreted in terms of timing theories.     

Stimulus change dis-habituates operant responding supported by water reinforcers

The present experiment examined whether habituation contributes to within-session decreases in operant responding for water reinforcers. The experiment asked if this responding can be dishabituated, a fundamental property of habituated behavior. During baseline, rats’ lever pressing was reinforced by water on a variable interval 15-s schedule. During experimental conditions, rats responded on the same schedule and a new stimulus was introduced for 5 min at 15, 30 or 45 min into the 60-min session. The new stimulus was extinction, continuous reinforcement or flashing lights in different conditions. Rate of responding primarily decreased within the session during baseline. Introducing a new stimulus sometimes suppressed (extinction, continuous reinforcement) and sometimes increased (flashing lights) responding while it was in effect. The new stimulus increased responding after it ended and before it was presented in the session. The results are incompatible with the idea that non-habituation satiety factors (e.g., cellular hydration and blood volume) contributed to within-session changes in responding. These satiety factors should increase with increases in consumption, decrease with decreases in consumption and remain constant with constant consumption of water. Nevertheless, all stimulus changes increased operant responding for water. These results support the idea that habituation contributes to within-session decreases in responding for water reinforcers.     

Influence of the reinforcement magnitude on omission effects

Reinforcement Omission Effects (ROEs), indicated by higher rate of responses after nonreinforced trials in a partial reinforcement schedule, have been interpreted as behavioral transient facilitation after nonreinforcement induced by primary frustration, and/or behavioral transient inhibition after reinforcement induced by demotivation or temporal control. The size of the ROEs should depend directly on the reinforcement magnitude. The present experiment aimed to clarify the relationship between reinforcement magnitude and the omission effects manipulating the magnitude linked to discriminative stimuli in a partial reinforcement FI schedule. The results showed that response rates were higher after omission than after reinforcement delivery. Besides, response rates were highest immediately after the reinforcement omission of a larger magnitude than of a smaller magnitude. These data are interpreted in terms of ROEs multiple process behavioral facilitation after nonreinforcement, and behavioral transient inhibition after reinforcement.     

Percentage reward in an operant analogue to foraging

Pigeons (Columba livia) responding on an operant, conditioning analogue to foraging could either accept or reject a variable-interval 20-s schedule which always led to food or a variable-interval 5-s schedule which led to food on only a percentage of trials. The probability of accepting the certain alternative increased as the percentage of food trials for the uncertain alternative decreased. The probability of accepting the uncertain alternative increased with the percentage of food trials for this alternative. Subjects receiving all of their food in the experiment (‘closed economy’) and those requiring supplementation (‘open economy’) preferred whichever alternative provided the higher overall mean rate of reinforcement.     

The proportion of fixed interval trials to probe trials affects acquisition of the peak procedure fixed interval timing task

A common procedure for studying the ability of animals to time is the peak procedure. With the peak procedure, animals are first trained on a fixed interval schedule (i.e., 30s). After the animals have been well trained on the fixed interval schedule, probe trials are introduced. On probe trials, the stimulus is presented longer (i.e., 90s) and the animal does not receive reinforcement for responding. When animals are first presented with probe trials responding remains flat following the point that reinforcement normally occurs on fixed interval trials. The descending slope that eventually emerges is acquired with experience with probe trials. The present experiments manipulated the percentage of probe trials compared to FI trials across groups of rats. It was hypothesized that the descending limb of peak responding would be acquired more quickly when there were many probe trials per session as this might facilitate extinction of responding beyond the interval that reinforcement normally occurs. It was found, however, that acquisition of peak responding occurred best when there were few probe trials per session.     

Resistance to extinction, generalization decrement, and conditioned reinforcement

This study investigated generalization decrement during an extinction resistance-to-change test for pigeon key pecking using a two-component multiple schedule with equal variable-interval 3-min schedules and different reinforcer amounts (one component presented 2-s access to reinforcement and the other 8s). After establishing baseline responding, subjects were assigned to one of the two extinction conditions: hopper stimuli (hopper and hopper light were activated but no food was available) or Control (inactive hopper and hopper light). Responding in the 8-s component was more resistant to extinction than responding in the 2-s component, the hopper stimuli group was more resistant to extinction compared to the Control group, and an interaction between amount of reinforcement, extinction condition, and session block was present. This finding supports generalization decrement as a factor that influences resistance to extinction. Hopper-time data (the amount of time subjects spent with their heads in the hopper) were compared to resistance-to-change data in an investigation of the role of conditioned reinforcement on resistance to change.     

Contextual determinants of temporal control: Behavioral contrast in a free-operant psychophysical procedure

The question of how temporal control of responding might be influenced by contingency changes in other contexts was investigated. Each of three pigeons first was exposed to a two-component multiple schedule in which a two-key free-operant psychophysical procedure operated in one component and a variable-interval schedule operated in the other component. The variable-interval schedule then was changed to extinction while the free-operant psychophysical procedure remained unchanged. Finally, the variable-interval schedule was reintroduced. Response rates on the left key and the estimated temporal threshold under the free-operant psychophysical procedure increased for each pigeon when the alternate component schedule was changed to extinction and then decreased again when the variable-interval schedule was reintroduced. The results suggest one way that temporal control is affected by its context, and may be interpreted through the direct effects of overall reinforcement rate on temporal control mechanisms or the disruptive effects of alternative sources of reinforcement on temporally controlled behavior.     

Effects of methamphetamine on response rate: a microstructural analysis

Key pecking in pigeons was maintained under a multiple random-interval (RI) 1-min, RI 4-min schedule of food presentation. Several doses (0.3-5.6 mg/kg) of methamphetamine were administered, and effects on overall response rates and on the microstructure of responding were characterized. In three of the four pigeons, methamphetamine dose-dependently decreased overall response rate in both components; in the fourth pigeon, intermediate doses increased response rates. Log-survivor analyses did not produce the clear "broken-stick" pattern previously reported with rats [Shull, R.L., Gaynor, S.T., Grimes, J.A., 2001. Response rate viewed as engagement bouts: effects of relative reinforcement and schedule type. J. Exp. Anal. Behav. 75, 247-274]. A fine-grained analysis of inter-response times (IRTs) revealed clear bands of responding around certain IRT durations. Methamphetamine tended to decrease the frequency of IRTs in the shorter bands and increase the frequency of IRTs across all bins greater than 2s. These results suggest that (a) survivor analyses may not extend to pigeon key pecking, (b) microstructural analyses can reveal order not evident with overall response rate, and (c) a detailed analysis of responding might prove more useful than summary measures in characterizing drug effects on behavior.     

Rats more readily acquire a time-of-day go no-go discrimination than a time-of-day choice discrimination

Male Sprague-Dawley rats were used to compare six time-place training procedures that differed with respect to housing or training conditions. All procedures involved training food-deprived rats to enter one choice arm of a T-maze during a morning test session and to enter the other choice arm during an afternoon session to obtain Cocoa Puffs(R). The task proved to be difficult. Only 39 of 49 rats attained a criterion of nine correct choices on ten consecutive trials within a total of 120 trials. Making one choice arm distinct, limiting consecutive same correct choices to two, giving one session during the light and one during the dark portion of the light cycle, extinguishing perseveration of the same choice responses, and housing the rats with a natural light cycle all failed to significantly decrease the errors or trials to a 90% correct choice criterion. In contrast, all responding rats (n=7) showed significantly better than chance performance within 48 trials when the task was a go no-go discrimination based on time of day. Learning to make a response during a session when a choice to either choice arm is reinforced and to withhold responding during a session when a choice to neither choice arm is reinforced was relatively easy for the rats to acquire. Continued high level performance after the light cycle was eliminated, a random feeding schedule was initiated, and other time-related cues were masked suggests that the rats used internal cues or an internal clock to make the correct go or no-go response. It was concluded that rats are prepared to use time of day as an occasion-setting stimulus, but have difficulty using time of day as a signal for a specific response.     

Psychological distance to reward: equating the number of stimulus and response segments

Psychological distance to reward, or the segmentation effect, refers to the preference for a terminal link of a concurrent-chains schedule consisting of a simple reinforcement schedule (e.g. fixed interval [FI] 30s) relative to its chained-schedule counterpart (e.g. chained FI 15s FI 15s). This experiment was conducted to examine whether the segmentation effect is due to the number of terminal-link stimulus and response segments per se. Three pigeons pecked under a concurrent-chains schedule in which identical variable-interval (VI) schedules operated in the initial links. In each session, half the terminal-link entries followed one initial-link key and the other half followed the other initial-link key. The initial-link keys correlated with the different terminal links were manipulated across conditions. In the first three conditions, each terminal link contained a chained fixed-time (FT) FT schedule, and in the final three conditions, each terminal link contained a chained FI FI schedule. In each condition, in one terminal link (alternating), the order of two key colors correlated with the different schedule segments alternated across terminal-link entries, whereas in the other terminal link (constant), the order of two other key colors was identical for each entry. With the chained FT FT schedule terminal links, there was indifference between the alternating and constant terminal links within and across pigeons, as indexed by initial-link choice proportions. In addition, terminal-link response rates were relatively low. With the chained FI FI schedule terminal links, for each pigeon, there was relatively more preference for the alternating terminal link and terminal-link response rates increased relative to conditions with the chained FT FT schedule terminal links. These data suggest that the segmentation effect is not due simply to the number of terminal-link stimulus or response segments per se, but rather to a required period of responding during a stimulus segment that never is paired with reinforcement.     

Progesterone attenuates impulsive action in a Go/No-Go task for sucrose pellets in female and male rats

Impulsivity, or a tendency to act without anticipation of future consequences, is associated with drug abuse. Impulsivity is typically separated into two main measures, impulsive action and impulsive choice. Given the association of impulsivity and drug abuse, treatments that reduce impulsivity have been proposed as an effective method for countering drug addiction. Progesterone has emerged as a promising treatment, as it is associated with decreased addiction-related behaviors and impulsive action. The goal of the present study was to determine the effects of progesterone (PRO) on impulsive action for food: a Go/No-Go task. Female and male rats responded for sucrose pellets during a Go component when lever pressing was reinforced on a variable-interval 30-s schedule. During the alternate No-Go component, withholding a lever press was reinforced on a differential reinforcement of other (DRO) behavior 30-s schedule, where a lever press reset the DRO timer. Impulsive action was operationally defined as the inability to withhold a response during the No-Go component (i.e. the number of DRO resets). Once Go/No-Go behavior was stable, responding between rats treated with PRO (0.5 mg/kg) or vehicle was examined. Progesterone significantly decreased the total number of DRO resets in both males and females, but it did not affect VI responding for sucrose pellets. This suggests that PRO decreases motor impulsivity for sucrose pellets without affecting motivation for food. Thus, PRO may reduce motor impulsivity, a behavior underlying drug addiction.     

Temporal differentiation in two strains of small rodents: a wood mouse (Apodemus sylvaticus) and an albino mouse (Mus musculus OF1)

Wood mice (Apodemus sylvaticus) and OF1 albino mice (Mus musculus) were compared over durations ranging from 0.5 to 7 s, using the differential reinforcement of response duration schedule (DRRD) and a 'platform' response, i.e. staying on a small platform for a specified criterion duration to be reinforced. Species-related differences were found for mean response durations, efficiency and the number of trials needed to reach a preset performance criterion. Coefficients of variation of response durations did not differ. Overall, OF1 mice needed more trials than wood mice to reach a temporal criterion. However, over 3-7 s, data from both strains almost fitted the behavioral assumptions of Scalar Timing theory. Performance of mongolian gerbils (Meriones unguiculatus) trained in a similar setting was shown for visual comparison.     

Effect of contingent auditory stimuli on concurrent schedule performance: an alternative punisher to electric shock

This study explored whether load auditory stimuli could be used as functional punishing stimuli in place of electric shock. Three experiments examined the effect of a loud auditory stimulus on rats’ responding maintained by a concurrent reinforcement schedule. In Experiment 1, overall response rate decreased when a concurrent 1.5 s tone presentation schedule was superimposed on the concurrent variable interval (VI) 180-s, VI 180-s reinforcement schedule. On the contrary, response rate increased when a click presentation schedule was added. In Experiment 2, the extent of the response suppression with a 1.5 s tone presentation varied as a function of the frequency of the reinforcement schedule maintaining responses; the leaner the schedule employed, the greater the response suppression. In Experiment 3, response suppression was observed to be inversely related to the duration of the tone; response facilitation was observed when a 3.0-s tone was used. In Experiments 1 and 2, a preference shift towards the alternative with richer reinforcement was observed when the tone schedule was added. In contrast, the preference shifted towards the leaner alternative when the click or longer duration stimulus was used. These results imply that both the type and duration of a loud auditory stimulus, as well as the reinforcement schedule maintaining responses, have a critical role in determining the effect of the stimuli on responding. They also suggest that a loud auditory stimulus can be used as a positive punisher in a choice situation for rats, when the duration of the tone is brief, and the reinforcement schedule maintaining responses is lean.     

An Assessment of Fixed Interval Timing in Free-Flying Honey Bees (Apis mellifera ligustica): An Analysis of Individual Performance

Interval timing is a key element of foraging theory, models of predator avoidance, and competitive interactions. Although interval timing is well documented in vertebrate species, it is virtually unstudied in invertebrates. In the present experiment, we used free-flying honey bees (Apis mellifera ligustica) as a model for timing behaviors. Subjects were trained to enter a hole in an automated artificial flower to receive a nectar reinforcer (i.e. reward). Responses were continuously reinforced prior to exposure to either a fixed interval (FI) 15-sec, FI 30-sec, FI 60-sec, or FI 120-sec reinforcement schedule. We measured response rate and post-reinforcement pause within each fixed interval trial between reinforcers. Honey bees responded at higher frequencies earlier in the fixed interval suggesting subject responding did not come under traditional forms of temporal control. Response rates were lower during FI conditions compared to performance on continuous reinforcement schedules, and responding was more resistant to extinction when previously reinforced on FI schedules. However, no “scalloped” or “break-and-run” patterns of group or individual responses reinforced on FI schedules were observed; no traditional evidence of temporal control was found. Finally, longer FI schedules eventually caused all subjects to cease returning to the operant chamber indicating subjects did not tolerate the longer FI schedules.     

Transitions between cooperative and non-cooperative responding in the 'Pigeon's Dilemma'

Two pigeons played Iterated Prisoner's Dilemma (IPD) against a simulated opponent pre-programmed to play ‘Tit-For-Tat’ (TFT) and ‘Random’ (RND) strategies. Each pigeon received differential amounts of access to food following choices of either ‘cooperate’ or ‘defect’ on a trial. After 1000 trials against TFT and 500 trials against RND, results indicated that choice allocation was optimal when the birds played against RND but was sub-optimal when the birds played against TFT. In order to determine why the pigeons responded suboptimally against TFT, a trial-by-trial analysis of the data was conducted. The analysis revealed that once a pigeon had received the ‘Sucker's’ payoff (S), it was more likely to defect and receive the ‘Temptation’ payoff (T) than to cooperate and receive the ‘Reward’ (R) payoff. Local reinforcement contingencies appear to determine suboptimal responding against TFT in the iterated Pigeon's Dilemma.     

Resistance to variable-time schedules produced by spaced-response reinforcement schedules

Two experiments are reported in which rats were exposed to either a tandem fixed-ratio 1 differential-reinforcement-of-low-rate 10 sec schedule (tand FR1 DRL 10) (Experiment 1) or to a DRL 10 sec schedule (Experiment 2) prior to being exposed to a variable-time (VT) schedule. Response decrement was not universally found during the VT phase, one rat emmitting an increase in response rate relative to baseline (Experiment 1), while another showed neither an increase nor decrease during VT relative to baseline (Experiment2). The VT schedule induced less efficient responding in both experiments. Interresponse time (IRT) distributions (Experiment 2) indicated that the VT changed the pattern of responding. These results, when combined with others from our laboratory, indicate that although unsignalled variable delay of reinforcement may be a sufficient condition for producing resistance to response-independent reinforcement, it is not a necessary condition. It was concluded that a response competition model of the kind proposed by Henton and Iversen (1978) might have explanatory merit in this kind of experimental situation.     

Operant behavior in dwarf hamsters (Phodopus campbelli): Effects of rate of reinforcement and reinforcer flavor variety

We investigated operant behavior in a novel species, the dwarf hamster (Phodopus campbelli). In two experiments, hamsters were trained to lever-press for food reinforcement. In Experiment 1, rate of reinforcement was manipulated across conditions using four variable-interval schedules of reinforcement (delivering one to eight reinforcers per min). As predicted, within-session decreases in responding were steepest on the richest schedule. In Experiment 2, lever-pressing was reinforced by either a constant or a variety of flavored food pellets. Within-session decreases in responding were steeper when the reinforcer flavor remained constant than when it was varied within the session. In both experiments, subjects hoarded most reinforcers in their cheek pouches rather than consuming them in the operant chambers. These results are incompatible with post-ingestive satiety variables as explanations for within-session decreases in operant responding and suggest that habituation to repeatedly presented reinforcers best accounts for subjects’ response patterns. Additionally, a mathematical model that describes behavior undergoing habituation also described the present results, thus strengthening the conclusion that habituation mediates the reinforcing efficacy of food.