The operant reserve: a computer simulation in (accelerated) real time

In Skinner's Reflex Reserve theory, reinforced responses added to a reserve depleted by responding. It could not handle the finding that partial reinforcement generated more responding than continuous reinforcement, but it would have worked if its growth had depended not just on the last response but also on earlier responses preceding a reinforcer, each weighted by delay. In that case, partial reinforcement generates steady states in which reserve decrements produced by responding balance increments produced when reinforcers follow responding. A computer simulation arranged schedules for responses produced with probabilities proportional to reserve size. Each response subtracted a fixed amount from the reserve and added an amount weighted by the reciprocal of the time to the next reinforcer. Simulated cumulative records and quantitative data for extinction, random-ratio, random-interval, and other schedules were consistent with those of real performances, including some effects of history. The model also simulated rapid performance transitions with changed contingencies that did not depend on molar variables or on differential reinforcement of inter-response times. The simulation can be extended to inhomogeneous contingencies by way of continua of reserves arrayed along response and time dimensions, and to concurrent performances and stimulus control by way of different reserves created for different response classes.     

Choice in a variable environment: Effects of d-amphetamine on sensitivity to reinforcement

Four pigeons responded under a 7-component mixed schedule in which each component arranged a different left:right reinforcer ratio (27:1, 9:1, 3:1, 1:1, 1:3, 1:9, 1:27). Components were unsignaled, and the order within each session was randomly determined. After extensive exposure to these contingencies, effects of a range of doses of d-amphetamine (0.3-5.6 mg/kg) on estimates of sensitivity to reinforcement at several levels of analysis were assessed. Under non-drug conditions, the structure of choice was similar to that previously reported under this procedure. That is, responding adjusted within components to the reinforcer ratio in effect (i.e., sensitivity estimates were higher in the 2nd than in the 1st half of components), and individual reinforcers produced “preference pulses” (i.e., each food presentation produced an immediate, local, shift in preference toward the response that just produced food). Although there was a general tendency for d-amphetamine to reduce overall sensitivity to reinforcement, the size of this effect and its reliability varied across pigeons. Further analysis, however, revealed that intermediate d-amphetamine doses consistently reduced sensitivity immediately following reinforcer presentations; that is, these doses consistently attenuated preference pulses.     

Constraints on the process of interresponse-time reinforcement as the explanation of variable-interval performance

Computer modelling was used to investigate the adequacy of the principle of reinforcement of single interresponse-times (IRTs) as an explanation of variable-interval performance. Variations in three components of models employing IRT reinforcement were simulated; these were (i) rules for selecting previously-reinforced IRTs from the IRT memory as “candidates” for output (acceptance rules), (ii) processes involving the transformation of previously-reinforced IRTs to generate behavioural variability, and (iii) the role of the minimum IRT value permitted by the models. In general, simulation output resembled experimental data only when (i) acceptance rules discriminated strongly against long, and in favour of short, IRTs; and (ii) when IRTs were transformed by distributions in which standard deviation was a large fraction (50–125%) of the mean, thus generating substantial IRT variability. Simulation results were analyzed by means of Herrnstein's hyperbolic equation, and in general, it was found that the output of IRT reinforcement models was insensitive to the rate of reinforcement received, as well as conforming less well to Herrnstein's equation than did experimental data. Overall, therefore, it was concluded that models of the reinforcement of single IRTs could provide an adequate simulation of VI performance only when the processes within them operated within narrow constraints. Even then, models involving the reinforcement of single IRTs could adequately simulate only those VI data in which response rate was insensitive to reinforcement rate.     

The effect of second-half reinforcer type on responding for sucrose in the first half of the session

The present study investigated whether the sucrose-reinforced lever pressing of rats in the first half of a 50-min session would be sensitive to upcoming food-pellet reinforcement in the second half. In Experiment 1, the type of reinforcer in the first half of the session was always liquid sucrose and type of reinforcer in the second half (liquid sucrose or food pellets) varied across conditions. Sucrose concentration varied across groups (1, 5, or 25%). Results showed that rates and patterns of responding for 1%, and sometimes for 5%, sucrose reinforcers in the first half of the session were higher and steeper, respectively, when food-pellet, rather than sucrose, reinforcement occurred in the second half. Responding for 25% sucrose was not similarly affected. Experiment 2 replicated the results of Experiment 1 using a within-subjects design. Although the present results represent induction (i.e. the opposite of contrast), they are consistent with some results on consummatory contrast. They also further demonstrate that responding on interval schedules of reinforcement can be altered prospectively. By doing so, however, they pose potential problems for current theories for why operant response rates change within the session.     

Human performance on a two-alternative rapid-acquisition choice task

Davison and Baum [Davison, M., Baum, W. M., 2000. Choice in a variable environment: every reinforcer counts. Journal of the Experimental Analysis of Behavior 74, 1-24.] developed a concurrent-schedule procedure where, within each session, different reinforcer ratios were arranged across components separated by brief black-outs. Behaviour adapted quickly to the reinforcer ratios and reinforcers also had local effects on responding. This procedure has been used with pigeons and rats. In the present experiment, we adapted the Davison and Baum procedure to study the effects of reinforcement on human choice behaviour. Eighteen participants were presented with four different reinforcer ratios within a single 50-minute session. Mean sensitivity to the reinforcer ratios increased within components, and preference was greater for the just-reinforced response alternative immediately following reinforcer delivery, similar to the results from non-human experiments. Although there were limitations to the current procedure, the local time scale analyses are a novel way of examining human operant behaviour.     

Investigations of timing during the schedule and reinforcement intervals with wheel-running reinforcement

Across two experiments, a peak procedure was used to assess the timing of the onset and offset of an opportunity to run as a reinforcer. The first experiment investigated the effect of reinforcer duration on temporal discrimination of the onset of the reinforcement interval. Three male Wistar rats were exposed to fixed-interval (FI) 30-s schedules of wheel-running reinforcement and the duration of the opportunity to run was varied across values of 15, 30, and 60s. Each session consisted of 50 reinforcers and 10 probe trials. Results showed that as reinforcer duration increased, the percentage of postreinforcement pauses longer than the 30-s schedule interval increased. On probe trials, peak response rates occurred near the time of reinforcer delivery and peak times varied with reinforcer duration. In a second experiment, seven female Long-Evans rats were exposed to FI 30-s schedules leading to 30-s opportunities to run. Timing of the onset and offset of the reinforcement period was assessed by probe trials during the schedule interval and during the reinforcement interval in separate conditions. The results provided evidence of timing of the onset, but not the offset of the wheel-running reinforcement period. Further research is required to assess if timing occurs during a wheel-running reinforcement period.     

The matching law and effects of reinforcer rate and magnitude on choice in transition

Four pigeons responded in a concurrent-schedule procedure in which reinforcer rates and magnitudes changed unpredictably across sessions according to independent random series. Programmed relative reinforcement rates and magnitudes were always either 2:1 or 1:2. Pigeons' response allocation tended to stabilize within sessions and multiple regression analyses showed that it was determined by rates and magnitudes from the current session. Sensitivity coefficients were positive and statistically significant for current-session reinforcement and magnitude ratios. Although there were individual differences in sensitivity to rate and magnitude, their interaction was not significant across subjects. Rate and magnitude both controlled responding in single sessions and individual interreinforcer intervals. Analyses of responding within sessions showed that preference was more extreme when the richer rate and larger magnitude were associated with the same alternative than when they were associated with different alternatives. Overall, results support the concatenated generalized matching law's assumptions of additivity and independence as applied to choice in transition.     

Effects of prefeeding, extinction, and distraction during sample and comparison presentation on sensitivity to reinforcer frequency in matching to sample

The present experiment examined the effects of several test manipulations on discrimination, accuracy and sensitivity to reinforcer frequency in a conditional discrimination. Four pigeons responded on a multiple schedule of matching to sample procedures in which the reinforcer-frequency ratio for correct comparison choice responding was varied across components within session from 1:9 to 9:1. Following stability, the effects of prefeeding, extinction, and distraction during sample and comparison presentation were assessed. Discrimination accuracy decreased under prefeeding, extinction, and distraction during sample presentation. Sensitivity to reinforcer frequency decreased under prefeeding and extinction. Decreases in sensitivity were positively related to decreases in discrimination accuracy. The decreases in discrimination accuracy and sensitivity under prefeeding and extinction are interpreted as being due to decreases in attending to the sample and comparison stimuli, respectively, possibly mediated by motivational effects of these manipulations. This interpretation is consistent with current conceptualizations of the contingencies that govern conditional-discrimination performance.     

Quantitative analyses of methamphetamine's effects on self-control choices: implications for elucidating behavioral mechanisms of drug action

The purpose of the present research was to utilize quantitative methods to identify behavioral mechanisms involved in the effects of stimulant drugs on choice in a self-control procedure. A logarithmic equation based upon a combination of the matching law and hyperbolic discounting was used to separate drug-induced changes in sensitivity to reinforcement delay from drug-induced changes in sensitivity to reinforcement amount. Pigeons responded under a concurrent-chains schedule. In the initial link, two keys were illuminated simultaneously and access to the terminal link was controlled by a single random-interval (RI) schedule; pecks on one or the other key lead to its terminal link with a 0.5 probability. In the terminal links, one alternative provided 1-s access to food (the smaller reinforcer) and the other alternative provided 4-s access to food (the larger reinforcer). The signaled delay to the smaller reinforcer always was 2s, whereas the signaled delay to the larger reinforcer increased from 2 to 40s within each session, across 10-min blocks. In general, intermediate doses of methamphetamine increased preference for the larger more delayed reinforcer. Quantitative analyses indicated that, in most cases, methamphetamine decreased sensitivity to reinforcement delay. In a few instances, concomitant decreases in sensitivity to reinforcement amount also occurred. These results suggest that a reduced sensitivity to reinforcement delay may be important behavioral mechanism of the effects of stimulants on self-control choices, and that this effect sometimes can be accompanied by a decreased sensitivity to reinforcement amount.     

Resistance to change as a function of concurrent reinforcer magnitude

Six pigeons responded on two keys in each of three signalled multiple-schedule components, and resistance to disruption of responding on one (target) key by extinction and by response-independent food presented during inter-component blackouts was studied. Alternative reinforcement of different magnitudes was contingent on pecking a non-target key in two components, and in the third only the target response was reinforced. Resistance to change varied with the overall quantity of reinforcement in the component, regardless of whether reinforcers were contingent on the target or non-target response, but did not differ across the two key locations. These results using different magnitudes of reinforcement confirm previous findings using rate of reinforcement as the variable, and suggest that resistance to change is dependent on stimulus-reinforcer rather than response-reinforcer contingencies.     

Operant behavior in dwarf hamsters (Phodopus campbelli): Effects of rate of reinforcement and reinforcer flavor variety

We investigated operant behavior in a novel species, the dwarf hamster (Phodopus campbelli). In two experiments, hamsters were trained to lever-press for food reinforcement. In Experiment 1, rate of reinforcement was manipulated across conditions using four variable-interval schedules of reinforcement (delivering one to eight reinforcers per min). As predicted, within-session decreases in responding were steepest on the richest schedule. In Experiment 2, lever-pressing was reinforced by either a constant or a variety of flavored food pellets. Within-session decreases in responding were steeper when the reinforcer flavor remained constant than when it was varied within the session. In both experiments, subjects hoarded most reinforcers in their cheek pouches rather than consuming them in the operant chambers. These results are incompatible with post-ingestive satiety variables as explanations for within-session decreases in operant responding and suggest that habituation to repeatedly presented reinforcers best accounts for subjects’ response patterns. Additionally, a mathematical model that describes behavior undergoing habituation also described the present results, thus strengthening the conclusion that habituation mediates the reinforcing efficacy of food.     

Training a new response using conditioned reinforcement

Some of the most frequently used methods in the study of conditioned reinforcement seem to be insufficient to demonstrate the effect. The clearest way to assess this phenomenon is the training of a new response. In the present study, rats were exposed to a situation in which a primary reinforcer and an arbitrary stimulus were paired and subsequently the effect of this arbitrary event was assessed by presenting it following a new response. Subjects under these conditions emitted more responses compared to their own responding before the pairing and to their responding on a similar operandum that was available concurrently that had no programmed consequences. Response rates also were higher compared to responding by subjects in similar conditions in which there was no contingency (a) between the arbitrary stimulus and the reinforcer, (b) between the response and the arbitrary stimulus or (c) both. Results are discussed in terms of necessary and sufficient conditions to study conditioned reinforcement.     

Whisking as a “voluntary” response: operant control of whisking parameters and effects of whisker denervation

The rat’s ability to vary its whisking “strategies” to meet the functional demands of a discriminative task suggests that whisking may be characterized as a “voluntary” behavior—an operant—and like other operants, should be modifiable by appropriate manipulations of response–reinforcer contingencies. To test this hypothesis we have used high-resolution, optoelectronic “real-time” recording procedures to monitor the movements of individual whiskers and reinforce specific movement parameters (amplitude, frequency). In one operant paradigm (N = 9) whisks with protractions above a specified amplitude were reinforced (Variable Interval 30?s) in the presence of a tone, but extinguished (EXT) in its absence. In a second paradigm (N = 3), rats were reinforced on two different VI schedules (VI-20s/VI-120s) signaled, respectively, by the presence or absence of the tone. Selective reinforcement of whisking movements maintained the behavior over many weeks of testing and brought it under stimulus and schedule control. Subjects in the first paradigm learned to increase responding in the presence of the tone and inhibit responding in its absence. In the second paradigm, subjects whisked at significantly different rates in the two stimulus conditions. Bilateral deafferentation of the whisker pad did not impair conditioned whisking or disrupt discrimination behavior. Our results confirm the hypothesis that rodent whisking has many of the properties of an operant response. The ability to bring whisking movement parameters under operant control should facilitate electrophysiological and lesion/behavioral studies of this widely used “model” sensorimotor system.     

"The stone which the builders rejected...": Delay of reinforcement and response rate on fixed-interval and related schedules

The article deals with response rates (mainly running and peak or terminal rates) on simple and on some mixed-FI schedules and explores the idea that these rates are determined by the average delay of reinforcement for responses occurring during the response periods that the schedules generate. The effects of reinforcement delay are assumed to be mediated by a hyperbolic delay of reinforcement gradient. The account predicts that (a) running rates on simple FI schedules should increase with increasing rate of reinforcement, in a manner close to that required by Herrnstein's equation, (b) improving temporal control during acquisition should be associated with increasing running rates, (c) two-valued mixed-FI schedules with equiprobable components should produce complex results, with peak rates sometimes being higher on the longer component schedule, and (d) that effects of reinforcement probability on mixed-FI should affect the response rate at the time of the shorter component only. All these predictions were confirmed by data, although effects in some experiments remain outside the scope of the model. In general, delay of reinforcement as a determinant of response rate on FI and related schedules (rather than temporal control on such schedules) seems a useful starting point for a more thorough analysis of some neglected questions about performance on FI and related schedules.     

Resistance to extinction and behavioral momentum

In the metaphor of behavioral momentum, reinforcement is assumed to strengthen discriminated operant behavior in the sense of increasing its resistance to disruption, and extinction is viewed as disruption by contingency termination and reinforcer omission. In multiple schedules of intermittent reinforcement, resistance to extinction is an increasing function of reinforcer rate, consistent with a model based on the momentum metaphor. The partial-reinforcement extinction effect, which opposes the effects of reinforcer rate, can be explained by the large disruptive effect of terminating continuous reinforcement despite its strengthening effect during training. Inclusion of a term for the context of reinforcement during training allows the model to account for a wide range of multiple-schedule extinction data and makes contact with other formulations. The relation between resistance to extinction and reinforcer rate on single schedules of intermittent reinforcement is exactly opposite to that for multiple schedules over the same range of reinforcer rates; however, the momentum model can give an account of resistance to extinction in single as well as multiple schedules. An alternative analysis based on the number of reinforcers omitted to an extinction criterion supports the conclusion that response strength is an increasing function of reinforcer rate during training.     

Reinforcer satiation and resistance to change of responding maintained by qualitatively different reinforcers

In previous research on resistance to change, differential disruption of operant behavior by satiation has been used to assess the relative strength of responding maintained by different rates or magnitudes of the same reinforcer in different stimulus contexts. The present experiment examined resistance to disruption by satiation of one reinforcer type when qualitatively different reinforcers were arranged in different contexts. Rats earned either food pellets or a 15% sucrose solution on variable-interval 60-s schedules of reinforcement in the two components of a multiple schedule. Resistance to satiation was assessed by providing free access either to food pellets or the sucrose solution prior to or during sessions. Responding systematically decreased more relative to baseline in the component associated with the satiated reinforcer. These findings suggest that when qualitatively different reinforcers maintain responding, relative resistance to change depends upon the relations between reinforcers and disrupter types.     

Effects of reinforcer magnitude on response acquisition with unsignaled delayed reinforcement

Sixteen rats received eight 1-h sessions of a tandem fixed-ratio 1 differential-reinforcement-of-other-behavior 30-s schedule with reinforcer magnitude at one or six food pellet(s) across groups of eight rats. The larger reinforcer magnitude established the lever press more effectively. First, mean schedule completions differed across groups, in terms of both their overall difference and in their increase across sessions. Second, whereas the larger reinforcer magnitude established reliable response acquisition in all eight rats by the end of the experiment, the smaller reinforcer magnitude only established reliable response acquisition in four of eight rats. The present results systematically replicate earlier findings obtaining more reliable response acquisition with unsignaled delayed reinforcement with greater food deprivation. Taken together, this work demonstrates the influence of motivational variables on response acquisition with unsignaled delayed reinforcement.     

Rapid acquisition of preference in concurrent schedules: effects of reinforcement amount

Four pigeons were trained on concurrent variable-interval 30-s schedules. Relative reinforcer amounts arranged across the two alternatives were varied across sessions according to a pseudorandom binary sequence [cf., Hunter, I., Davison, M., 1985. Determination of a behavioral transfer function: white-noise analysis of session-to-session response-ratio dynamics on concurrent VI schedules. J. Exp. Anal. Behav. 43, 43-59]; the ratios (left/right) were either 1/7 or 7/1. Reinforcer amount was manipulated by varying the number of 1.2s hopper presentations. Sessions ended after 30 reinforcers (15 for each alternative). After approximately 30 sessions, response ratios for all pigeons began to track the changes in amount ratio (i.e., subjects' responding showed a moderate increase in sensitivity of responding to reinforcer amount). Characteristics of responding were similar to procedures in which reinforcer rate and immediacy have been manipulated, although sensitivity estimates for amount were lower than those previously obtained with rate and immediacy. This procedure may serve as a useful method for studying the effects of certain environmental manipulations (e.g., drug administration) on sensitivity to reinforcer amount.     

Role of conditioned reinforcers in the initiation, maintenance and extinction of drug-seeking behavior.

The development of a secondary reinforcer as a result of associating a neutral stimulus (buzzer) with intravenous (IV) doses of morpine was studied in rats. Secondary reinforcement developed in the absence of physical dependence and followed the association of the stimulus with either response-contingent or non-contingent injections of morphine. Strength of the conditioned reinforcer, measured in terms of responding on a lever for the stimulus plus infusion of saline solution, was proportional to the unit dosage of morphine employed in pairings of buzzer and drug. When extinction of the lever-press response for IV morphine was conducted (by substituting saline for morphine solution) in the absence of the conditioned reinforcing stimulus, it was seen later that the stimulus could still elicit lever responses, until it too had been present for a sufficient interval of non-reinforced responding. Similarly, extinction of the response for morphine by blocking its action with naloxone in the absence of the stimulus did not eliminate the conditioned reinforcement. Another study showed that a passive, subcutaneous (SC) dose of morphine served to maintain lever-pressing on a contingency of buzzer plus saline infusion. Furthermore, the stimuli resulting from the presence of morphine (after a SC injection) were able to reinstate the lever-responding with only the buzzer-saline contingency when such responses had previously been extinguished. Moreover, it was shown that d-amphetamine could restore responding under the same conditions, and that morphine could also do so for rats in which the primary reinforcer had been d-amphetamine. It is suggested that animal data such as these show that procedures designed for the elimination of human drug-taking behavior must take into account secondary reinforcers as well as the primary reinforcer(s).     

Theoretical effects of radioligand diffusional gradients and microscopic neuroreceptor distribution inin vivo kinetic studies

A simplified one-dimensional model system was used to test the possibility that physically realistic parameters would lead to the prediction of microscopic heterogeneity of radioligand distribution in the brain and that microscopic heterogeneity of radioligand and neuroreceptor distribution could influence the macroscopically observedin vivo kinetics. The model was represented mathematically by a partial differential equation which is similar to the heat diffusion equation, but with special boundary conditions. The equation was solved analytically under the condition of negligible receptor occupancy by inversion of the Laplace transform and in the more general case of arbitrary receptor occupancy by cubic spline approximation. In simulations with physically reasonable values for rate constants and parameters, we find that significant radioligand gradients can occur. Thus, the level of radioligand in the immediate vicinity of the receptor may be substantially different from the average level in a macroscopically measured region of interest. In order to analyze the simulated data, we derived a rigorous steady-state solution, including both a statement of necessary and sufficient conditions for the validity of the steady-state approximation as well as a demonstration of the proper technique for assessing the consistency of the derived parameter with the requirements of the approximation. The radioligand heterogeneity leads to significant errors in the parameters estimated in the steady-state kinetic analysis. In particular, the pseudo first-order rate constant for radioligand-neuroreceptor association, which is often used as a measure of the total amount of neuroreceptor, is underestimated. The first-order rate constant for radioligand-neuroreceptor dissociation is also underestimated. These effects can partially account for the experimentally-observed discrepancy betweenin vivo andin vitro estimates of these kinetic parameters.