Interval time-place learning by rats: varying reinforcement contingencies

Two experiments with rats were conducted to study interval time-place learning when the spatiotemporal contingencies of food availability were more similar to those likely to be encountered in natural environments, than those employed in prior research. In Experiment 1, food was always available on three levers on a variable ratio (VR) 35 schedule. A VR8 schedule was in effect on Lever 1 for 5 min, then on Lever 2 for 5 min, and so forth. While rats learned to restrict the majority of their responding to the lever that provided the highest density of reinforcement, they seemed to rely on a win-stay/lose-shift strategy rather than a timing strategy. In Experiment 2, the four levers provided food on variable ratios of 15, 8, 15, and 30, each for 3 min. As expected the rats learned these contingencies. A novel finding was that the rats had a spike in response rate immediately following a change from a higher to lower reinforcement density. It is concluded that rats exposed to spatiotemporal contingencies behave so as to maximize the rate of obtained reinforcement.     

The operant reserve: a computer simulation in (accelerated) real time

In Skinner's Reflex Reserve theory, reinforced responses added to a reserve depleted by responding. It could not handle the finding that partial reinforcement generated more responding than continuous reinforcement, but it would have worked if its growth had depended not just on the last response but also on earlier responses preceding a reinforcer, each weighted by delay. In that case, partial reinforcement generates steady states in which reserve decrements produced by responding balance increments produced when reinforcers follow responding. A computer simulation arranged schedules for responses produced with probabilities proportional to reserve size. Each response subtracted a fixed amount from the reserve and added an amount weighted by the reciprocal of the time to the next reinforcer. Simulated cumulative records and quantitative data for extinction, random-ratio, random-interval, and other schedules were consistent with those of real performances, including some effects of history. The model also simulated rapid performance transitions with changed contingencies that did not depend on molar variables or on differential reinforcement of inter-response times. The simulation can be extended to inhomogeneous contingencies by way of continua of reserves arrayed along response and time dimensions, and to concurrent performances and stimulus control by way of different reserves created for different response classes.     

Ontogeny has a phylogeny: background to adjunctive behaviors in pigeons and budgerigars

Animals coping with operant conditioning tasks often show behaviors that are not recorded by keys, levers and similar response transducers. Nevertheless, these adjunctive behaviors should not be disposed of by classifying them as incidental. Often they are found to be at least partially influenced by the experimentally programmed contingencies, and under certain conditions they can in turn influence conditioned behaviors. Here we describe the occurrence and characteristics of two such behaviors, stimulus grasping in operantly key-pecking pigeons and intra-delay stereotypies in a delayed matching-to-sample task with budgerigars. It is argued that for a proper account of these behaviors it is necessary to refer to a behavioral systems approach that appeals to longer ranging ontogenetic and phylogenetic histories than is usually considered in the psychological literature. The gaping towards on-key stimuli by pigeons is attributed to the hypothesis that operantly conditioned key-pecks probably relate to a grasp-pecking response that is normally executed towards non-edible items covering food. The intra-delay behaviors shown by the budgerigars are assumed to have originated from stress-induced displacement responses that adventitiously came under the influence of differential reinforcement contingencies. Finally, we discuss what kinds of evidence are needed to put these hypothetical explanations on a more certain footing.     

Choice between contingencies of variation: Effects of the requirement of variation upon preference

The present study investigated whether choices between contingencies of variation are affected by the degree of variability required. For such, five pigeons were exposed to a concurrent chain schedule. In the initial links, responses in one key initiated the terminal link with the most stringent variation requirement while responses in the other key initiated the terminal link with the least stringent variation requirement. In both terminal links, four-responses sequences were reinforced according to a variation criterion, which favored less frequent and less recent sequences. The probability of reinforcement in the terminal link with the least stringent criterion was manipulated in order to generate similar percentage and rate of reinforcers in both terminal links. Choices for the terminal link with the least stringent criterion were more frequent than choices for the terminal link with the most stringent criterion. It is possible that situations that demand lower levels of behavior variability are chosen due to the lower response cost correlated to those situations.     

A two-stage model for concurrent sequences

Schneider and Davison [Schneider, S.M., Davison, M., 2005. Demarcated response sequences and the generalised matching law. Behav. Proc. 70, 51–61] showed that the generalised matching law applied to concurrent free-operant two-response sequences. When sufficient temporal spacing is required between the responses, however, neither the response-level nor the sequence-level forms of the generalised matching law provide good fits. An alternative “two-stage sensitivity” model with fewer free parameters features two types of sensitivity to the reinforcement contingencies on sequences. When temporal spacing between the responses is long, the “response distribution sensitivity” parameter describes sensitivity only of the individual responses to the sequence-level contingencies. At a threshold level, this sensitivity reaches a maximum. When spacing is shorter than threshold, the “response order sensitivity” parameter reflects a new sensitivity to the order of the responses within sequences. As this sensitivity approaches its maximum, sequence matching is achieved. For both stages, a changeover parameter describes bias against sequences that require changeovers between the two responses. The model fit data ranging from near-response matching with long minimum inter-response times (IRTs) to sequence matching with no minimum IRTs, using two species and a variety of sequence reinforcer distributions. Rats differed from pigeons in achieving sequence matching only at a nonzero minimum IRT. In a comparison based on pigeon data with no minimum IRT, the two-stage sensitivity model was more efficient than the generalised matching law according to the Akaike criterion. The logic of the model suggests a new way of understanding the mechanisms underlying behavioural units.     

Effects of partial reinforcement and time between reinforced trials on terminal response rate in pigeon autoshaping

Partial reinforcement often leads to asymptotically higher rates of responding and number of trials with a response than does continuous reinforcement in pigeon autoshaping. However, comparisons typically involve a partial reinforcement schedule that differs from the continuous reinforcement schedule in both time between reinforced trials and probability of reinforcement. Two experiments examined the relative contributions of these two manipulations to asymptotic response rate. Results suggest that the greater responding previously seen with partial reinforcement is primarily due to differential probability of reinforcement and not differential time between reinforced trials. Further, once established, differences in responding are resistant to a change in stimulus and contingency. Secondary response theories of autoshaped responding (theories that posit additional response-augmenting or response-attenuating mechanisms specific to partial or continuous reinforcement) cannot fully accommodate the current body of data. It is suggested that researchers who study pigeon autoshaping train animals on a common task prior to training them under different conditions.     

Effect of contingent auditory stimuli on concurrent schedule performance: an alternative punisher to electric shock

This study explored whether load auditory stimuli could be used as functional punishing stimuli in place of electric shock. Three experiments examined the effect of a loud auditory stimulus on rats’ responding maintained by a concurrent reinforcement schedule. In Experiment 1, overall response rate decreased when a concurrent 1.5 s tone presentation schedule was superimposed on the concurrent variable interval (VI) 180-s, VI 180-s reinforcement schedule. On the contrary, response rate increased when a click presentation schedule was added. In Experiment 2, the extent of the response suppression with a 1.5 s tone presentation varied as a function of the frequency of the reinforcement schedule maintaining responses; the leaner the schedule employed, the greater the response suppression. In Experiment 3, response suppression was observed to be inversely related to the duration of the tone; response facilitation was observed when a 3.0-s tone was used. In Experiments 1 and 2, a preference shift towards the alternative with richer reinforcement was observed when the tone schedule was added. In contrast, the preference shifted towards the leaner alternative when the click or longer duration stimulus was used. These results imply that both the type and duration of a loud auditory stimulus, as well as the reinforcement schedule maintaining responses, have a critical role in determining the effect of the stimuli on responding. They also suggest that a loud auditory stimulus can be used as a positive punisher in a choice situation for rats, when the duration of the tone is brief, and the reinforcement schedule maintaining responses is lean.     

Time-based reward maximization

Humans and animals time intervals from seconds to minutes with high accuracy but limited precision. Consequently, time-based decisions are inevitably subjected to our endogenous timing uncertainty, and thus require temporal risk assessment. In this study, we tested temporal risk assessment ability of humans when participants had to withhold each subsequent response for a minimum duration to earn reward and each response reset the trial time. Premature responses were not penalized in Experiment 1 but were penalized in Experiment 2. Participants tried to maximize reward within a fixed session time (over eight sessions) by pressing a key. No instructions were provided regarding the task rules/parameters. We evaluated empirical performance within the framework of optimality that was based on the level of endogenous timing uncertainty and the payoff structure. Participants nearly tracked the optimal target inter-response times (IRTs) that changed as a function of the level of timing uncertainty and maximized the reward rate in both experiments. Acquisition of optimal target IRT was rapid and abrupt without any further improvement or worsening. These results constitute an example of optimal temporal risk assessment performance in a task that required finding the optimal trade-off between the ‘speed’ (timing) and ‘accuracy’ (reward probability) of timed responses for reward maximization.     

Reinforcement of variations and repetitions along three independent response dimensions

Reinforcement was presented contingent upon human subjects simultaneously varying three dimensions of an operant response. The response was drawing rectangles on a computer screen. The dimensions were area of the rectangle, its location on the screen, and its shape. In Experiment 1, an experimental group was reinforced for satisfying the three-part variability contingency. A control group was equally reinforced for drawing rectangles but independently of levels of variability. Results showed that the experimental group varied significantly more along each of the dimensions than did the control group. In Experiment 2, another group of subjects was reinforced for repeating instances along one of the dimensions, e.g. repeatedly draw a rectangle in approximately the same location, while simultaneously varying along the other two dimensions. The subjects learned to satisfy these contingencies as well. These results show reinforcement simultaneously and independently controls the variability of three orthogonal dimensions of a response.     

Performance correlates of social behavior and organization: Effects of group formation on operant performance in rhesus monkeys (M. mulatta)

The performance of six adult rhesus monkeys on a fixed interval 1-min reinforcement schedule was examined under conditions where the reinforcement probabilities were either 1.00 or .80. The animals were tested with a reinforcement probability of 1.00 immediately after removal from their social group, tested again several months later with reinforcement probabilities of 1.00 and .80, and then retested with a reinforcement probability of .80 before, during, and after the formation of a new social group with the six monkeys. The results supported an earlier report in which it was found that high ranking animals responded at a lower rate under a reinforcement probability of 1.00 than did low ranking animals and that the former also had a higher ratio of nonreinforced to reinforced responses than the lower ranked animals. These relationships were present in five of the six animals immediately after removal of the animals from their original group and during and after formation of the new group. They did not appear on tests conducted after the animals had been out of a social group for several months. A third result was a suppression of operant performance which appeared in all animals following group formation. The magnitude and duration of this effect was inversely related to social rank and the time the animal had been a member of the group. The data are discussed in terms of a carry-over of the effects of recent social experience on two factors: a more deliberate rate of response by higher ranking animals under 100% reinforcement and an inhibition of response bursting following omission of reinforcement by the lower ranking monkeys. Supported by USAMRDC Contract No. DADA 17-73-C-3007.     

Learned helplessness: effects of response requirement and interval between treatment and testing

Three experiments investigated learned helplessness in rats manipulating response requirements, shock duration, and intervals between treatment and testing. In Experiment 1, rats previously exposed to uncontrollable or no shocks were tested under one of four different contingencies of negative reinforcement: FR 1 or FR 2 escape contingency for running, and FR1 escape contingency for jumping (differing for the maximum shock duration of 10s or 30s). The results showed that the uncontrollable shocks produced a clear operant learning deficit (learned helplessness effect) only when the animals were tested under the jumping FR 1 escape contingency with 10-s max shock duration. Experiment 2 isolated of the effects of uncontrollability from shock exposure per se and showed that the escape deficit observed using the FR 1 escape jumping response (10-s shock duration) was produced by the uncontrollability of shock. Experiment 3 showed that using the FR 1 jumping escape contingency in the test, the learned helplessness effect was observed one, 14 or 28 days after treatment. These results suggest that running may not be an appropriate test for learned helplessness, and that many diverging results found in the literature might be accounted for by the confounding effects of respondent and operant contingencies present when running is required of rats.     

Extinction of an appetitive operant response after administration of MSH

Hungry rats were trained to press a lever in order to obtain food on a fixed ratio (FR) or variable ratio (VR) of reinforcement. Rats trained on the FR schedule and injected with synthetic α-MSH had delayed extinction of the task as compared with control rats injected with diluent. The results show that MSH affects the behavior of rats in another type of behavioral situation involving an appetitive operant response.     

Reaction times identify a Pavlovian component in a two-choice discrimination

Six pigeons discriminated on discrete trials between two colors. In Experiment 1, two luminous spots were both either blue or green and the reinforced responses were “peck left” for blue and “peck right” for green. In Experiment 2, the hue of a center spot controlled subsequent choice pecks to left or right. In both experiments response bias was manipulated in two ways. During stimulus frequency (“SF”) sessions correct responses brought food on 40% of trials; in “imbalanced” blocks of sessions one hue appeared on 80% of trials and the other on 20%. During reinforcement probability (“RNF”) sessions the hues appeared equally often, but in imbalanced blocks the hues signaled different reinforcement probabilities, either 64% or 16%. In “balanced” control blocks the hues appeared equally often and were both reinforced at 40%. The experiments gave similar results. When bias was computed from choice percentages the imbalanced conditions yielded substantial response bias, and the amount of bias was about the same under RNF and SF treatments. However, reaction times (RTs) gave a different outcome. RNF imbalance slowed responses directed at the less reinforced stimulus, but SF imbalance had little RT effect (Experiment 1) or no effect (Experiment 2). These results suggest that choice was controlled by an instrumental stimulus-response-reinforcement association, whereas RTs were controlled by a Pavlovian stimulus-reinforcement association.     

Transitional choice behavior in concurrent-chain schedules

The choice responses of four pigeons were examined in 20 periods of transition in a concurrent-chain procedure with variable-interval schedules as initial links and fixed delays to reinforcement as terminal links. In some conditions, the delays to reinforcement were different for the two terminal links, and changes in preference were recorded after the delays for the two response keys were switched. In other conditions, the reinforcer delays were equal for the two keys, but which key delivered 80% of the reinforcers was periodically switched. Choice proportions changed more quickly after a switch in reinforcement percentages than after a switch in the delays, thereby contradicting the hypothesis that faster changes would occur when the switch in conditions was easier to discriminate. Analyses of response sequences showed that the effects of individual reinforcers were larger and lasted longer in conditions with changing reinforcement percentages than in conditions with changing terminal-link delays. Rates of change in choice behavior do not appear to be limited by the unpredictability of variable reinforcement schedules, because the changes in behavior were slow and gradual even when there was a large and sudden change in reinforcer delays.     

Delay of gratification and delay discounting in rats

Delay discounting (DD) and delay of gratification (DG) are two measures of impulsive behavior often viewed as reflecting the same or equivalent processes. However, there are some key differences in the contingencies of reinforcement between the procedures that may have implications for understanding impulsivity. This study used DD and DG procedures to determine if differences in contingencies of reinforcement specified by DD and DG alters how much organisms discount the value of delayed reinforcers. Twenty-four water-deprived rats performed one of two Adjusting Amount procedures, which consisted of repeated choices between a fixed amount of water (250 &mgr;l) delivered after a delay (0, 4, 8, 16, or 32 s) and an adjusting, usually lesser amount delivered immediately. Half of the rats (n=12) performed a DD procedure designed to assess preference for immediate over delayed reinforcers in which they had discrete choices between the immediate and delayed amounts of water. A DG procedure was used for the other half of the rats (n=12). In the DG procedure rats also selected between immediate and delayed alternatives, but if they chose the delayed alternative they could switch to and receive the immediate alternative at any time during the delay to the larger reward. In the DD procedure switching responses were not reinforced but were still recorded and used for analyses. The DD functions of the two groups did not differ significantly. However, at the longer delays, the DG group made significantly fewer switching responses than the DD group. A possible role of response inhibition in the DG procedure is discussed.     

Resistance to change as a function of concurrent reinforcer magnitude

Six pigeons responded on two keys in each of three signalled multiple-schedule components, and resistance to disruption of responding on one (target) key by extinction and by response-independent food presented during inter-component blackouts was studied. Alternative reinforcement of different magnitudes was contingent on pecking a non-target key in two components, and in the third only the target response was reinforced. Resistance to change varied with the overall quantity of reinforcement in the component, regardless of whether reinforcers were contingent on the target or non-target response, but did not differ across the two key locations. These results using different magnitudes of reinforcement confirm previous findings using rate of reinforcement as the variable, and suggest that resistance to change is dependent on stimulus-reinforcer rather than response-reinforcer contingencies.     

Influence of budget and reinforcement location on risk-sensitive preference

Little is known about the effect that procedural variables have on risk-sensitive preference. This study assessed the effect of procedural variables on pigeons' choice between a fixed and variable amount of reinforcement (amount risk) and, in a separate condition, between a fixed and variable delay until reinforcement (delay risk). Experiment 1 investigated the impact of water reinforcement and risk dimension when pigeons were in a restrictive budget, where access to water was less than that necessary to maintain current body weight, and a condition where the pigeons had ample access to water. Pigeons exhibited a greater tendency to prefer the variable alternative for delay risk than for amount risk in both restrictive and ample budgets. Varying water budget had no effect on risk preference. Experiment 2 investigated the influence of water reinforcer location while in a restrictive budget, in which reinforcers were delivered to a single location, two distinct locations, or a randomly selected location. With amount risk, pigeons were risk averse when reinforcers were delivered in separate or random locations and were indifferent to risk when delivered to a single location. With delay risk, pigeons were generally risk prone with no effect from reinforcement location. The finding that pigeons were risk averse when reinforcers were delivered to separate locations and were indifferent to risk when delivered to a single location offers a methodological explanation to the inconsistent findings in the literature with amount risk.     

Training a new response using conditioned reinforcement

Some of the most frequently used methods in the study of conditioned reinforcement seem to be insufficient to demonstrate the effect. The clearest way to assess this phenomenon is the training of a new response. In the present study, rats were exposed to a situation in which a primary reinforcer and an arbitrary stimulus were paired and subsequently the effect of this arbitrary event was assessed by presenting it following a new response. Subjects under these conditions emitted more responses compared to their own responding before the pairing and to their responding on a similar operandum that was available concurrently that had no programmed consequences. Response rates also were higher compared to responding by subjects in similar conditions in which there was no contingency (a) between the arbitrary stimulus and the reinforcer, (b) between the response and the arbitrary stimulus or (c) both. Results are discussed in terms of necessary and sufficient conditions to study conditioned reinforcement.     

Interval time-place learning in young children

While previous research has investigated the ability of animals to learn the spatial and temporal contingencies of biologically significant events (known as time-place learning), this ability has not been studied in humans. Children ranging from 5 to 10 years old were tested on a modified interval time-place learning task using a touchscreen computer. Results demonstrate the children were able to quickly learn both the timing and the sequence of this task. Despite a lack of anticipation on baseline trials, the children continued to follow the spatio-temporal contingencies in probe sessions where these contingencies were removed. Performance on the probe sessions provide strong evidence that the children had learned the spatio-temporal contingencies. Future research is needed to determine what age-related changes in iTPL occur. Furthermore, it is argued that this procedure can be used to extend interval timing in research in children, including, but not limited to, investigation of scalar timing with longer durations than have previously been investigated.