Reinforcement and learning

Evidence has been accumulating to support the process of reinforcement as a potential mechanism in speciation. In many species, mate choice decisions are influenced by cultural factors, including learned mating preferences (sexual imprinting) or learned mate attraction signals (e.g., bird song). It has been postulated that learning can have a strong impact on the likelihood of speciation and perhaps on the process of reinforcement, but no models have explicitly considered learning in a reinforcement context. We review the evidence that suggests that learning may be involved in speciation and reinforcement, and present a model of reinforcement via learned preferences. We show that not only can reinforcement occur when preferences are learned by imprinting, but that such preferences can maintain species differences easily in comparison with both autosomal and sex-linked genetically inherited preferences. We highlight the need for more explicit study of the connection between the behavioral process of learning and the evolutionary process of reinforcement in natural systems.     

Lectin and Neuropeptide Labeling of Separate Populations of Dorsal Root Ganglion Neurons and Associated “Nociceptor” Thin Axons in Rat Testis and Cornea Whole-Mount Preparations

As part of a program to explore patterns of innervation by nociceptor-related thin sensory axons in a variety of peripheral regions, we have labeled calcitonin gene-related peptide immunoreactive (CGRP-IR) nerve fibers in whole mounts of rat testicular tunica vasculosa and cornea. Efforts were undertaken to visualize the numerically significant fluoride-resistant acid phosphatase (FRAP)-containing axon population, whose peripheral endings have heretofore remained undemonstrable due to technical limitations of currently available acid phosphatase methods. Various histochemical markers that colocalize with FRAP in dorsal root ganglion (DRG) and spinal cord were examined, and a plant lectin, Griffonia simplicifolia I-B₄, has been identified that not only selectively labels FRAP(+) sensory ganglion cells and central terminals in spinal cord, but also differentially stains a large number of thin axons in testicular and corneal whole mounts. Slender lectin-labeled fibers are abundant in cornea, and are distributed throughout tunica vasculosa preparations unrelated to blood vessels. CGRP-IR axons, in contrast, maintain close adherence to vascular patterns and are more coarse and varicose in appearance.

Lectin staining therefore provides the first practical and specific method for visualization of peripheral FRAP(+) axons consisting principally of sensory C fibers but possibly including a small number of unmyelinated autonomic axons. It should now be feasible, using individual whole-mount preparations from various peripheral nociceptor-innervated tissues, to examine the distributions of both peptidergic and FRAP(+) fibers, which together comprise the vast majority of thin sensory axons. It may then be possible to correlate the observed anatomical patterns with knowledge regarding properties of corresponding physiologically characterized receptive fields.     

Tug-of-war model for the two-bandit problem: Nonlocally-correlated parallel exploration via resource conservation

We propose a model - the “tug-of-war (TOW) model” - to conduct unique parallel searches using many nonlocally-correlated search agents. The model is based on the property of a single-celled amoeba, the true slime mold Physarum, which maintains a constant intracellular resource volume while collecting environmental information by concurrently expanding and shrinking its branches. The conservation law entails a “nonlocal correlation” among the branches, i.e., volume increment in one branch is immediately compensated by volume decrement(s) in the other branch(es). This nonlocal correlation was shown to be useful for decision making in the case of a dilemma. The multi-armed bandit problem is to determine the optimal strategy for maximizing the total reward sum with incompatible demands, by either exploiting the rewards obtained using the already collected information or exploring new information for acquiring higher payoffs involving risks. Our model can efficiently manage the “exploration-exploitation dilemma” and exhibits good performances. The average accuracy rate of our model is higher than those of well-known algorithms such as the modified ?-greedy algorithm and modified softmax algorithm, especially, for solving relatively difficult problems. Moreover, our model flexibly adapts to changing environments, a property essential for living organisms surviving in uncertain environments.     

A Learning Strategy for Predator Preying on Edible and Inedible Prey

In this paper I propose a reinforcement learning model for a predator preying upon two types of prey, the unpalatable (noxious) models, and the palatable mimics. The latter type of prey resembles the models in appearance so as to derive some protection from the predator who must avoid the unpalatable models. Essentially the predator is treated as a learning automaton adopting a simple reinforcement learning strategy in order to increase its consumption of palatable prey and reduce the consumption of unpalatable ones. The populations of both mimics and models are assumed to grow logistically.     

Temporal generalization accounts for response resurgence in the peak procedure

The peak interval (PI) procedure is commonly used to evaluate animals' ability to produce timed intervals. It consists of presenting fixed interval (FI) schedules in which some of the trials are replaced by extended non-reinforced trials. Responding will often resume (resurge) at the end of the non-reinforced trials unless precautions are taken to prevent it. Response resurgence was replicated in rats and pigeons. Variation of the durations of the FI and the non-reinforced probe trials showed it to be dependent on the time when reinforcement is expected. Timing of both the normal time to reinforcement, and the subsequent time to reinforcement during the probe trials followed Weber's law. A quantitative model of resurgence is described, suggesting how animals respond to the signaling properties of reinforcement omission. Model results were simulated using a stochastic binary counter.     

Choice in a variable environment: Effects of d-amphetamine on sensitivity to reinforcement

Four pigeons responded under a 7-component mixed schedule in which each component arranged a different left:right reinforcer ratio (27:1, 9:1, 3:1, 1:1, 1:3, 1:9, 1:27). Components were unsignaled, and the order within each session was randomly determined. After extensive exposure to these contingencies, effects of a range of doses of d-amphetamine (0.3-5.6 mg/kg) on estimates of sensitivity to reinforcement at several levels of analysis were assessed. Under non-drug conditions, the structure of choice was similar to that previously reported under this procedure. That is, responding adjusted within components to the reinforcer ratio in effect (i.e., sensitivity estimates were higher in the 2nd than in the 1st half of components), and individual reinforcers produced “preference pulses” (i.e., each food presentation produced an immediate, local, shift in preference toward the response that just produced food). Although there was a general tendency for d-amphetamine to reduce overall sensitivity to reinforcement, the size of this effect and its reliability varied across pigeons. Further analysis, however, revealed that intermediate d-amphetamine doses consistently reduced sensitivity immediately following reinforcer presentations; that is, these doses consistently attenuated preference pulses.     

Extinction as discrimination: the molar view

The traditional molecular view of behavior explains extinction as the dissipation or inhibition of strength, formerly built up by contiguous reinforcement. In obstinate opposition to this explanation was the partial-reinforcement extinction effect: a partially reinforced response extinguishes more slowly than a continuously reinforced response. It suggests instead that extinction is discrimination. Four pigeons were exposed to daily sessions in which a variable period of food delivery, produced by pecking on a variable-interval schedule, was followed by extinction. The rate of food delivery was varied over a wide range across conditions. Varying the amount of food per delivery inversely with rate of delivery kept response rate from varying excessively. The results confirmed and extended the partial-reinforcement effect; persistence of pecking and time to extinction were inversely related to rate of obtaining food. The results support the molar view of extinction, not as loss of strength of a particular discrete response, but as a transition from one allocation of time among activities to another. Although molecular theories dismiss discrimination due to repeated training and extinction as an impurity or complication, repeated cycles of availability and privation are probably typical of the environment in which most vertebrate species evolved.     

Timing and magnitude of lumbar spine contribution to trunk forward bending and backward return in patients with acute low back pain

Alterations in the lumbo-pelvic coordination denote changes in neuromuscular control of trunk motion as well as load sharing between passive and active tissues in the lower back. Differences in timing and magnitude aspects of lumbo-pelvic coordination between patients with chronic low back pain (LBP) and asymptomatic individuals have been reported; yet, the literature on lumbo-pelvic coordination in patients with acute LBP is scant. A case-control study was conducted to explore the differences in timing and magnitude aspects of lumbo-pelvic coordination between females with (n=19) and without (n=19) acute LBP. Participants in each group completed one experimental session wherein they performed trunk forward bending and backward return at preferred and fast paces. The amount of lumbar contribution to trunk motion (as the magnitude aspect) as well as the mean absolute relative phase (MARP) and deviation phase (DP) between thoracic and pelvic rotations (as the timing aspect) of lumbo-pelvic coordination were calculated. The lumbar contribution to trunk motion in the 2nd and the 3rd quarters of both forward bending and backward return phases was significantly smaller in the patient than the control group. The MARP and the DP were smaller in the patient vs. the control group during entire motion. The reduced lumbar contribution to trunk motion as well as the more in-phase and less variable lumbo-pelvic coordination in patients with acute LBP compared to the asymptomatic controls is likely the result of a neuromuscular adaptation to reduce painful deformation and to protect injured lower back tissues.