The effect of non-contingent negative and positive reinforcement operations on the acquisition of superstitious behaviors

The differential effects of non-contingent positive and negative reinforcement operations on the acquisition of superstitious behaviors and rules were investigated in two experiments. College students were instructed to try to produce and/or keep the word "GOOD" on a computer screen (positive reinforcement), or to try to prevent and/or remove the word "BAD" from the screen (negative reinforcement) using response keys. Data from both experiments indicated that participants exposed to lean schedules of negative reinforcers believed that they had greater control over non-contingent stimulus events than participants exposed to either rich or lean schedules of positive reinforcers. These findings and results from research investigating everyday superstitious activities suggest that, relative to positive reinforcement, negative reinforcement operations may provide a more fertile condition for the development and maintenance of superstitious behaviors.     

Learning to differentiate microintervals of time using verbal feedback

Reaction time (RT) and the number of correct estimations of time microintervals (10 and 180 ms) between two visual stimuli were recorded in healthy subjects. It has been shown that 10 ms interval is better estimated when the stimuli are presented in the right visual field, i.e. when they are addressed directly to the left hemisphere. At the same time the number of correct estimations of 180 ms interval is greater and their RT is less when the stimuli are addressed directly to the right hemisphere. This points to different hemispheric mechanisms of time microintervals estimation. Study of the influence of different forms of verbal reinforcement on this learning has shown that after positive reinforcement (the word "good") the number of correct estimations is on average by 10% greater than after negative reinforcement (the word "error"). This may be connected with such processes as isolation and identification of erroneous reaction.     

Animal Minds and Animal Emotions

The possibility of conscious experiences of emotions in non-humananimals has been much less explored than that of conscious experiencesassociated with carrying out complex cognitive tasks. However,no great cognitive powers are needed to feel hunger or painand it may be that the capacity to feel emotions is widespreadin the animal kingdom. Since plants can show surprisingly sophisticated"choice" and "decision-making" mechanisms and yet we would notwish to imply that they are conscious, attribution of emotionsto animals has to be done with care. Whether or not an animalpossesses anticipatory mechanisms associated with positive andnegative reinforcement learning may be a guide as to whetherit has evolved emotions.     

Effect of acute and chronic deprivation of catecholaminergic system activity induced by 6-hydroxydopamine on the behavior of the rat

On 202 male rats of Wistar line, a study was carried out of the effect of chronic and acute deprivations of the brain CA-systems activity resulting from administration of 6-OHDA on investigating behaviour and learning. Chronic deprivation of CA-systems activity by neonatal administration of 6-OHDA (100 mg/kg subcutaneously) and their acute deprivation by intracerebral administration of 6-OHDA to adult rats (150 mkg in each lateral ventriculus) was accompanied by similar deep changes of behaviour. Both forms of deprivation reduced the investigating activity of the animals in the open field. In both cases, the above 6-OHDA dozes sharply impeded the learning of animals with emotionally negative reinforcement, with no significant influence on learning with emotionally positive reinforcement. Both forms of deprivation of CA-systems activity weakened the reaction of frustration elicited by a sharp reduction of food reinforcement.     

Possible mechanisms of the divergent effects of ACTH4--10 and its analog containing the D-isomer of phenylalanine on behavior

The effect of two oligopeptides--ACTG4-10 and hexopeptide met-glu-his-D-phen-lis-L-phen (Dphen-GP) on memorizing the situation and on orienting-investigating reaction was studied in albino rats by the method of elaboration of food-procuring habit in T-maze and by the method of "open field". It was shown that these peptides in a dose of 15 mcg/kg with certain periods of administration, have an opposite effect on maze learning but a similar effect on memorizing in the "open field". ACTG4-10 slightly increases motor activity in the "open field", whereas Dphen-GP decreases it considerably. It is suggested that ACTG4-10 improves the formation of trace processes independently of the sign of reinforcement, whereas Dphen-GP selectively enhances defensive reaction and memorizing, connected with negative reinforcement.     

Changes in the cerebral serotonin and noradrenaline levels in the brains of white rats trained with emotionally different reinforcement]

The changes in the serotonin (5-OT) and noradrenaline (NA) content in the brain of rats taught on the emotionally-positive (food) and emotionally-negative (pain) reinforcement were compared. The process of animal learning was accompanied by increase in the biogenic amine level. But when the teaching was conducted on the emotionally-positive reinforcement there was a greater increase in the 5-OT and NA level than in the case of the emotionally negative reinforcement. The process of animal teaching on food reinforcement was accompanied by an elevation of 5-OT chiefly in the cerebral cortex which apparently reflected the active functioning of the 5-OT system. An intensification of the NA system occurred in teaching the animals in the defence situation. A conclusion was thus drawn that the character of the changes in the biogenic amine level in the brain during learning depended on the emotions experienced by the animal (the emotional reinforcement utilized).     

Similar effects of a beta-carboline and of flumazenil in negatively and positively reinforced learning tasks in mice.

Methyl beta-carboline-3-carboxylate (beta-CCM) and flumazenil (Ro15-1788) are known to be respectively an inverse agonist and an antagonist of the central benzodiazepine-receptor. Surprisingly, these two drugs have shown a similar enhancing effect in a negatively reinforced multiple-trial brightness discrimination task in mice. Thus, to evaluate the role of anxiety in this task, the action of these two drugs were compared in the same learning task with a positive or a negative reinforcement. Mice were trained for sessions of ten trials per day for six consecutive days. The sessions during the first three days took place after administration of beta-CCM (0.3 mg/kg), flumazenil (15 mg/kg) or vehicles of these drugs. A negative reinforcement (electric foot-shock) was used in a first experiment, and a positive one (food reward) in a second experiment. Results showed that, whatever the reinforcement, the two drugs enhance learning in a brightness discrimination task. The hypothesis is that flumazenil could have an inverse agonist profile in learning tasks. The question remains as to whether the flumazenil enhancing learning process results from increased arousal and/or anxiogenic factors, or from a negative modulatory influence of endogenous diazepam-like ligands for benzodiazepine receptors.     

Reinforcement and learning

Evidence has been accumulating to support the process of reinforcement as a potential mechanism in speciation. In many species, mate choice decisions are influenced by cultural factors, including learned mating preferences (sexual imprinting) or learned mate attraction signals (e.g., bird song). It has been postulated that learning can have a strong impact on the likelihood of speciation and perhaps on the process of reinforcement, but no models have explicitly considered learning in a reinforcement context. We review the evidence that suggests that learning may be involved in speciation and reinforcement, and present a model of reinforcement via learned preferences. We show that not only can reinforcement occur when preferences are learned by imprinting, but that such preferences can maintain species differences easily in comparison with both autosomal and sex-linked genetically inherited preferences. We highlight the need for more explicit study of the connection between the behavioral process of learning and the evolutionary process of reinforcement in natural systems.     

Negative Affect Reduces Performance in Implicit Sequence Learning

Background

It is well documented that positive rather than negative moods encourage integrative processing of conscious information. However, the extent to which implicit or unconscious learning can be influenced by affective states remains unclear.

Methodology/Principal Findings

A Serial Reaction Time (SRT) task with sequence structures requiring integration over past trials was adopted to examine the effect of affective states on implicit learning. Music was used to induce and maintain positive and negative affective states. The present study showed that participants in negative rather than positive states learned less of the regularity. Moreover, the knowledge was shown by a Bayesian analysis to be largely unconscious as participants were poor at recognizing the regularity.

Conclusions/Significance

The results demonstrated that negative rather than positive affect inhibited implicit learning of complex structures. Our findings help to understand the effects of affective states on unconscious or implicit processing.     

A parameter control method in reinforcement learning to rapidly follow unexpected environmental changes

In order to rapidly follow unexpected environmental changes, we propose a parameter control method in reinforcement learning that changes each of learning parameters in appropriate directions. We determine each appropriate direction on the basis of relationships between behaviors and neuromodulators by considering an emergency as a key word. Computer experiments show that the agents using our proposed method could rapidly respond to unexpected environmental changes, not depending on either two reinforcement learning algorithms (Q-learning and actor-critic (AC) architecture) or two learning problems (discontinuous and continuous state-action problems).     

Deep Learning of Orthographic Representations in Baboons

What is the origin of our ability to learn orthographic knowledge? We use deep convolutional networks to emulate the primate''s ventral visual stream and explore the recent finding that baboons can be trained to discriminate English words from nonwords [1]. The networks were exposed to the exact same sequence of stimuli and reinforcement signals as the baboons in the experiment, and learned to map real visual inputs (pixels) of letter strings onto binary word/nonword responses. We show that the networks'' highest levels of representations were indeed sensitive to letter combinations as postulated in our previous research. The model also captured the key empirical findings, such as generalization to novel words, along with some intriguing inter-individual differences. The present work shows the merits of deep learning networks that can simulate the whole processing chain all the way from the visual input to the response while allowing researchers to analyze the complex representations that emerge during the learning process.     

Ethanol seeking by long evans rats is not always a goal-directed behavior

Background

Two parallel and interacting processes are said to underlie animal behavior, whereby learning and performance of a behavior is at first via conscious and deliberate (goal-directed) processes, but after initial acquisition, the behavior can become automatic and stimulus-elicited (habitual). With respect to instrumental behaviors, animal learning studies suggest that the duration of training and the action-outcome contingency are two factors involved in the emergence of habitual seeking of “natural” reinforcers (e.g., sweet solutions, food or sucrose pellets). To rigorously test whether behaviors reinforced by abused substances such as ethanol, in particular, similarly become habitual was the primary aim of this study.

Methodology/Principal Findings

Male Long Evans rats underwent extended or limited operant lever press training with 10% sucrose/10% ethanol (10S10E) reinforcement (variable interval (VI) or (VR) ratio schedule of reinforcement), or with 10% sucrose (10S) reinforcement (VI schedule only). Once training and pretesting were complete, the impact of outcome devaluation on operant behavior was evaluated after lithium chloride injections were paired with the reinforcer, or unpaired 24 hours later. After limited, but not extended instrumental training, lever pressing by groups trained under VR with 10S10E and under VI with 10S was sensitive to outcome devaluation. In contrast, responding by both the extended and limited training 10S10E VI groups was not sensitive to ethanol devaluation during the test for habitual behavior.

Conclusions/Significance

Operant behavior by rats trained to self-administer an ethanol-sucrose solution showed variable sensitivity to a change in the value of ethanol, with relative insensitivity developing sooner in animals that received time-variable ethanol reinforcement during training sessions. One important implication, with respect to substance abuse in humans, is that initial learning about the relationship between instrumental actions and the opportunity to consume ethanol-containing drinks can influence the time course for the development or expression of habitual ethanol seeking behavior.     

Effect of diazepam on the formation and reproduction of conditioned cortical reactions in man

Intramuscular injections of seduxen (10 mg), a benzodiasepin drug, with expressed tranquilizing and alarm suppressing effects, were used with the aim of experimental testing of a hypothesis on the neural mechanism of conditioned changes of the P300 wave. Under the action of seduxen no temporary connections could be formed between a perceived nonverbal visual stimulus and a unrecognized emotional word. The seduxen did not suppress the conditioned cortical reaction acquired due to combination of nonverbal stimulus with unrecognized emotional word. The suggestion is made that the necessary condition of temporary connection elaboration by means of a unrecognized verbal stimulus is the implication of the limbic structures where negative emotional reactions are integrated.     

Probability analysis of elements of rat behavior after sudden and signalled interruption of food reinforcement

Probability analysis was carried out of the appearance of single elements of rats behaviour in the process of extinction of a conditioned alimentary motor reflex. The dynamics of effector behavioural components at a sudden cessation of reinforcement (usual schedule of extinction) was compared with cessation of reinforcement signalled by a previously differentiated signal and with reinforcement cessation preceded by a stimulus initially unknown to the animal. If the reinforcement cessation is signalled by a previously differentiated (negative) stimulus, in response to its action the animals "loose the aim", what is revealed in a rapid complete reduction of all elements of the goal-directed alimentary behaviour. Obviously differentiation signal actualises the memory trace of "nonreinforcement" which was formed in the previous negative experience of the animal; this is revealed in accelerated inhibition of the alimentary motor reflex under extinction.     

Punishment prediction by dopaminergic neurons in Drosophila

The temporal pairing of a neutral stimulus with a reinforcer (reward or punishment) can lead to classical conditioning, a simple form of learning in which the animal assigns a value (positive or negative) to the formerly neutral stimulus. Olfactory classical conditioning in Drosophila is a prime model for the analysis of the molecular and neuronal substrate of this type of learning and memory. Neuronal correlates of associative plasticity have been identified in several regions of the insect brain. In particular, the mushroom bodies have been shown to be necessary for aversive olfactory memory formation. However, little is known about which neurons mediate the reinforcing stimulus. Using functional optical imaging, we now show that dopaminergic projections to the mushroom-body lobes are weakly activated by odor stimuli but respond strongly to electric shocks. However, after one of two odors is paired several times with an electric shock, odor-evoked activity is significantly prolonged only for the "punished" odor. Whereas dopaminergic neurons mediate rewarding reinforcement in mammals, our data suggest a role for aversive reinforcement in Drosophila. However, the dopaminergic neurons' capability of mediating and predicting a reinforcing stimulus appears to be conserved between Drosophila and mammals.     

Evaluating the addition of positive reinforcement for learning a frightening task: a pilot study with horses

Horse training often relies upon negative reinforcement (NR). This study tested the hypothesis that adding positive reinforcement (PR) to NR would enhance learning in horses (n = 34) being taught to walk over a tarp (novel/typically frightening task). Subjects were Arabians, and the same person handled all of them. This person handled half "traditionally" (NR only)--that is, halter/lead were pulled; when horse stepped forward, pressure was released; process repeated until criterion met (horse crossed the tarp with little/no obvious anxiety). The same person handled the other half traditionally--but with addition of PR < food + verbal praise > (NR + PR). Subjects "failed" the task if they refused to walk onto the tarp after 10 min. Nine horses failed; 6 of 9 failures were from NR only--no significant difference detected (p = .41). The study detected no difference in time to first crossing of the tarp (p = .30) or total time to achieve calmness criterion (p = .67). Overall, adding PR did not significantly enhance learning this task. However, there were practical implications--adding PR made the task safer/less fatiguing for the handler.     

Transient Decline in Hippocampal Theta Activity during the Acquisition Process of the Negative Patterning Task

Hippocampal function is important in the acquisition of negative patterning but not of simple discrimination. This study examined rat hippocampal theta activity during the acquisition stages (early, middle, and late) of the negative patterning task (A+, B+, AB-). The results showed that hippocampal theta activity began to decline transiently (for 500 ms after non-reinforced stimulus presentation) during the late stage of learning in the negative patterning task. In addition, this transient decline in hippocampal theta activity in the late stage was lower in the negative patterning task than in the simple discrimination task. This transient decline during the late stage of task acquisition may be related to a learning process distinctive of the negative patterning task but not the simple discrimination task. We propose that the transient decline of hippocampal theta activity reflects inhibitory learning and/or response inhibition after the presentation of a compound stimulus specific to the negative patterning task.     

Fear of Negative Evaluation Biases Social Evaluation Inference: Evidence from a Probabilistic Learning Task

Background

Fear of negative evaluation (FNE) defines social anxiety yet the process of inferring social evaluation, and its potential role in maintaining social anxiety, is poorly understood. We developed an instrumental learning task to model social evaluation learning, predicting that FNE would specifically bias learning about the self but not others.

Methods

During six test blocks (3 self-referential, 3 other-referential), participants (n = 100) met six personas and selected a word from a positive/negative pair to finish their social evaluation sentences “I think [you are / George is]…”. Feedback contingencies corresponded to 3 rules, liked, neutral and disliked, with P[positive word correct] = 0.8, 0.5 and 0.2, respectively.

Results

As FNE increased participants selected fewer positive words (β = −0.4, 95% CI −0.7, −0.2, p = 0.001), which was strongest in the self-referential condition (FNE × condition 0.28, 95% CI 0.01, 0.54, p = 0.04), and the neutral and dislike rules (FNE × condition × rule, p = 0.07). At low FNE the proportion of positive words selected for self-neutral and self-disliked greatly exceeded the feedback contingency, indicating poor learning, which improved as FNE increased.

Conclusions

FNE is associated with differences in processing social-evaluative information specifically about the self. At low FNE this manifests as insensitivity to learning negative self-referential evaluation. High FNE individuals are equally sensitive to learning positive or negative evaluation, which although objectively more accurate, may have detrimental effects on mental health.     

Rethinking motor learning and savings in adaptation paradigms: model-free memory for successful actions combines with internal models

Although motor learning is likely to involve multiple processes, phenomena observed in error-based motor learning paradigms tend to be conceptualized in terms of only a single process: adaptation, which occurs through updating an internal model. Here we argue that fundamental phenomena like movement direction biases, savings (faster relearning), and interference do not relate to adaptation but instead are attributable to two additional learning processes that can be characterized as model-free: use-dependent plasticity and operant reinforcement. Although usually "hidden" behind adaptation, we demonstrate, with modified visuomotor rotation paradigms, that these distinct model-based and model-free processes combine to learn an error-based motor task. (1) Adaptation of an internal model channels movements toward successful error reduction in visual space. (2) Repetition of the newly adapted movement induces directional biases toward the?repeated movement. (3) Operant reinforcement through association of the adapted movement with successful error reduction is responsible for savings.     

Clarifying the Role of the Rostral dmPFC/dACC in Fear/Anxiety: Learning,Appraisal or Expression?

Recent studies have begun to carve out a specific role for the rostral part of the dorsal medial prefrontal cortex (dmPFC) and adjacent dorsal anterior cingulate cortex (dACC) in fear/anxiety. Within a novel general framework of dorsal mPFC/ACC areas subserving the appraisal of threat and concomitant expression of fear responses and ventral mPFC/ACC areas subserving fear regulation, the rostral dmPFC/dACC has been proposed to specifically mediate the conscious, negative appraisal of threat situations including, as an extreme variant, catastrophizing. An alternative explanation that has not been conclusively ruled out yet is that the area is involved in fear learning. We tested two different fear expression paradigms in separate fMRI studies (study 1: instructed fear, study 2: testing of Pavlovian conditioned fear) with independent groups of healthy adult subjects. In both paradigms the absence of reinforcement precluded conditioning. We demonstrate significant BOLD activation of an identical rostral dmPFC/dACC area. In the Pavlovian paradigm (study 2), the area only activated robustly once prior conditioning had finished. Thus, our data argue against a role of the area in fear learning. We further replicate a repeated observation of a dissociation between peripheral-physiological fear responding and rostral dmPFC/dACC activation, strongly suggesting the area does not directly generate fear responses but rather contributes to appraisal processes. Although we succeeded in preventing extinction of conditioned responding in either paradigm, the data do not allow us to definitively exclude an involvement of the area in fear extinction learning. We discuss the broader implications of this finding for our understanding of mPFC/ACC function in fear and in negative emotion more generally.