Classical reward conditioning in Drosophila melanogaster

Negatively reinforced olfactory conditioning has been widely employed to identify learning and memory genes, signal transduction pathways and neural circuitry in Drosophila. To delineate the molecular and cellular processes underlying reward-mediated learning and memory, we developed a novel assay system for positively reinforced olfactory conditioning. In this assay, flies were involuntarily exposed to the appetitive unconditioned stimulus sucrose along with a conditioned stimulus odour during training and their preference for the odour previously associated with sucrose was measured to assess learning and memory capacities. After one training session, wild-type Canton S flies displayed reliable performance, which was enhanced after two training cycles with 1-min or 15-min inter-training intervals. Higher performance scores were also obtained with increasing sucrose concentration. Memory in Canton S flies decayed slowly when measured at 30 min, 1 h and 3 h after training; whereas, it had declined significantly at 6 h and 12 h post-training. When learning mutant t beta h flies, which are deficient in octopamine, were challenged, they exhibited poor performance, validating the utility of this assay. As the Drosophila model offers vast genetic and transgenic resources, the new appetitive conditioning described here provides a useful tool with which to elucidate the molecular and cellular underpinnings of reward learning and memory. Similar to negatively reinforced conditioning, this reward conditioning represents classical olfactory conditioning. Thus, comparative analyses of learning and memory mutants in two assays may help identify the molecular and cellular components that are specific to the unconditioned stimulus information used in conditioning.     

Selection for associative learning of color stimuli reveals correlated evolution of this learning ability across multiple stimuli and rewards

We are only starting to understand how variation in cognitive ability can result from local adaptations to environmental conditions. A major question in this regard is to what extent selection on cognitive ability in a specific context affects that ability in general through correlated evolution. To address this question, we performed artificial selection on visual associative learning in female Nasonia vitripennis wasps. Using appetitive conditioning in which a visual stimulus was offered in association with a host reward, the ability to learn visual associations was enhanced within 10 generations of selection. To test for correlated evolution affecting this form of learning, the ability to readily form learned associations in females was also tested using an olfactory instead of a visual stimulus in the appetitive conditioning. Additionally, we assessed whether the improved associative learning ability was expressed across sexes by color‐conditioning males with a mating reward. Both females and males from the selected lines consistently demonstrated an increased associative learning ability compared to the control lines, independent of learning context or conditioned stimulus. No difference in relative volume of brain neuropils was detected between the selected and control lines.     

Excitatory backward conditioning in an appetitive conditioned reinforcement preparation with rats

Four experiments were conducted to examine appetitive backward conditioning in a conditioned reinforcement preparation. In all experiments, off-line classical conditioning was conducted following lever-press training on two levers. Presentations of a sucrose solution by a liquid dipper served as an unconditioned stimulus (US) and two auditory stimuli served as conditioned stimuli (CSs); one was paired with the US in either a forward (Experiment 1a) or a backward (Experiments 1b, 2, and 3) relationship, and the other served as a control CS, which was not paired with the US. In testing, each lever-press response produced a presentation of one of the CSs instead of appetitive reinforcers. The response to a lever was facilitated, compared to the response to another lever, when the response produced the backward CS presentation as well as when it produced the forward CS presentation; that is, the backward CS served as an excitatory conditioned reinforcer.     

Nucleus accumbens neurons dynamically respond to appetitive and aversive associative learning

To survive, individuals must learn to associate cues in the environment with emotionally relevant outcomes. This association is partially mediated by the nucleus accumbens (NAc), a key brain region of the reward circuit that is mainly composed by GABAergic medium spiny neurons (MSNs), that express either dopamine receptor D1 or D2. Recent studies showed that both populations can drive reward and aversion, however, the activity of these neurons during appetitive and aversive Pavlovian conditioning remains to be determined. Here, we investigated the relevance of D1- and D2-neurons in associative learning, by measuring calcium transients with fiber photometry during appetitive and aversive Pavlovian tasks in mice. Sucrose was used as a positive valence unconditioned stimulus (US) and foot shock was used as a negative valence US. We show that during appetitive Pavlovian conditioning, D1- and D2-neurons exhibit a general increase in activity in response to the conditioned stimuli (CS). Interestingly, D1- and D2-neurons present distinct changes in activity after sucrose consumption that dynamically evolve throughout learning. During the aversive Pavlovian conditioning, D1- and D2-neurons present an increase in the activity in response to the CS and to the US (shock). Our data support a model in which D1- and D2-neurons are concurrently activated during appetitive and aversive conditioning.

     

The short-latency dopamine signal: a role in discovering novel actions?

An influential concept in contemporary computational neuroscience is the reward prediction error hypothesis of phasic dopaminergic function. It maintains that midbrain dopaminergic neurons signal the occurrence of unpredicted reward, which is used in appetitive learning to reinforce existing actions that most often lead to reward. However, the availability of limited afferent sensory processing and the precise timing of dopaminergic signals suggest that they might instead have a central role in identifying which aspects of context and behavioural output are crucial in causing unpredicted events.     

A Predictive Reinforcement Model of Dopamine Neurons for Learning Approach Behavior

A neural network model of how dopamine and prefrontal cortex activity guides short- and long-term information processing within the cortico-striatal circuits during reward-related learning of approach behavior is proposed. The model predicts two types of reward-related neuronal responses generated during learning: (1) cell activity signaling errors in the prediction of the expected time of reward delivery and (2) neural activations coding for errors in the prediction of the amount and type of reward or stimulus expectancies. The former type of signal is consistent with the responses of dopaminergic neurons, while the latter signal is consistent with reward expectancy responses reported in the prefrontal cortex. It is shown that a neural network architecture that satisfies the design principles of the adaptive resonance theory of Carpenter and Grossberg (1987) can account for the dopamine responses to novelty, generalization, and discrimination of appetitive and aversive stimuli. These hypotheses are scrutinized via simulations of the model in relation to the delivery of free food outside a task, the timed contingent delivery of appetitive and aversive stimuli, and an asymmetric, instructed delay response task.     

Stimulus-dependent adjustment of reward prediction error in the midbrain

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.     

Olfactory conditioning of proboscis activity in Drosophila melanogaster

Olfactory learning and memory processes in Drosophila have been well investigated with aversive conditioning, but appetitive conditioning has rarely been documented. Here, we report for the first time individual olfactory conditioning of proboscis activity in restrained Drosophila melanogaster. The protocol was adapted from those developed for proboscis extension conditioning in the honeybee Apis mellifera. After establishing a scale of small proboscis movements necessary to characterize responses to olfactory stimulation, we applied Pavlovian conditioning, with five trials consisting of paired presentation of a banana odour and a sucrose reward. Drosophila showed conditioned proboscis activity to the odour, with a twofold increase of percentage of responses after the first trial. No change occurred in flies experiencing unpaired presentations of the stimuli, confirming an associative basis for this form of olfactory learning. The adenylyl cyclase mutant rutabaga did not exhibit learning in this paradigm. This protocol generated at least a short-term memory of 15 min, but no significant associative memory was detected at 1 h. We also showed that learning performance was dependent on food motivation, by comparing flies subjected to different starvation regimes.     

Ketamine Effects on Memory Reconsolidation Favor a Learning Model of Delusions

Delusions are the persistent and often bizarre beliefs that characterise psychosis. Previous studies have suggested that their emergence may be explained by disturbances in prediction error-dependent learning. Here we set up complementary studies in order to examine whether such a disturbance also modulates memory reconsolidation and hence explains their remarkable persistence. First, we quantified individual brain responses to prediction error in a causal learning task in 18 human subjects (8 female). Next, a placebo-controlled within-subjects study of the impact of ketamine was set up on the same individuals. We determined the influence of this NMDA receptor antagonist (previously shown to induce aberrant prediction error signal and lead to transient alterations in perception and belief) on the evolution of a fear memory over a 72 hour period: they initially underwent Pavlovian fear conditioning; 24 hours later, during ketamine or placebo administration, the conditioned stimulus (CS) was presented once, without reinforcement; memory strength was then tested again 24 hours later. Re-presentation of the CS under ketamine led to a stronger subsequent memory than under placebo. Moreover, the degree of strengthening correlated with individual vulnerability to ketamine''s psychotogenic effects and with prediction error brain signal. This finding was partially replicated in an independent sample with an appetitive learning procedure (in 8 human subjects, 4 female). These results suggest a link between altered prediction error, memory strength and psychosis. They point to a core disruption that may explain not only the emergence of delusional beliefs but also their persistence.     

Repeated extinction and reversal learning of an approach response supports an arousal-mediated learning model

We assessed the effects of repeated extinction and reversals of two conditional stimuli (CS+/CS−) on an appetitive conditioned approach response in rats. Three results were observed that could not be accounted for by a simple linear operator model such as the one proposed by Rescorla and Wagner (1972): (1) responding to a CS− declined faster when a CS+ was simultaneously extinguished; (2) reacquisition of pre-extinction performance recovered rapidly within one session; and (3) reversal of CS+/CS− contingencies resulted in a more rapid recovery to the current CS− (former CS+) than the current CS+, accompanied by a slower acquisition of performance to the current CS+. An arousal parameter that mediates learning was introduced to a linear operator model to account for these effects. The arousal-mediated learning model adequately fit the data and predicted data from a second experiment with different rats in which only repeated reversals of CS+/CS− were assessed. According to this arousal-mediated learning model, learning is accelerated by US-elicited arousal and it slows down in the absence of US. Because arousal varies faster than conditioning, the model accounts for the decline in responding during extinction mainly through a reduction in arousal, not a change in learning. By preserving learning during extinction, the model is able to account for relapse effects like rapid reacquisition, renewal, and reinstatement.     

Light-induced activation of distinct modulatory neurons triggers appetitive or aversive learning in Drosophila larvae

During classical conditioning, a positive or negative value is assigned to a previously neutral stimulus, thereby changing its significance for behavior. If an odor is associated with a negative stimulus, it can become repulsive. Conversely, an odor associated with a reward can become attractive. By using Drosophila larvae as a model system with minimal brain complexity, we address the question of which neurons attribute these values to odor stimuli. In insects, dopaminergic neurons are required for aversive learning, whereas octopaminergic neurons are necessary and sufficient for appetitive learning. However, it remains unclear whether two independent neuronal populations are sufficient to mediate such antagonistic values. We report the use of transgenically expressed channelrhodopsin-2, a light-activated cation channel, as a tool for optophysiological stimulation of genetically defined neuronal populations in Drosophila larvae. We demonstrate that distinct neuronal populations can be activated simply by illuminating the animals with blue light. Light-induced activation of dopaminergic neurons paired with an odor stimulus induces aversive memory formation, whereas activation of octopaminergic/tyraminergic neurons induces appetitive memory formation. These findings demonstrate that antagonistic modulatory subsystems are sufficient to substitute for aversive and appetitive reinforcement during classical conditioning.     

Effects of L-dopa during Auditory Instrumental Learning in Humans

The dopaminergic neurotransmitter system is critically involved in promoting plasticity in auditory cortex. We combined functional magnetic resonance imaging (fMRI) and a pharmacological manipulation to investigate dopaminergic modulation of neural activity in auditory cortex during instrumental learning. Volunteers either received 100 mg L-dopa (Madopar) or placebo in an appetitive, differential instrumental conditioning paradigm, which involved learning that a specific category of frequency modulated tones predicts a monetary reward when fast responses were made in a subsequent reaction time task. The other category of frequency modulated tones was not related to a reward. Our behavioral data provides evidence that dopaminergic stimulation differentially impacts on the speed of instrumental responding in rewarded and unrewarded trials. L-dopa increased neural BOLD activity in left auditory cortex to tones in rewarded and unrewarded trials. This increase was related to plasma L-dopa levels and learning rate. Our data thus provides evidence for dopaminergic modulation of neural activity in auditory cortex, which occurs for both auditory stimuli related to a later reward and those not related to a reward.     

桔小实蝇雄成虫联系性嗅觉学习

【目的】为了探究桔小实蝇 Bactrocera dorsalis (Hendel)雄成虫的嗅觉学习能力。【方法】本研究采用经典性嗅觉条件反射训练法（classical olfactory conditioning）在室内对固定的羽化后14-17日龄的桔小实蝇雄成虫进行气味与食物的联合学习训练, 即薄荷精油和10%蔗糖溶液联合的奖赏性训练（appetitive conditioning）以及甲基丁香酚(methyl eugenol, ME)和饱和盐溶液联合的惩罚性训练（aversive conditioning）,并以伸喙反射行为（proboscis extension reflex, PER）作为学习与否的判定标准。【结果】经过奖赏性训练后,桔小实蝇雄成虫对薄荷精油的伸喙反射率可从0%增加至68%;而经过惩罚性训练后,桔小实蝇对甲基丁香酚的伸喙反射率可从100%降低至36.54%,且这种伸喙反射率的变化是通过气味条件刺激（conditioned stimulus）和食物非条件刺激（unconditioned stimulus）的对称性联合而产生的。【结论】结果表明,桔小实蝇雄性成虫具有较强的联系性嗅觉学习能力,并且两种刺激的联合是形成学习记忆的必要条件。     

Relief as a reward: hedonic and neural responses to safety from pain

Relief fits the definition of a reward. Unlike other reward types the pleasantness of relief depends on the violation of a negative expectation, yet this has not been investigated using neuroimaging approaches. We hypothesized that the degree of negative expectation depends on state (dread) and trait (pessimism) sensitivity. Of the brain regions that are involved in mediating pleasure, the nucleus accumbens also signals unexpected reward and positive prediction error. We hypothesized that accumbens activity reflects the level of negative expectation and subsequent pleasant relief. Using fMRI and two purpose-made tasks, we compared hedonic and BOLD responses to relief with responses during an appetitive reward task in 18 healthy volunteers. We expected some similarities in task responses, reflecting common neural substrates implicated across reward types. However, we also hypothesized that relief responses would differ from appetitive rewards in the nucleus accumbens, since only relief pleasantness depends on negative expectations. The results confirmed these hypotheses. Relief and appetitive reward task activity converged in the ventromedial prefrontal cortex, which also correlated with appetitive reward pleasantness ratings. In contrast, dread and pessimism scores correlated with relief but not with appetitive reward hedonics. Moreover, only relief pleasantness covaried with accumbens activation. Importantly, the accumbens signal appeared to specifically reflect individual differences in anticipation of the adverse event (dread, pessimism) but was uncorrelated to appetitive reward hedonics. In conclusion, relief differs from appetitive rewards due to its reliance on negative expectations, the violation of which is reflected in relief-related accumbens activation.     

Self-Initiated Behavioral Act-Related Neuronal Activity in the Region of the Raphe Nuclei of the Cat

In experiments on awake cats, we recorded the activity of 79 putative serotonergic (STE) neurons localized within the region of the brainstem dorsal and anterior central raphe nuclei. The animals were trained to perform a self-initiated (voluntary) movement, to press a pedal by the forelimb; an additional limitation was to perform the movement not earlier than after a definite time interval. Changes in the activity of STE neurons related to the preparation for and performance of the movement and reactions to presentation of a feedback conditioning signal preceding the reward and receipt of the food reward were most clearly manifested. More than 50% of the units changed their activity before the movement initiation. Most neurons responded to presentation of a positive conditioning signal by phasic activation, while a negative signal informing them of the absence of the reward evoked considerably weaker reactions. We hypothesize that reactions of STE neurons forestalling the movement initiation can provide activation of the neocortex necessary for the movement performance within a preset time interval. Activating and inhibitory reactions observed within the period of expectation of a feedback conditioning signal and developing after presentation of this signal can be related to a noticeable role of the STE system in the formation of memory engrams and development of emotional states.     

Genotypic Influence on Aversive Conditioning in Honeybees,Using a Novel Thermal Reinforcement Procedure

In Pavlovian conditioning, animals learn to associate initially neutral stimuli with positive or negative outcomes, leading to appetitive and aversive learning respectively. The honeybee (Apis mellifera) is a prominent invertebrate model for studying both versions of olfactory learning and for unraveling the influence of genotype. As a queen bee mates with about 15 males, her worker offspring belong to as many, genetically-different patrilines. While the genetic dependency of appetitive learning is well established in bees, it is not the case for aversive learning, as a robust protocol was only developed recently. In the original conditioning of the sting extension response (SER), bees learn to associate an odor (conditioned stimulus - CS) with an electric shock (unconditioned stimulus - US). This US is however not a natural stimulus for bees, which may represent a potential caveat for dissecting the genetics underlying aversive learning. We thus first tested heat as a potential new US for SER conditioning. We show that thermal stimulation of several sensory structures on the bee’s body triggers the SER, in a temperature-dependent manner. Moreover, heat applied to the antennae, mouthparts or legs is an efficient US for SER conditioning. Then, using microsatellite analysis, we analyzed heat sensitivity and aversive learning performances in ten worker patrilines issued from a naturally inseminated queen. We demonstrate a strong influence of genotype on aversive learning, possibly indicating the existence of a genetic determinism of this capacity. Such determinism could be instrumental for efficient task partitioning within the hive.     

Can Honey Bees Learn the Removal of a Stimulus as a Conditioning Cue?

Experiments investigated a Pavlovian conditioning situation where the presence and absence of the stimulus are reversed temporally with respect to the presentation of a reward. Instead of a conditioned stimulus (e.g. odor) signaling the presence of a reward, the stimulus (e.g. odor) is present in the environment except just prior to the presence of the reward. Thus, the absence of the stimulus, or offset of the stimulus (e.g. absence of odor), serves as a conditioned stimulus and is the reward cue. Honey bees (Apis mellifera) were used as a model invertebrate system, and the proboscis‐conditioning paradigm was used as the test procedure. Using both simple Pavlovian conditioning and discrimination‐learning protocols, animals learned to associate the onset of an odor as conditioned stimuli when paired with a sucrose reward. They could also learn to associate the onset of a puff of air with a sucrose reward. However, bees could not associate the offset of an order stimulus with the presentation of a sucrose reward in either a simple conditioning or a discrimination‐learning situation. These results support the model that a very different cognitive architecture is used by invertebrates to deal with certain environmental situations, including signaled avoidance.     

Appetitive and aversive learning in Spodoptera littoralis larvae

Adult Lepidoptera are capable of associative learning. This helps them to forage flowers or to find suitable oviposition sites. Larval learning has never been seriously considered because they have limited foraging capabilities and usually depend on adults as concerns their food choices. We tested if Spodoptera littoralis larvae can learn to associate an odor with a tastant using a new classical conditioning paradigm. Groups of larvae were exposed to an unconditioned stimulus (US: fructose or quinine mixed with agar) paired with a conditioned stimulus (CS: hexanol, geraniol or pentyl acetate) in a petri dish. Their reaction to CS was subsequently tested in a petri dish at different time intervals after conditioning. Trained larvae showed a significant preference or avoidance to CS when paired with US depending on the reinforcer used. The training was more efficient when larvae were given a choice between an area where CS-US was paired and an area with no CS (or another odor). In these conditions, the memory formed could be recalled at least 24 h after pairing with an aversive stimulus and only 5 min after pairing with an appetitive stimulus. This learning was specific to CS because trained larvae were able to discriminate CS from another odor that was present during the training but unrewarded. These results suggest that Lepidoptera larvae exhibit more behavioral plasticity than previously appreciated.     

Reward,motivation, and reinforcement learning

There is substantial evidence that dopamine is involved in reward learning and appetitive conditioning. However, the major reinforcement learning-based theoretical models of classical conditioning (crudely, prediction learning) are actually based on rules designed to explain instrumental conditioning (action learning). Extensive anatomical, pharmacological, and psychological data, particularly concerning the impact of motivational manipulations, show that these models are unreasonable. We review the data and consider the involvement of a rich collection of different neural systems in various aspects of these forms of conditioning. Dopamine plays a pivotal, but complicated, role.