The role of the striatum in aversive learning and aversive prediction errors

Neuroeconomic studies of decision making have emphasized reward learning as critical in the representation of value-driven choice behaviour. However, it is readily apparent that punishment and aversive learning are also significant factors in motivating decisions and actions. In this paper, we review the role of the striatum and amygdala in affective learning and the coding of aversive prediction errors (PEs). We present neuroimaging results showing aversive PE-related signals in the striatum in fear conditioning paradigms with both primary (shock) and secondary (monetary loss) reinforcers. These results and others point to the general role for the striatum in coding PEs across a broad range of learning paradigms and reinforcer types.     

Strain-specific alteration of zebrafish feeding behavior in response to aversive stimuli

Behavioral management of risk, in which organisms must balance the requirements of obtaining food resources with the risk of predation, has been of considerable interest to ethologists for many years. Although numerous experiments have shown that animals alter their foraging behavior depending on the levels of perceived risk and demand for nutrients, few have considered the role of genetic variation in the trade-off between these variables. We performed a study of four zebrafish (Danio rerio (Hamilton, 1822)) strains to test for genetic variation in foraging behavior and whether this variation affected their response to both aversive stimuli and nutrient restriction. Zebrafish strains differed significantly in their latency to begin foraging from the surface of the water under standard laboratory conditions. Fish fed sooner when nutrients were restricted, although this was only significant in the absence of aversive stimuli. Aversive stimuli caused fish to delay feeding in a strain-specific manner. Strains varied in food intake and specific growth rate, and feeding latency was significantly correlated with food intake. Our results indicate significant genetic variation in foraging behavior and the perception of risk in zebrafish, with a pattern of strain variation consistent with behavioral adaptation to captivity.     

Neural encoding in ventral striatum during olfactory discrimination learning

A growing body of evidence implicates the ventral striatum in using information acquired through associative learning. The present study examined the activity of ventral striatal neurons in awake, behaving rats during go/no-go odor discrimination learning and reversal. Many neurons fired selectively to odor cues predictive of either appetitive (sucrose) or aversive (quinine) outcomes. Few neurons were selective when first exposed to the odors, but many acquired this differential activity as rats learned the significance of the cues. A substantial proportion of these neurons encoded the cues' learned motivational significance, and these neurons tended to reverse their firing selectivity after reversal of odor-outcome contingencies. Other neurons that became selectively activated during learning did not reverse, but instead appeared to encode specific combinations of cues and associated motor responses. The results support a role for ventral striatum in using the learned significance, both appetitive and aversive, of predictive cues to guide behavior.     

Nomifensine amplifies subsecond dopamine signals in the ventral striatum of freely-moving rats

The present experiment examined the effect of the dopamine transporter blocker nomifensine on subsecond fluctuations in dopamine concentrations, or dopamine transients, in the nucleus accumbens and olfactory tubercle. Extracellular dopamine was measured in real time using fast-scan cyclic voltammetry at micron-dimension carbon fibers in freely-moving rats. Dopamine transients occurred spontaneously throughout the ventral striatum in the absence of apparent sensory input or change in behavioral response. The frequency of dopamine transients increased at the presentation of salient stimuli to the rat (food, novel odors and unexpected noises). Administration of 7 mg/kg nomifensine amplified spontaneous dopamine transients by increasing both amplitude and duration, consistent with its known action at the dopamine transporter and emphasizing the dopaminergic origin of the signals. Moreover, nomifensine increased the frequency of detected dopamine transients, both during baseline conditions and at the presentation of stimuli, but more profoundly in the nucleus accumbens than in the olfactory tubercle. This difference was not explained by nomifensine effects on the kinetics of dopamine release and uptake, as its effects on electrically-evoked dopamine signals were similar in both regions. These findings demonstrate the heterogeneity of dopamine transients in the ventral striatum and establish that nomifensine elevates the tone of rapid dopamine signals in the brain.     

Sex differences in regional brain response to aversive pelvic visceral stimuli

To explore sex differences in the response of seven brain regions to an aversive pelvic visceral stimulus, functional magnetic resonance images were acquired from 13 healthy adults (6 women) during 15 s of cued rectal distension at two pressures: 25 mmHg (uncomfortable), and 45 mmHg (mild pain), as well as during an expectation condition (no distension). Random-effects analyses combining subject data voxelwise found 45-mmHg pressure significantly activated the insular and anterior cingulate cortices in both sexes. In men only, the left thalamus and ventral striatum were also activated. Although all activations appeared more extensive in men, no sex difference attained significance. To explore the presence of deactivations, which are generally cancelled by more numerous activations when subjects are combined for each voxel, the number of activated voxels, number of deactivated voxels, and ratio of deactivated voxels to total voxels affected were assessed via random-effects, mixed-model analyses combining subject data at the region level. Greater insula activation in men compared with women was seen during the expectation condition and during the 25-mmHg distension. Greater deactivations in women were seen in the amygdala (25-mmHg distension) and midcingulate (45-mmHg distension). Women had a significantly higher proportion of deactivated voxels than men in all four subcortical structures during 25-mmHg distension. Greater familiarity of females with physiological pelvic visceral discomfort may have enhanced brain systems that dampen arousal networks during lower levels of discomfort.     

Glutamate release in the nucleus accumbens during competitive presentation of aversive and appetitive stimuli

By means of in vivo microdialysis combined with HPLC analysis, we have shown that extracellular glutamate level in the rat n. accumbens increases during a simultaneous presentation of a palatable diet and a tone previously paired with a footshock, the magnitude of the extracellular glutamate increase being proportional to the latency of food taking. In contrast, extracellular glutamate level remains unchanged when the diet is presented after the conditioned aversive stimulus or when the tone is given alone. These data suggest that the glutamate release evoked by the competitive presentation of the diet and the conditioned aversive stimulus appears to be related to the inhibition of a planned feeding response, whereas the choice between behavioural strategies may not contribute to this phenomenon.     

Predictive neural coding of reward preference involves dissociable responses in human ventral midbrain and ventral striatum

Food preferences are acquired through experience and can exert strong influence on choice behavior. In order to choose which food to consume, it is necessary to maintain a predictive representation of the subjective value of the associated food stimulus. Here, we explore the neural mechanisms by which such predictive representations are learned through classical conditioning. Human subjects were scanned using fMRI while learning associations between arbitrary visual stimuli and subsequent delivery of one of five different food flavors. Using a temporal difference algorithm to model learning, we found predictive responses in the ventral midbrain and a part of ventral striatum (ventral putamen) that were related directly to subjects' actual behavioral preferences. These brain structures demonstrated divergent response profiles, with the ventral midbrain showing a linear response profile with preference, and the ventral striatum a bivalent response. These results provide insight into the neural mechanisms underlying human preference behavior.     

Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli

Midbrain dopamine (DA) neurons are not homogeneous but differ in their molecular properties and responses to external stimuli. We examined whether the modulation of excitatory synapses on DA neurons by rewarding or aversive stimuli depends on the brain area to which these DA neurons project. We identified DA neuron subpopulations in slices after injection of "Retrobeads" into single target areas of adult mice and found differences in basal synaptic properties. Administration of cocaine selectively modified excitatory synapses on DA cells projecting to nucleus accumbens (NAc) medial shell while an aversive stimulus selectively modified synapses on DA cells projecting to medial prefrontal cortex. In contrast, synapses on DA neurons projecting to NAc lateral shell were modified by both rewarding and aversive stimuli, which presumably reflects saliency. These results suggest that the mesocorticolimbic DA system may be comprised of three anatomically distinct circuits, each modified by distinct aspects of motivationally relevant stimuli.     

Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala

Animals and humans learn to approach and acquire pleasant stimuli and to avoid or defend against aversive ones. However, both pleasant and aversive stimuli can elicit arousal and attention, and their salience or intensity increases when they occur by surprise. Thus, adaptive behavior may require that neural circuits compute both stimulus valence--or value--and intensity. To explore how these computations may be implemented, we examined neural responses in the primate amygdala to unexpected reinforcement during learning. Many amygdala neurons responded differently to reinforcement depending upon whether or not it was expected. In some neurons, this modulation occurred only for rewards or aversive stimuli, but not both. In other neurons, expectation similarly modulated responses to both rewards and punishments. These different neuronal populations may subserve two sorts of processes mediated by the amygdala: those activated by surprising reinforcements of both valences-such as enhanced arousal and attention-and those that are valence-specific, such as fear or reward-seeking behavior.     

Coordinated activity of ventral tegmental neurons adapts to appetitive and aversive learning

Our understanding of how value-related information is encoded in the ventral tegmental area (VTA) is based mainly on the responses of individual putative dopamine neurons. In contrast to cortical areas, the nature of coordinated interactions between groups of VTA neurons during motivated behavior is largely unknown. These interactions can strongly affect information processing, highlighting the importance of investigating network level activity. We recorded the activity of multiple single units and local field potentials (LFP) in the VTA during a task in which rats learned to associate novel stimuli with different outcomes. We found that coordinated activity of VTA units with either putative dopamine or GABA waveforms was influenced differently by rewarding versus aversive outcomes. Specifically, after learning, stimuli paired with a rewarding outcome increased the correlation in activity levels between unit pairs whereas stimuli paired with an aversive outcome decreased the correlation. Paired single unit responses also became more redundant after learning. These response patterns flexibly tracked the reversal of contingencies, suggesting that learning is associated with changing correlations and enhanced functional connectivity between VTA neurons. Analysis of LFP recorded simultaneously with unit activity showed an increase in the power of theta oscillations when stimuli predicted reward but not an aversive outcome. With learning, a higher proportion of putative GABA units were phase locked to the theta oscillations than putative dopamine units. These patterns also adapted when task contingencies were changed. Taken together, these data demonstrate that VTA neurons organize flexibly as functional networks to support appetitive and aversive learning.     

Ultrasonic vocalization in response to unavoidable aversive stimuli in rats: effects of benzodiazepines

The effects of two benzodiazepine derivatives (diazepam, 0.5-1 mg/kg; alprazolam, 1.25-2.5 mg/kg) on ultrasonic calling elicited in adult rats by unavoidable aversive stimuli (footshocks) were investigated. The results show that either diazepam or alprazolam affected the duration of ultrasonic calls. In particular, a significant decrease in the length of ultrasounds was found in the group of animals treated with these benzodiazepines. The effects of diazepam were counteracted by the benzodiazepine-antagonist Ro 15-1788. On the other hand, neither a neuroleptic agent, such as haloperidol (0.5-1 mg/kg), nor an antidepressant, such as desipramine (5-10 mg/kg) influenced the parameters of ultrasonic emission in this experimental situation. The present results suggest that ultrasonic vocalization in response to unavoidable aversive stimuli could be considered as a potential new tool for studying drugs with antianxiety properties.     

Dissociable reward and timing signals in human midbrain and ventral striatum

Reward prediction error (RPE) signals are central to current models of reward-learning. Temporal difference (TD) learning models posit that these signals should be modulated by predictions, not only of magnitude but also timing of reward. Here we show that BOLD activity in the VTA conforms to such TD predictions: responses to unexpected rewards are modulated by a temporal hazard function and activity between a predictive stimulus and reward is depressed in proportion to predicted reward. By contrast, BOLD activity in ventral striatum (VS) does not reflect a TD RPE, but instead encodes a signal on the variable relevant for behavior, here timing but not magnitude of reward. The results have important implications for dopaminergic models of cortico-striatal learning and suggest a modification of the conventional view that VS BOLD necessarily reflects inputs from dopaminergic VTA neurons signaling an RPE.     

The mediator mechanisms of the ventral hippocampus in anxiety states formed by different aversive exposures]

Microinjections of monoamines and amino acids into rat ventral hippocampus showed functional relevance of GABA- or dopamine- and 5-HT-ergic mechanisms of this structure of brain formed by aversive actions of diverse biological importance rather than glutamate-ergic ones. The participation of hippocampus monoamine- and acidergic mechanisms in the origin of diverse aversive anxiety is discussed.     

Effects of aversive stimuli on prospective memory. An event-related fMRI study

Prospective memory (PM) describes the ability to execute a previously planned action at the appropriate point in time. Although behavioral studies clearly showed that prospective memory performance is affected by the emotional significance attributed to the intended action, no study so far investigated the brain mechanisms subserving the modulatory effect of emotional salience on PM performance. The general aim of the present study was to explore brain regions involved in prospective memory processes when PM cues are associated with emotional stimuli. In particular, based on the hypothesised critical role of the prefrontal cortex in prospective memory in the presence of emotionally salient stimuli, we expected a stronger involvement of aPFC when the retrieval and execution of the intended action is cued by an aversive stimulus. To this aim BOLD responses of PM trials cued by aversive facial expressions were compared to PM trials cued by neutral facial expressions. Whole brain analysis showed that PM task cued by aversive stimuli is differentially associated with activity in the right lateral prefrontal area (BA 10) and in the left caudate nucleus. Moreover a temporal shift between the response of the caudate nucleus that preceded that of aPFC was observed. These findings suggest that the caudate nucleus might provide an early analysis of the affective properties of the stimuli, whereas the anterior lateral prefrontal cortex (BA10) would be involved in a slower and more deliberative analysis to guide goal-directed behaviour.     

Amphetamine augments vesicular dopamine release in the dorsal and ventral striatum through different mechanisms

Amphetamine has well‐established actions on pre‐synaptic dopamine signaling, such as inhibiting uptake and degradation, activating synthesis, depleting vesicular stores, and promoting dopamine‐transporter reversal and non‐exocytotic release. Recent in vivo studies have identified an additional mechanism: augmenting vesicular release. In this study, we investigated how amphetamine elicits this effect. Our hypothesis was that amphetamine enhances vesicular dopamine release in dorsal and ventral striata by differentially targeting dopamine synthesis and degradation. In urethane‐anesthetized rats, we employed voltammetry to monitor dopamine, electrical stimulation to deplete stores or assess vesicular release and uptake, and pharmacology to isolate degradation and synthesis. While amphetamine increased electrically evoked dopamine levels, inhibited uptake, and up‐regulated vesicular release in both striatal sub‐regions in controls, this psychostimulant elicited region‐specific effects on evoked levels and vesicular release but not uptake in drug treatments. Evoked levels better correlated with vesicular release compared with uptake, supporting enhanced vesicular release as an important amphetamine mechanism. Taken together, these results suggested that amphetamine enhances vesicular release in the dorsal striatum by activating dopamine synthesis and inhibiting dopamine degradation, but targeting an alternative mechanism in the ventral striatum. Region‐distinct activation of vesicular dopamine release highlights complex cellular actions of amphetamine and may have implications for its behavioral effects.