The subjective value of delayed and probabilistic outcomes: Outcome size matters for gains but not for losses

The subjective value of a reward (gain) is related to factors such as its size, the delay to its receipt and the probability of its receipt. We examined whether the subjective value of losses was similarly affected by these factors in 128 adults. Participants chose between immediate/certain gains or losses and larger delayed/probabilistic gains or losses. Rewards of $100 were devalued as a function of their delay (“discounted”) relatively less than $10 gains while probabilistic $100 rewards were discounted relatively more than $10 rewards. However, there was no effect of outcome size on discounting of delayed or probabilistic losses. For delayed outcomes of each size, the degree to which gains were discounted was positively correlated with the degree to which losses were discounted, whereas for probabilistic outcomes, no such correlation was observed. These results suggest that the processes underlying the subjective valuation of losses are different from those underlying the subjective valuation of gains.     

Saccade reward signals in posterior cingulate cortex

Movement selection depends on the outcome of prior behavior. Posterior cingulate cortex (CGp) is strongly connected with both limbic and oculomotor circuitry, and CGp neurons respond following saccades, suggesting a role in signaling the motivational outcome of gaze shifts. To test this hypothesis, single CGp neurons were studied in monkeys while they shifted gaze to visual targets for liquid rewards that varied in size or were delivered probabilistically. CGp neurons responded following saccades as well as following reward delivery, and these responses were correlated with reward size. CGp neurons also responded following the omission of predicted rewards. The timing of CGp activation and its modulation by reward could provide signals useful for updating representations of expected saccade value.     

Interdependent utilities: how social ranking affects choice behavior

Organization in hierarchical dominance structures is prevalent in animal societies, so a strong preference for higher positions in social ranking is likely to be an important motivation of human social and economic behavior. This preference is also likely to influence the way in which we evaluate our outcome and the outcome of others, and finally the way we choose. In our experiment participants choose among lotteries with different levels of risk, and can observe the choice that others have made. Results show that the relative weight of gains and losses is the opposite in the private and social domain. For private outcomes, experience and anticipation of losses loom larger than gains, whereas in the social domain, gains loom larger than losses, as indexed by subjective emotional evaluations and physiological responses. We propose a theoretical model (interdependent utilities), predicting the implication of this effect for choice behavior. The relatively larger weight assigned to social gains strongly affects choices, inducing complementary behavior: faced with a weaker competitor, participants adopt a more risky and dominant behavior.     

Decision-Making for Risky Gains and Losses among College Students with Internet Gaming Disorder

Individuals with Internet gaming disorder (IGD) tend to exhibit disadvantageous risky decision-making not only in their real life but also in laboratory tasks. Decision-making is a complex multifaceted function and different cognitive processes are involved in decision-making for gains and losses. However, the relationship between impaired decision-making and gain versus loss processing in the context of IGD is poorly understood. The main aim of the present study was to separately evaluate decision-making for risky gains and losses among college students with IGD using the Cups task. Additionally, we further examined the effects of outcome magnitude and probability level on decision-making related to risky gains and losses respectively. Sixty college students with IGD and 42 matched healthy controls (HCs) participated. Results indicated that IGD subjects exhibited generally greater risk taking tendencies than HCs. In comparison to HCs, IGD subjects made more disadvantageous risky choices in the loss domain (but not in the gain domain). Follow-up analyses indicated that the impairment was associated to insensitivity to changes in outcome magnitude and probability level for risky losses among IGD subjects. In addition, higher Internet addiction severity scores were associated with percentage of disadvantageous risky options in the loss domain. These findings emphasize the effect of insensitivity to losses on disadvantageous decisions under risk in the context of IGD, which has implications for future intervention studies.     

Reward magnitude and timing in pigeons

We investigated the interaction of motivation and timing by manipulating the expected reward magnitude during a peak procedure. Four pigeons were tested with three different reward magnitudes, operationalized as duration of food access. Each stimulus predicted a different reward magnitude on a 5 s fixed-interval schedule. Trials with different reward magnitudes were randomly intermingled in a session. Most pigeons responded less often and started responding later on peak trials when a smaller reward was expected, but showed no differences in response termination or peak times. Reward magnitude was independently corroborated through unreinforced choice trials, when pigeons chose between the three stimuli presented simultaneously. These results contribute to a growing body of evidence that the expected reward magnitude influences the decision to start anticipatory responding in tasks where the reward becomes available after a fixed interval, but does not alter peak times, nor the decision to stop responding on peak trials.     

Sensitivity and bias in decision-making under risk: evaluating the perception of reward, its probability and value

Background

There are few clinical tools that assess decision-making under risk. Tests that characterize sensitivity and bias in decisions between prospects varying in magnitude and probability of gain may provide insights in conditions with anomalous reward-related behaviour.

Objective

We designed a simple test of how subjects integrate information about the magnitude and the probability of reward, which can determine discriminative thresholds and choice bias in decisions under risk.

Design/Methods

Twenty subjects were required to choose between two explicitly described prospects, one with higher probability but lower magnitude of reward than the other, with the difference in expected value between the two prospects varying from 3 to 23%.

Results

Subjects showed a mean threshold sensitivity of 43% difference in expected value. Regarding choice bias, there was a ‘risk premium’ of 38%, indicating a tendency to choose higher probability over higher reward. An analysis using prospect theory showed that this risk premium is the predicted outcome of hypothesized non-linearities in the subjective perception of reward value and probability.

Conclusions

This simple test provides a robust measure of discriminative value thresholds and biases in decisions under risk. Prospect theory can also make predictions about decisions when subjective perception of reward or probability is anomalous, as may occur in populations with dopaminergic or striatal dysfunction, such as Parkinson''s disease and schizophrenia.     

Early Effects of Reward Anticipation Are Modulated by Dopaminergic Stimulation

The abilities to predict future rewards and assess the value of reward delivery are crucial aspects of adaptive behavior. While the mesolimbic system, including dopaminergic midbrain, ventral striatum and prefrontal cortex have long been associated with reward processing, recent studies also indicate a prominent role of early visual brain regions. However, the precise underlying neural mechanisms still remain unclear. To address this issue, we presented participants with visual cues predicting rewards of high and low magnitudes and probability (2×2 factorial design), while neural activity was scanned using magnetoencephalography. Importantly, one group of participants received 150 mg of the dopamine precursor levodopa prior to the experiment, while another group received a placebo. For the placebo group, neural signals of reward probability (but not magnitude) emerged at ∼100 ms after cue presentation at occipital sensors in the event-related magnetic fields. Importantly, these probability signals were absent in the levodopa group indicating a close link. Moreover, levodopa administration reduced oscillatory power in the high (20–30 Hz) and low (13–20 Hz) beta band during both reward anticipation and delivery. Taken together, our findings indicate that visual brain regions are involved in coding prospective reward probability but not magnitude and that these effects are modulated by dopamine.     

Behavioural correlate of choice confidence in a discrete trial paradigm

How animals make choices in a changing and often uncertain environment is a central theme in the behavioural sciences. There is a substantial literature on how animals make choices in various experimental paradigms but less is known about the way they assess a choice after it has been made in terms of the expected outcome. Here, we used a discrete trial paradigm to characterise how the reward history shaped the behaviour on a trial by trial basis. Rats initiated each trial which consisted of a choice between two drinking spouts that differed in their probability of delivering a sucrose solution. Critically, sucrose was delivered after a delay from the first lick at the spouts--this allowed us to characterise the behavioural profile during the window between the time of choice and its outcome. Rats' behaviour converged to optimum choice, both during the acquisition phase and after the reversal of contingencies. We monitored the post-choice behaviour at a temporal precision of 1 millisecond; lick-response profiles revealed that rats spent more time at the spout with the higher reward probability and exhibited a sparser lick pattern. This was the case when we exclusively examined the unrewarded trials, where the outcome was identical. The differential licking profiles preceded the differential choice ratios and could thus predict the changes in choice behaviour.     

Motivational state and reward content determine choice behavior under risk in mice

Risk is a ubiquitous feature of the environment for most organisms, who must often choose between a small and certain reward and a larger but less certain reward. To study choice behavior under risk in a genetically well characterized species, we trained mice (C57BL/6) on a discrete trial, concurrent-choice task in which they must choose between two levers. Pressing one lever (safe choice) is always followed by a small reward. Pressing the other lever (risky choice) is followed by a larger reward, but only on some of the trials. The overall payoff is the same on both levers. When mice were not food deprived, they were indifferent to risk, choosing both levers with equal probability regardless of the level of risk. In contrast, following food or water deprivation, mice earning 10% sucrose solution were risk-averse, though the addition of alcohol to the sucrose solution dose-dependently reduced risk aversion, even before the mice became intoxicated. Our results falsify the budget rule in optimal foraging theory often used to explain behavior under risk. Instead, they suggest that the overall demand or desired amount for a particular reward determines risk preference. Changes in motivational state or reward identity affect risk preference by changing demand. Any manipulation that increases the demand for a reward also increases risk aversion, by selectively increasing the frequency of safe choices without affecting frequency of risky choices.     

Significance of motivation balance for a choice of dog's behavior under conditions of environmental uncertainty

Two experimental models with a choice between two reinforcements were used for assessment of individual typological features of dogs. In the first model dogs were given the choice of homogeneous food reinforcements: between less valuable constantly delivered reinforcement and more valuable reinforcement but delivered with low probabilities. In the second model the dogs had the choice of heterogeneous reinforcements: between performing alimentary and defensive reactions. Under conditions of rise of uncertainty owing to a decrease in probability of getting the valuable food, two dogs continued to prefer the valuable reinforcement, while the third animal gradually shifted its behavior from the choice of a highly valuable but infrequent reward to a less valuable but easily achieved reinforcement. Under condition of choice between the valuable food reinforcement and avoidance of electrocutaneous stimulation, the first two dogs preferred food, whereas the third animal which had been previously oriented to the choice of the low-valuable constant reinforcement, steadily preferred the avoidance behavior. The data obtained are consistent with the hypothesis according to which the individual typological characteristics of animals's (human's) behavior substantially depend on two parameters: extent of environmental uncertainty and subjective features of reinforcement assessment.     

Response of wetland soil carbon to groundwater conservation: Probabilistic outcomes from error propagation

Water loss compromises functions performed by wetland ecosystems. Efforts to rehabilitate wetland function typically begin with attempts to reestablish hydrology. These activities are often not monitored, so tools to extract information from them could partly offset the lost opportunity to learn from whole-ecosystem hydrological manipulation. In 2002, groundwater abstraction was lessened by 35% throughout 1700 km² of west-central Florida (USA). I assembled a pathway of correlations to project how this hydrological manipulation affected water levels and soil carbon (C) storage in overlying wetlands. Parameter values and residual error in these statistical models were resampled from known variances, thereby propagating uncertainty through the pathway of relationships, and expressing the response of soil C probabilistically. Projected soil C probability distributions were most distinguishable between full and moderate (30% less) abstraction. With more severe abstraction cutbacks, gains in projected soil C became more marginal and uncertain, suggesting that wetland soil C pools are not notably impacted by low-volume groundwater abstraction. Reducing uncertainty in projected soil C will require better understanding the dynamic response of soil C to increases in the amount of time that wetland soil is inundated. The step-by-step error propagation routine presented here is a platform for assimilating information from diverse sources in order to project probabilistic responses of ecosystem function to wetland restoration attempts, and it helps identify where further certainty is most wanted in a pathway of cause–effect relationships.     

Teaching and learning in a probabilistic prisoner's dilemma

The prisoner's dilemma is much studied in social psychology and decision-making because it models many real-world conflicts. In everyday terms, the choice to 'cooperate' (maximize reward for the group) or 'defect' (maximize reward for the individual) is often attributed to altruistic or selfish motives. Alternatively, behavior during a dilemma may be understood as a function of reinforcement and punishment. Human participants played a prisoner's-dilemma-type game (for points exchangeable for money) with a computer that employed either a teaching strategy (a probabilistic version of tit-for-tat), in which the computer reinforced or punished participants' cooperation or defection, or a learning strategy (a probabilistic version of Pavlov), in which the computer's responses were reinforced and punished by participants' cooperation and defection. Participants learned to cooperate against both computer strategies. However, in a second experiment which varied the context of the game, they learned to cooperate only against one or other strategy; participants did not learn to cooperate against tit-for-tat when they believed that they were playing against another person; participants did not learn to cooperate against Pavlov when the computer's cooperation probability was signaled by a spinner. The results are consistent with the notion that people are biased not only to cooperate or defect on individual social choices, but also to employ one or other strategy of interaction in a pattern across social choices.     

Risk-sensitive foraging in rats: the effects of response-effort and reward-amount manipulations on choice behavior

The literature on risk-sensitive foraging theory provides several accounts of species that fluctuate between risk-averse and risk-prone strategies. The daily energy budget rule suggests that shifts in foraging strategy are precipitated by changes in the forager's energy budget. Researchers have attempted to alter the organism's energy budget using a variety of techniques such as food deprivation, manipulation of ambient temperatures, and delays to food reward; however, response-effort manipulations have been relatively neglected. A choice preparation using a wheel-running response and rats examined risk-sensitive preferences when both response effort and reward amounts were manipulated. Concurrently available reinforcement schedules (FI/60 and VI/60) yielded equivalent food amounts per unit time in all treatments. Two levels of response effort (20 or 120 g tangential resistance) and two levels of reward amount (three or nine pellets) were combined to form four distinct response-effort/reward-amount pairings. Increasing reward amounts significantly shifted choice toward the FI schedule in both response-effort conditions. The incidence of choice preference and the magnitude of shifts in choice were greater for the high response-effort conditions than for the low response-effort conditions. Implications of the significant interaction between response effort and reward amount are discussed in terms of a general energy-budget model.     

The attention–aversion gap: how allocation of attention relates to loss aversion

Loss aversion is often assumed to be a basic and far-reaching psychological regularity in behavior. Yet empirical evidence is accumulating to challenge the assumption of widespread loss aversion in choice. We suggest that a key reason for the apparently elusive nature of loss aversion may be that its manifestation in choice is state-dependent and distinct from a more state-independent principle of heightened attention to losses relative to gains. Using data from process-tracing studies, we show that people invest more attentional resources when evaluating losses than when evaluating gains, even when their choices do not reflect loss aversion. Our evidence converges with previous findings on how losses influence exploratory search as well as physiological, hormonal, and neural responses. Increased attention to losses relative to gains seems to be a necessary but not a sufficient condition for loss aversion in choice.     

The value of success: acquiring gains, avoiding losses, and simply being successful

A large network of spatially contiguous, yet anatomically distinct regions in medial frontal cortex is involved in reward processing. Although it is clear these regions play a role in critical aspects of reward-related learning and decision-making, the individual contributions of each component remains unclear. We explored dissociations in reward processing throughout several key regions in the reward system and aimed to clarify the nature of previously observed outcome-related activity in a portion of anterior medial orbitofrontal cortex (mOFC). Specifically, we tested whether activity in anterior mOFC was related to processing successful actions, such that this region would respond similarly to rewards with and without tangible benefits, or whether this region instead encoded only quantifiable outcome values (e.g., money). Participants performed a task where they encountered monetary gains and losses (and non-gains and non-losses) during fMRI scanning. Critically, in addition to the outcomes with monetary consequences, the task included trials that provided outcomes without tangible benefits (participants were simply told that they were correct or incorrect). We found that anterior mOFC responded to all successful outcomes regardless of whether they carried tangible benefits (monetary gains and non-losses) or not (controls). These results support the hypothesis that anterior mOFC processes rewards in terms of a common currency and is capable of providing reward-based signals for everything we value, whether it be primary or secondary rewards or simply a successful experience without objectively quantifiable benefits.     

Response properties of human amygdala subregions: evidence based on functional MRI combined with probabilistic anatomical maps

The human amygdala is thought to play a pivotal role in the processing of emotionally significant sensory information. The major subdivisions of the human amygdala-the laterobasal group (LB), the superficial group (SF), and the centromedial group (CM)-have been anatomically delineated, but the functional response properties of these amygdala subregions in humans are still unclear. We combined functional MRI with cyto-architectonically defined probabilistic maps to analyze the response characteristics of amygdala subregions in subjects presented with auditory stimuli. We found positive auditory stimulation-related signal changes predominantly in probabilistically defined LB, and negative responses predominantly in SF and CM. In the left amygdala, mean response magnitude in the core area of LB with 90-100% assignment probability was significantly larger than in the core areas of SF and CM. These differences were observed for pleasant and unpleasant stimuli. Our findings reveal that the probabilistically defined anatomical subregions of the human amygdala show distinctive fMRI response patterns. The stronger auditory responses in LB as compared with SF and CM may reflect a predominance of auditory inputs to human LB, similar to many animal species in which the majority of sensory, including auditory, afferents project to this subdivision of the amygdala. Our study indicates that the intrinsic functional differentiation of the human amygdala may be probed using fMRI combined with probabilistic anatomical maps.     

Neural Sensitivity to Absolute and Relative Anticipated Reward in Adolescents

Adolescence is associated with a dramatic increase in risky and impulsive behaviors that have been attributed to developmental differences in neural processing of rewards. In the present study, we sought to identify age differences in anticipation of absolute and relative rewards. To do so, we modified a commonly used monetary incentive delay (MID) task in order to examine brain activity to relative anticipated reward value (neural sensitivity to the value of a reward as a function of other available rewards). This design also made it possible to examine developmental differences in brain activation to absolute anticipated reward magnitude (the degree to which neural activity increases with increasing reward magnitude). While undergoing fMRI, 18 adolescents and 18 adult participants were presented with cues associated with different reward magnitudes. After the cue, participants responded to a target to win money on that trial. Presentation of cues was blocked such that two reward cues associated with $.20, $1.00, or $5.00 were in play on a given block. Thus, the relative value of the $1.00 reward varied depending on whether it was paired with a smaller or larger reward. Reflecting age differences in neural responses to relative anticipated reward (i.e., reference dependent processing), adults, but not adolescents, demonstrated greater activity to a $1 reward when it was the larger of the two available rewards. Adults also demonstrated a more linear increase in ventral striatal activity as a function of increasing absolute reward magnitude compared to adolescents. Additionally, reduced ventral striatal sensitivity to absolute anticipated reward (i.e., the difference in activity to medium versus small rewards) correlated with higher levels of trait Impulsivity. Thus, ventral striatal activity in anticipation of absolute and relative rewards develops with age. Absolute reward processing is also linked to individual differences in Impulsivity.     

Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value

Risky decision-making is altered in humans and animals with damage to the orbitofrontal cortex. However, the cellular function of the intact orbitofrontal cortex in processing information relevant for risky decisions is unknown. We recorded responses of single orbitofrontal neurons while monkeys viewed visual cues representing the key decision parameters, reward risk and value. Risk was defined as the mathematical variance of binary symmetric probability distributions of reward magnitudes; value was defined as non-risky reward magnitude. Monkeys displayed graded behavioral preferences for risky outcomes, as they did for value. A population of orbitofrontal neurons showed a distinctive risk signal: their cues and reward responses covaried monotonically with the variance of the different reward distributions without monotonically coding reward value. Furthermore, a small but statistically significant fraction of risk responses also coded reward value. These risk signals may provide physiological correlates for the role of the orbitofrontal cortex in risk processing.     

Association between Stock Market Gains and Losses and Google Searches

Experimental studies in the area of Psychology and Behavioral Economics have suggested that people change their search pattern in response to positive and negative events. Using Internet search data provided by Google, we investigated the relationship between stock-specific events and related Google searches. We studied daily data from 13 stocks from the Dow-Jones and NASDAQ100 indices, over a period of 4 trading years. Focusing on periods in which stocks were extensively searched (Intensive Search Periods), we found a correlation between the magnitude of stock returns at the beginning of the period and the volume, peak, and duration of search generated during the period. This relation between magnitudes of stock returns and subsequent searches was considerably magnified in periods following negative stock returns. Yet, we did not find that intensive search periods following losses were associated with more Google searches than periods following gains. Thus, rather than increasing search, losses improved the fit between people’s search behavior and the extent of real-world events triggering the search. The findings demonstrate the robustness of the attentional effect of losses.