Worth the ‘EEfRT’? The Effort Expenditure for Rewards Task as an Objective Measure of Motivation and Anhedonia

Background

Of the putative psychopathological endophenotypes in major depressive disorder (MDD), the anhedonic subtype is particularly well supported. Anhedonia is generally assumed to reflect aberrant motivation and reward responsivity. However, research has been limited by a lack of objective measures of reward motivation. We present the Effort-Expenditure for Rewards Task (EEfRT or “effort”), a novel behavioral paradigm as a means of exploring effort-based decision-making in humans. Using the EEfRT, we test the hypothesis that effort-based decision-making is related to trait anhedonia.

Methods/Results

61 undergraduate students participated in the experiment. Subjects completed self-report measures of mood and trait anhedonia, and completed the EEfRT. Across multiple analyses, we found a significant inverse relationship between anhedonia and willingness to expend effort for rewards.

Conclusions

These findings suggest that anhedonia is specifically associated with decreased motivation for rewards, and provide initial validation for the EEfRT as a laboratory-based behavioral measure of reward motivation and effort-based decision-making in humans.     

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.     

Differential Reward Learning for Self and Others Predicts Self-Reported Altruism

In social environments, decisions not only determine rewards for oneself but also for others. However, individual differences in pro-social behaviors have been typically studied through self-report. We developed a decision-making paradigm in which participants chose from card decks with differing rewards for themselves and charity; some decks gave similar rewards to both, while others gave higher rewards for one or the other. We used a reinforcement-learning model that estimated each participant''s relative weighting of self versus charity reward. As shown both in choices and model parameters, individuals who showed relatively better learning of rewards for charity – compared to themselves – were more likely to engage in pro-social behavior outside of a laboratory setting indicated by self-report. Overall rates of reward learning, however, did not predict individual differences in pro-social tendencies. These results support the idea that biases toward learning about social rewards are associated with one''s altruistic tendencies.     

Reduced sensitivity to immediate reward during decision-making in older than younger adults

We examined whether older adults differ from younger adults in the degree to which they favor immediate over delayed rewards during decision-making. To examine the neural correlates of age-related differences in delay discounting we acquired functional MR images while participants made decisions between smaller but sooner and larger but later monetary rewards. The behavioral results show age-related reductions in delay discounting. Less impulsive decision-making in older adults was associated with lower ventral striatal activations to immediate reward. Furthermore, older adults showed an overall higher percentage of delayed choices and reduced activity in the dorsal striatum than younger adults. This points to a reduced reward sensitivity of the dorsal striatum in older adults. Taken together, our findings indicate that less impulsive decision-making in older adults is due to a reduced sensitivity of striatal areas to reward. These age-related changes in reward sensitivity may result from transformations in dopaminergic neuromodulation with age.     

Economic principles motivating social attention in humans

We know little about the processes by which we evaluate the opportunity to look at another person. We propose that behavioural economics provides a powerful approach to understanding this basic aspect of social attention. We hypothesized that the decision process culminating in attention to another person follows the same economic principles that govern choices about rewards such as food, drinks and money. Specifically, such rewards are discounted as a function of time, are tradable for other rewards, and reinforce work. Behavioural and neurobiological evidence suggests that looking at other people can also be described as rewarding, but to what extent these economic principles apply to social orienting remains unknown. Here, we show that the opportunity to view pictures of the opposite sex is discounted by delay to viewing, substitutes for money and reinforces work. The reward value of photos of the opposite sex varied with physical attractiveness and was greater in men, suggesting differential utility of acquiring visual information about the opposite sex in men and women. Together, these results demonstrate that choosing whom to look at follows a general set of economic principles, implicating shared neural mechanisms in both social and non-social decision making.     

Effect of Familiarity on Reward Anticipation in Children with and without Autism Spectrum Disorders

Background

Previous research on the reward system in autism spectrum disorders (ASD) suggests that children with ASD anticipate and process social rewards differently than typically developing (TD) children—but has focused on the reward value of unfamiliar face stimuli. Children with ASD process faces differently than their TD peers. Previous research has focused on face processing of unfamiliar faces, but less is known about how children with ASD process familiar faces. The current study investigated how children with ASD anticipate rewards accompanied by familiar versus unfamiliar faces.

Methods

The stimulus preceding negativity (SPN) of the event-related potential (ERP) was utilized to measure reward anticipation. Participants were 6- to 10-year-olds with (N = 14) and without (N = 14) ASD. Children were presented with rewards accompanied by incidental face or non-face stimuli that were either familiar (caregivers) or unfamiliar. All non-face stimuli were composed of scrambled face elements in the shape of arrows, controlling for visual properties.

Results

No significant differences between familiar versus unfamiliar faces were found for either group. When collapsing across familiarity, TD children showed larger reward anticipation to face versus non-face stimuli, whereas children with ASD did not show differential responses to these stimulus types. Magnitude of reward anticipation to faces was significantly correlated with behavioral measures of social impairment in the ASD group.

Conclusions

The findings do not provide evidence for differential reward anticipation for familiar versus unfamiliar face stimuli in children with or without ASD. These findings replicate previous work suggesting that TD children anticipate rewards accompanied by social stimuli more than rewards accompanied by non-social stimuli. The results do not support the idea that familiarity normalizes reward anticipation in children with ASD. Our findings also suggest that magnitude of reward anticipation to faces is correlated with levels of social impairment for children with ASD.     

Discounting of sequences of delayed rewards of different amounts

The main purpose of this study was to determine whether the magnitude effect is present in cases where delayed sequences of rewards are discounted. The magnitude effect refers to the inverse relationship between the amount of a reward and the steepness of temporal discounting. This study was conducted with a computer program to estimate the indifference points, which served as indicators of the present subjective value of delayed sequences of small and large rewards. In the indifference point the subjective value of a single, immediate reward was equal to the subjective value of the delayed sequence (or to the value of a single delayed reward). As a control condition, we added an experimental task involving choices between single immediate and single delayed rewards. The experiment showed that the sequences of large rewards are discounted less steeply than are the sequences of small rewards. This finding suggests that the magnitude effect is present within the delayed sequences of rewards. In addition, when outcomes are relatively large, the results suggest that a single reward is discounted less steeply than the sequence of a total nominal value equal to this single reward. However, for relatively small rewards, the difference is not statistically significant. The less steep discounting of sequences of large rewards may explain the reward-bundling effect, which refers to less steep discounting of longer sequences than of shorter ones: longer sequences usually have greater overall nominal value. The present study was conducted on hypothetical rewards, and the results should be validated using real rewards.     

Structure learning in human sequential decision-making

Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner.     

Neural economics and the biological substrates of valuation

A recent flurry of neuroimaging and decision-making experiments in humans, when combined with single-unit data from orbitofrontal cortex, suggests major additions to current models of reward processing. We review these data and models and use them to develop a specific computational relationship between the value of a predictor and the future rewards or punishments that it promises. The resulting computational model, the predictor-valuation model (PVM), is shown to anticipate a class of single-unit neural responses in orbitofrontal and striatal neurons. The model also suggests how neural responses in the orbitofrontal-striatal circuit may support the conversion of disparate types of future rewards into a kind of internal currency, that is, a common scale used to compare the valuation of future behavioral acts or stimuli.     

Early Effects of Reward Anticipation Are Modulated by Dopaminergic Stimulation

The abilities to predict future rewards and assess the value of reward delivery are crucial aspects of adaptive behavior. While the mesolimbic system, including dopaminergic midbrain, ventral striatum and prefrontal cortex have long been associated with reward processing, recent studies also indicate a prominent role of early visual brain regions. However, the precise underlying neural mechanisms still remain unclear. To address this issue, we presented participants with visual cues predicting rewards of high and low magnitudes and probability (2×2 factorial design), while neural activity was scanned using magnetoencephalography. Importantly, one group of participants received 150 mg of the dopamine precursor levodopa prior to the experiment, while another group received a placebo. For the placebo group, neural signals of reward probability (but not magnitude) emerged at ∼100 ms after cue presentation at occipital sensors in the event-related magnetic fields. Importantly, these probability signals were absent in the levodopa group indicating a close link. Moreover, levodopa administration reduced oscillatory power in the high (20–30 Hz) and low (13–20 Hz) beta band during both reward anticipation and delivery. Taken together, our findings indicate that visual brain regions are involved in coding prospective reward probability but not magnitude and that these effects are modulated by dopamine.     

Rhesus Monkeys' Valuation of Vocalizations during a Free-Choice Task

Adaptive behavior requires that animals integrate current and past information with their decision-making. One important type of information is auditory-communication signals (i.e., species-specific vocalizations). Here, we tested how rhesus monkeys incorporate the opportunity to listen to different species-specific vocalizations into their decision-making processes. In particular, we tested how monkeys value these vocalizations relative to the opportunity to get a juice reward. To test this hypothesis, monkeys chose one of two targets to get a varying juice reward; at one of those targets, in addition to the juice reward, a vocalization was presented. By titrating the juice amounts at the two targets, we quantified the relationship between the monkeys'' juice choices relative to the opportunity to listen to a vocalization. We found that, rhesus were not willing to give up a large juice reward to listen to vocalizations indicating that, relative to a juice reward, listening to vocalizations has a low value.     

Non-human primate token use shows possibilities but also limitations for establishing a form of currency

Non-human primates evaluate choices based on quantitative information and subjective valuation of options. Non-human primates can learn to value tokens as placeholders for primary rewards (such as food). With those tokens established as a potential form of ‘currency’, it is then possible to examine how they respond to opportunities to earn and use tokens in ways such as accumulating tokens or exchanging tokens with each other or with human experimenters to gain primary rewards. Sometimes, individuals make efficient and beneficial choices to obtain tokens and then exchange them at the right moments to gain optimal reward. Sometimes, they even accumulate such rewards through extended delay of gratification, or through other exchange-based interactions. Thus, non-human primates are capable of associating value to arbitrary tokens that may function as currency-like stimuli, but there also are strong limitations on how non-human primates can integrate such tokens into choice situations or use such tokens to fully ‘symbolize’ economic decision-making. These limitations are important to acknowledge when considering the evolutionary emergence of currency use in our species.This article is part of the theme issue ‘Existence and prevalence of economic behaviours among non-human primates’.     

Using Effort to Measure Reward Value of Faces in Children with Autism

According to one influential account, face processing atypicalities in autism reflect reduced reward value of faces, which results in limited attention to faces during development and a consequent failure to acquire face expertise. Surprisingly, however, there is a paucity of work directly investigating the reward value of faces for individuals with autism and the evidence for diminished face rewards in this population remains equivocal. In the current study, we measured how hard children with autism would work to view faces, using an effortful key-press sequence, and whether they were sensitive to the differential reward value of attractive and unattractive faces. Contrary to expectations, cognitively able children with autism did not differ from typically developing children of similar age and ability in their willingness to work to view faces. Moreover, the effort expended was strongly positively correlated with facial attractiveness ratings in both groups of children. There was also no evidence of atypical reward values for other, less social categories (cars and inverted faces) in the children with autism. These results speak against the possibility that face recognition difficulties in autism are explained by atypical reward value of faces.     

The effects of reward quality on risk-sensitivity in Rattus norvegicus

Risk-sensitive foraging theory (RSFT) was developed to explain a choice between a variable (risk-prone) or constant (risk-averse) option. In the RSFT literature, qualitative shifts in risk-sensitivity have been explained by fluctuations in daily caloric energy budget (DEB). The DEB rule describes foragers’ choices as being based on fitness and rate of gain. If the DEB rule is correct, rewards that differ in caloric returns should cause differences in foragers’ sensitivity to risk. However, few studies have explored the influence of reward quality on risk-sensitivity in mammals. The present study was designed to examine the effects of reward quality on risk-sensitivity when reward magnitude, delay to reward, body mass, and response effort were controlled. Results from the current study demonstrated that subjects rewarded with a high calorie reward (i.e., sugar) made significantly fewer choices for a variable option than subjects rewarded with a lower calorie reward (i.e., grain). These results are consistent with the predictions of the DEB rule, and add to the RSFT literature where reward quality was manipulated by describing difference in risk-sensitivity in mammals. Suggestions for future research include an examination of risk-sensitivity where flavor and caloric return are manipulated.     

Timing in reward and decision processes

Sensitivity to time, including the time of reward, guides the behaviour of all organisms. Recent research suggests that all major reward structures of the brain process the time of reward occurrence, including midbrain dopamine neurons, striatum, frontal cortex and amygdala. Neuronal reward responses in dopamine neurons, striatum and frontal cortex show temporal discounting of reward value. The prediction error signal of dopamine neurons includes the predicted time of rewards. Neurons in the striatum, frontal cortex and amygdala show responses to reward delivery and activities anticipating rewards that are sensitive to the predicted time of reward and the instantaneous reward probability. Together these data suggest that internal timing processes have several well characterized effects on neuronal reward processing.     

Teaching and learning in a probabilistic prisoner's dilemma

The prisoner's dilemma is much studied in social psychology and decision-making because it models many real-world conflicts. In everyday terms, the choice to 'cooperate' (maximize reward for the group) or 'defect' (maximize reward for the individual) is often attributed to altruistic or selfish motives. Alternatively, behavior during a dilemma may be understood as a function of reinforcement and punishment. Human participants played a prisoner's-dilemma-type game (for points exchangeable for money) with a computer that employed either a teaching strategy (a probabilistic version of tit-for-tat), in which the computer reinforced or punished participants' cooperation or defection, or a learning strategy (a probabilistic version of Pavlov), in which the computer's responses were reinforced and punished by participants' cooperation and defection. Participants learned to cooperate against both computer strategies. However, in a second experiment which varied the context of the game, they learned to cooperate only against one or other strategy; participants did not learn to cooperate against tit-for-tat when they believed that they were playing against another person; participants did not learn to cooperate against Pavlov when the computer's cooperation probability was signaled by a spinner. The results are consistent with the notion that people are biased not only to cooperate or defect on individual social choices, but also to employ one or other strategy of interaction in a pattern across social choices.     

Activity in posterior parietal cortex is correlated with the relative subjective desirability of action

Behavioral studies suggest that making a decision involves representing the overall desirability of all available actions and then selecting that action that is most desirable. Physiological studies have proposed that neurons in the parietal cortex play a role in selecting movements for execution. To test the hypothesis that these parietal neurons encode the subjective desirability of making particular movements, we exploited Nash's game theoretic equilibrium, during which the subjective desirability of multiple actions should be equal for human players. Behavior measured during a strategic game suggests that monkeys' choices, like those of humans, are guided by subjective desirability. Under these conditions, activity in the parietal cortex was correlated with the relative subjective desirability of actions irrespective of the specific combination of reward magnitude, reward probability, and response probability associated with each action. These observations may help place many recent findings regarding the posterior parietal cortex into a common conceptual framework.     

Relative Changes from Prior Reward Contingencies Can Constrain Brain Correlates of Outcome Monitoring

It is well-known that the affective value of an environment can be relative to whether it reflects an improvement or a worsening from a previous state. A potential explanation for this phenomenon suggests that relative changes from previous reward contingencies can constrain how brain monitoring systems form predictions about future events. In support of this idea, we found that changes per se relative to previous states of learned reward contingencies modulated the Feedback-Related Negativity (FRN), a human brain potential known to index discrepancies between predictions and affective outcomes. Specifically, we observed that environments with a 50% reward probability yielded different FRN patterns according to whether they reflected an improvement or a worsening from a previous environment. Further, we also found that this pattern of results was driven mainly by variations in the amplitude of ERPs to positive outcomes. Overall, these results suggest that relative changes in reward probability from previous learned environments can constrain how neural systems of outcome monitoring formulate predictions about the likelihood of future rewards and nonrewards.     

Multiple reward signals in the brain

The fundamental biological importance of rewards has created an increasing interest in the neuronal processing of reward information. The suggestion that the mechanisms underlying drug addiction might involve natural reward systems has also stimulated interest. This article focuses on recent neurophysiological studies in primates that have revealed that neurons in a limited number of brain structures carry specific signals about past and future rewards. This research provides the first step towards an understanding of how rewards influence behaviour before they are received and how the brain might use reward information to control learning and goal-directed behaviour.