Axiomatic methods, dopamine and reward prediction error

The phasic firing rate of midbrain dopamine neurons has been shown to respond both to the receipt of rewarding stimuli, and the degree to which such stimuli are anticipated by the recipient. This has led to the hypothesis that these neurons encode reward prediction error (RPE)-the difference between how rewarding an event is, and how rewarding it was expected to be. However, the RPE model is one of a number of competing explanations for dopamine activity that have proved hard to disentangle, mainly because they are couched in terms of latent, or unobservable, variables. This article describes techniques for dealing with latent variables common in economics and decision theory, and reviews work that uses these techniques to provide simple, non-parametric tests of the RPE hypothesis, allowing clear differentiation between competing explanations.     

Stimulus-dependent adjustment of reward prediction error in the midbrain

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.     

The therapeutic relationship and adherence to antipsychotic medication in schizophrenia

Objective

Previous research has shown that a better therapeutic relationship (TR) predicts more positive attitudes towards antipsychotic medication, but did not address whether it is also linked with actual adherence. This study investigated whether the TR is associated with adherence to antipsychotics in patients with schizophrenia.

Methods

134 clinicians and 507 of their patients with schizophrenia or a related psychotic disorder participated in a European multi-centre study. A logistic regression model examined how the TR as rated by patients and by clinicians is associated with medication adherence, adjusting for clinician clustering and symptom severity.

Results

Patient and clinician ratings of the TR were weakly inter-correlated (r_s = 0.13, p = 0.004), but each was independently linked with better adherence. After adjusting for patient rated TR and symptom severity, each unit increase in clinician rated TR was associated with an increase of the odds ratio of good compliance by 65.9% (95% CI: 34.6% to 104.5%). After adjusting for clinician rated TR and symptom severity, for each unit increase in patient rated TR the odds ratio of good compliance was increased by 20.8% (95% CI: 4.4% to 39.8%).

Conclusions

A better TR is associated with better adherence to medication among patients with schizophrenia. Patients'' and clinicians'' perspectives of the TR are both important, but may reflect distinct aspects.     

Spontaneous prediction error generation in schizophrenia

Goal-directed human behavior is enabled by hierarchically-organized neural systems that process executive commands associated with higher brain areas in response to sensory and motor signals from lower brain areas. Psychiatric diseases and psychotic conditions are postulated to involve disturbances in these hierarchical network interactions, but the mechanism for how aberrant disease signals are generated in networks, and a systems-level framework linking disease signals to specific psychiatric symptoms remains undetermined. In this study, we show that neural networks containing schizophrenia-like deficits can spontaneously generate uncompensated error signals with properties that explain psychiatric disease symptoms, including fictive perception, altered sense of self, and unpredictable behavior. To distinguish dysfunction at the behavioral versus network level, we monitored the interactive behavior of a humanoid robot driven by the network. Mild perturbations in network connectivity resulted in the spontaneous appearance of uncompensated prediction errors and altered interactions within the network without external changes in behavior, correlating to the fictive sensations and agency experienced by episodic disease patients. In contrast, more severe deficits resulted in unstable network dynamics resulting in overt changes in behavior similar to those observed in chronic disease patients. These findings demonstrate that prediction error disequilibrium may represent an intrinsic property of schizophrenic brain networks reporting the severity and variability of disease symptoms. Moreover, these results support a systems-level model for psychiatric disease that features the spontaneous generation of maladaptive signals in hierarchical neural networks.     

Learning from sensory and reward prediction errors during motor adaptation

Voluntary motor commands produce two kinds of consequences. Initially, a sensory consequence is observed in terms of activity in our primary sensory organs (e.g., vision, proprioception). Subsequently, the brain evaluates the sensory feedback and produces a subjective measure of utility or usefulness of the motor commands (e.g., reward). As a result, comparisons between predicted and observed consequences of motor commands produce two forms of prediction error. How do these errors contribute to changes in motor commands? Here, we considered a reach adaptation protocol and found that when high quality sensory feedback was available, adaptation of motor commands was driven almost exclusively by sensory prediction errors. This form of learning had a distinct signature: as motor commands adapted, the subjects altered their predictions regarding sensory consequences of motor commands, and generalized this learning broadly to neighboring motor commands. In contrast, as the quality of the sensory feedback degraded, adaptation of motor commands became more dependent on reward prediction errors. Reward prediction errors produced comparable changes in the motor commands, but produced no change in the predicted sensory consequences of motor commands, and generalized only locally. Because we found that there was a within subject correlation between generalization patterns and sensory remapping, it is plausible that during adaptation an individual''s relative reliance on sensory vs. reward prediction errors could be inferred. We suggest that while motor commands change because of sensory and reward prediction errors, only sensory prediction errors produce a change in the neural system that predicts sensory consequences of motor commands.     

Principles of pleasure prediction: specifying the neural dynamics of human reward learning

Accumulating evidence from nonhuman primates suggests that midbrain dopamine cells code reward prediction errors and that this signal subserves reward learning in dopamine-receiving brain structures. In this issue of Neuron, McClure et al. and O'Doherty et al. use event-related fMRI to provide some of the strongest evidence to date that the reward prediction error model of dopamine system activity applies equally well to human reward learning.     

Foraging preferences: Response to reward skew

Recent theoretical and empirical work on foraging behaviour suggests that animals may respond to both the means and variances in benefits associated with available resources. We attempt to extend this analysis by asking if reward skew (third moment about the mean) might influence preference when two options have equal means and equal variances. We examine how minimizing the probability of starvation might induce response to skew. In the Appendix we develop an expected ‘fitness’ model which follows from economic theory and indicates more general conditions concerning responses to skew. We also report experiments involving foraging white-crowned sparrows (Zonotrichia leucophrys). Under conditions where positive skew should be favoured, the birds' behaviour supports the prediction. However, their response to skew is not as strong as responses to variance noted in the same individuals.     

Plants reward seed dispersers in proportion to their effort: The relationship between pulp mass and seed mass in vertebrate dispersed plants

In this paper I develop a null model for the expected relationship between seed mass and the mass of dispersal structure (reward) for vertebrate-dispersed plant species. The model is based on the simple assumption that the reward associated with a given seed mass is commensurate with work required to move it, and predicts that reward mass should scale relative to seed mass with an exponent of 4/3 (1.3). I tested this relationship between- and within-species of vertebrate-dispersed plants from four families from tropical rain forest in north Queensland, Australia. At a community-level there was a significant isometric relationship between log mean pulp mass and log mean seed mass across species. When family membership was considered, the estimate for the common slope between families was 1.32, surprisingly similar to the exponent predicted from commensurate reward. In addition, the 95% CI of the common slope did not include unity, providing no support for isometry. There was also no evidence that the relationships between mean log pulp mass and mean log seed mass were significantly different between families. This simple null model may be a common “rule” governing mean allocation to reward in all plant–animal dispersal mutualisms and its confirmation is the first evidence that animal dispersers have shaped the evolution of seed traits. However, I found no evidence that the scaling relationships within-species were consistently predicted by commensurate reward – a “taxon-level effect”. I suggest that the taxon-level effect arises because mean seed and mean reward mass within each species arises due to community-wide, disperser-mediated selection to produce equally attractive fruits, whereas within-species allometries may be determined by selection for fruit traits that enhance either dispersal probabilities, offspring survival or both, and these will be contingent on the environmental context into which seeds are released. G. P. Cheplick     

Exposure to cold: aversive Pavlovian conditioning in individual Drosophila melanogaster

Abstract. Temperature changes can be especially threatening for ectotherms, such as Drosophila melanogaster (Diptera: Drosophilidea Meigen, 1830 ), and in this study we tested whether flies can associate olfactory stimuli with a sudden drop in temperature. Such Pavlovian conditioning would allow them to make appropriate behavioural and/or physiological responses in the future. We found that exposing individual flies to one of two odours in the presence of a sudden drop in temperature resulted in Pavlovian conditioning with flies subsequently avoiding the odour paired with cold. The characteristics of Pavlovian conditioning in flies were comparable to those observed for mammalian species. Specifically, the strength of conditioning increased with increasing intensity of the cold and decreased as the time interval between the olfactory stimulus (CS) and cold (US) was lengthened. Finally, the order in which CS and US were presented affected the strength of conditioning. Learning was observed when the CS preceded US and when the US immediately preceded the CS, but not when the CS preceded the US by 30 s or more. These results provide further evidence for learning in individual flies, and confirm that Pavlovian conditioning is a general mechanism used by organisms to obtain information about their environment.     

Banburismus and the brain: decoding the relationship between sensory stimuli,decisions, and reward

This article relates a theoretical framework developed by British codebreakers in World War II to the neural computations thought to be responsible for forming categorical decisions about sensory stimuli. In both, a weight of evidence is computed and accumulated to support or oppose the alternative interpretations. A decision is reached when the evidence reaches a threshold value. In the codebreaking scheme, the threshold determined the speed and accuracy of the decision process. Here we propose that in the brain, the threshold may be controlled by neural circuits that calculate the rate of reward.     

Servers for sequence-structure relationship analysis and prediction

We describe several algorithms and public servers that were developed to analyze and predict various features of protein structures. These servers provide information about the covalent state of cysteine (CYSREDOX), as well as about residues involved in non-covalent cross links that play an important role in the structural stability of proteins (SCIDE and SCPRED). We also discuss methods and servers developed to identify helical transmembrane proteins from large databases and rough genomic data, including two of the most popular transmembrane prediction methods, DAS and HMMTOP. Several biologically interesting applications of these servers are also presented. The servers are available through http://www.enzim.hu/servers.html.     

Cigarette smoking in male patients with chronic schizophrenia in a Chinese population: prevalence and relationship to clinical phenotypes

The high prevalence of smoking in schizophrenia of European background may be related to smoking's reducing clinical symptoms and medication side effects. Because smoking prevalence and its associations with clinical phenotypes are less well characterized in Chinese than European patients with schizophrenia, we assessed these smoking behaviors using clinician-administered questionnaires and the Fagerstrom Test for Nicotine Dependence (FTND) in 776 Chinese male schizophrenia and 560 control subjects. Patients also were rated on the Positive and Negative Symptom Scale (PANSS), the Simpson and Angus Extrapyramidal Symptom Rating Scale (SAES), and the Abnormal Involuntary Movement Scale (AIMS). We found that the schizophrenia patients had a higher lifetime incidence of smoking (79% vs 63%), were more likely to be heavy smokers (61% vs 31%), and had lower smoking cessation rates (4% vs 9%) (all p<0.0001) than controls. Among the schizophrenia patients smoking prevalence increased with age, with the largest difference from controls in the age cohort of 55-75 years: 75% vs 46% (p<0.0001). Among the schizophrenia smokers 73% started to smoke before the onset of their illness by an average of 7.6 years. The patients with schizophrenia who were current smokers scored significantly lower on the PANSS negative symptom subscore (p<0.005), and on the SAES symptom scale (p<0.04; Bonferroni corrected p>0.05) than the non-smoking patients. These results suggest that Chinese males with schizophrenia smoke more frequently than the general population. Further, smokers with schizophrenia may display fewer negative symptoms and possibly less parkinsonism than non-smokers with schizophrenia.     

Relevance of genetic relationship in GWAS and genomic prediction

The objective of this study was to analyze the relevance of relationship information on the identification of low heritability quantitative trait loci (QTLs) from a genome-wide association study (GWAS) and on the genomic prediction of complex traits in human, animal and cross-pollinating populations. The simulation-based data sets included 50 samples of 1000 individuals of seven populations derived from a common population with linkage disequilibrium. The populations had non-inbred and inbred progeny structure (50 to 200) with varying number of members (5 to 20). The individuals were genotyped for 10,000 single nucleotide polymorphisms (SNPs) and phenotyped for a quantitative trait controlled by 10 QTLs and 90 minor genes showing dominance. The SNP density was 0.1 cM and the narrow sense heritability was 25%. The QTL heritabilities ranged from 1.1 to 2.9%. We applied mixed model approaches for both GWAS and genomic prediction using pedigree-based and genomic relationship matrices. For GWAS, the observed false discovery rate was kept below the significance level of 5%, the power of detection for the low heritability QTLs ranged from 14 to 50%, and the average bias between significant SNPs and a QTL ranged from less than 0.01 to 0.23 cM. The QTL detection power was consistently higher using genomic relationship matrix. Regardless of population and training set size, genomic prediction provided higher prediction accuracy of complex trait when compared to pedigree-based prediction. The accuracy of genomic prediction when there is relatedness between individuals in the training set and the reference population is much higher than the value for unrelated individuals.     

Psychological distance to reward in monkeys

A concurrent-chains schedule was employed to examine crab-eating macaques' choice between segmented and unsegmented fixed-interval 30-s schedules. A stimulus change occurred during one of the fixed-interval schedules (segmented schedule), while no stimulus change occurred during the other (unsegmented schedule). The subjects were exposed to three consecutive stimulus-change conditions, in an attempt to explore whether correlation of the stimulus changes with a fixed interval of nonreinforcement affects the preference between the alternatives. When the stimulus change always occurred at a half point of the segmented schedule (the first and third conditions), all subjects preferred the fixed-interval schedule without stimulus changes. This replicated previous findings with pigeons and humans. On the other hand, when the intervals before and after the stimulus change varied on every entry to the segmented schedule with the sum of them being constant (the second condition), the subjects were indifferent to the alternatives. These results are consistent with the view that the preference for unsegmented schedules over segmented ones might be due to aversiveness which the initial component in the segmented schedule possesses as a result of its correlation with a fixed interval of nonreinforcement.     

The relationship between membrane pathology and language disorder in schizophrenia

Receptive language disorder in schizophrenia has been hypothesized to involve a fundamental deficit in the temporal (time-based) dynamics of brain function that includes disruptions to patterns of activation and synchronization. In this paper, candidate mechanisms and pathways that could account for this basic deficit are discussed. Parallels are identified between the patterns of language dysfunction observed for schizophrenia and dyslexia, two separate clinical disorders that may share a common abnormality in cell membrane phospholipids. A heuristic is proposed which details a trajectory involving an interaction of brain fatty acids and second-messenger function that modulates synaptic efficacy, and, in turn, influences language processing in schizophrenia patients. It is additionally hypothesized that a primary deficit of functional excitation originating in the cerebellum, in combination with a compensatory decrease of functional inhibition in the prefrontal cortex, influences receptive language dysfunction in schizophrenia.     

Catecholamine reward pathways and schizophrenia: the mechanism of the antipsychotic effect and the site of the primary disturbance.

Two catecholamine-containing pathways, the locus ceruleus system and the dopamine neurons arising from the ventral mid-brain, may be involved in reward. Dopamine neurons function as a system for energizing the organism's responses and directing them toward significant environmental stimuli, but the functions of the locus ceruleus system remain obscure. It appears increasingly likely that neuroleptic drugs exert their anti-psychotic effects in acute schizophrenia by blocking dopamine receptors, although the time course of the effect suggests that the mechanism is more complex than a simple reversal of a neurohumoral imbalance. Evidence from postmortem studies suggests that, at least in the chronic state, dopamine turnover is not increased, but that there may be an increase in postsynaptic receptor density in some cases, including some patients who apparently had not received medication in the year before death. The evidence is consistent with Olds and Travis' conjecture that "counteraction of positive feedback processes subserving positive reinforcement mechanisms may be a key to control of certain psychotic episodes".