Reward representations and reward-related learning in the human brain: insights from neuroimaging

This review outlines recent findings from human neuroimaging concerning the role of a highly interconnected network of brain areas including orbital and medial prefrontal cortex, amygdala, striatum and dopaminergic mid-brain in reward processing. Distinct reward-related functions can be attributed to different components of this network. Orbitofrontal cortex is involved in coding stimulus reward value and in concert with the amygdala and ventral striatum is implicated in representing predicted future reward. Such representations can be used to guide action selection for reward, a process that depends, at least in part, on orbital and medial prefrontal cortex as well as dorsal striatum.     

Temporal correlation based learning in neuron models

We study a learning rule based upon the temporal correlation (weighted by a learning kernel) between incoming spikes and the internal state of the postsynaptic neuron, building upon previous studies of spike timing dependent synaptic plasticity (Kempter, R., Gerstner, W., van Hemmen, J.L., Wagner, H., 1998. Extracting Oscillations: Neuronal coincidence detection with noisy periodic spike input. Neural computation 10, 1987–2017; Kempter, R., Gerstner, W., van Hemmen, J.L., 1999. Hebbian learning and spiking neurons. Physical Reviewm E59, 4498–4514; van Hemmen, J.L., 2001. Theory of synaptic plasticity. In: Moss, F., Gielen, S. (Eds.), Handbook of biological physics. vol. 4, Neuro Informatics, neural modelling, Elsevier, Amsterdam, pp. 771–823. Our learning rule for the synaptic weight w _ij is where the t _j,μ are the arrival times of spikes from the presynaptic neuron j and the function u(t) describes the state of the postsynaptic neuron i. Thus, the spike-triggered average contained in the inner integral is weighted by a kernel Γ(s), the learning window, positive for negative, negative for positive values of the time difference s between post- and presynaptic activity. An antisymmetry assumption for the learning window enables us to derive analytical expressions for a general class of neuron models and to study the changes in input-output relationships following from synaptic weight changes. This is a genuinely non-linear effect (Song, S., Miller, K., Abbott, L., 2000. Competitive Hebbian learning through spike timing dependent synaptic plasticity. Nature Neuroscience 3, 919–926).     

Vicarious learning from human models in monkeys

We examined whether monkeys can learn by observing a human model, through vicarious learning. Two monkeys observed a human model demonstrating an object-reward association and consuming food found underneath an object. The monkeys observed human models as they solved more than 30 learning problems. For each problem, the human models made a choice between two objects, one of which concealed a piece of apple. In the test phase afterwards, the monkeys made a choice of their own. Learning was apparent from the first trial of the test phase, confirming the ability of monkeys to learn by vicarious observation of human models.     

Temporal prediction errors in a passive learning task activate human striatum

Functional MRI experiments in human subjects strongly suggest that the striatum participates in processing information about the predictability of rewarding stimuli. However, stimuli can be unpredictable in character (what stimulus arrives next), unpredictable in time (when the stimulus arrives), and unpredictable in amount (how much arrives). These variables have not been dissociated in previous imaging work in humans, thus conflating possible interpretations of the kinds of expectation errors driving the measured brain responses. Using a passive conditioning task and fMRI in human subjects, we show that positive and negative prediction errors in reward delivery time correlate with BOLD changes in human striatum, with the strongest activation lateralized to the left putamen. For the negative prediction error, the brain response was elicited by expectations only and not by stimuli presented directly; that is, we measured the brain response to nothing delivered (juice expected but not delivered) contrasted with nothing delivered (nothing expected).     

Semi-supervised analysis of human brain tumours from partially labeled MRS information, using manifold learning models

Medical diagnosis can often be understood as a classification problem. In oncology, this typically involves differentiating between tumour types and grades, or some type of discrete outcome prediction. From the viewpoint of computer-based medical decision support, this classification requires the availability of accurate diagnoses of past cases as training target examples. The availability of such labeled databases is scarce in most areas of oncology, and especially so in neuro-oncology. In such context, semi-supervised learning oriented towards classification can be a sensible data modeling choice. In this study, semi-supervised variants of Generative Topographic Mapping, a model of the manifold learning family, are applied to two neuro-oncology problems: the diagnostic discrimination between different brain tumour pathologies, and the prediction of outcomes for a specific type of aggressive brain tumours. Their performance compared favorably with those of the alternative Laplacian Eigenmaps and Semi-Supervised SVM for Manifold Learning models in most of the experiments.     

Hedging your bets by learning reward correlations in the human brain

Human subjects are proficient at tracking the mean and variance of rewards and updating these via prediction errors. Here, we addressed whether humans can also learn about higher-order relationships between distinct environmental outcomes, a defining ecological feature of contexts where multiple sources of rewards are available. By manipulating the degree to which distinct outcomes are correlated, we show that subjects implemented an explicit model-based strategy to learn the associated outcome correlations and were adept in using that information to dynamically adjust their choices in a task that required a minimization of outcome variance. Importantly, the experimentally generated outcome correlations were explicitly represented neuronally in right midinsula with a learning prediction error signal expressed in rostral anterior cingulate cortex. Thus, our data show that the human brain represents higher-order correlation structures between rewards, a core adaptive ability whose immediate benefit is optimized sampling.     

Hierarchical models in the brain

This paper describes a general model that subsumes many parametric models for continuous data. The model comprises hidden layers of state-space or dynamic causal models, arranged so that the output of one provides input to another. The ensuing hierarchy furnishes a model for many types of data, of arbitrary complexity. Special cases range from the general linear model for static data to generalised convolution models, with system noise, for nonlinear time-series analysis. Crucially, all of these models can be inverted using exactly the same scheme, namely, dynamic expectation maximization. This means that a single model and optimisation scheme can be used to invert a wide range of models. We present the model and a brief review of its inversion to disclose the relationships among, apparently, diverse generative models of empirical data. We then show that this inversion can be formulated as a simple neural network and may provide a useful metaphor for inference and learning in the brain.     

From spatial-data to 3D models of the developing human brain

Visualisation and interpretation of gene expression data have been crucial to advances in our understanding of mechanisms underlying early brain development. As most developmental processes involve complex changes in size, shape and structure, spatial-data can most readily provide information at multiple levels (cell type, cell location in relation to tissue organisation or body axes, etc.), that can be related to these complex changes. Although three-dimensional (3D) spatial-data are ideal, the restricted availability of suitable tissues makes it difficult to generate these for genes expressed at early human fetal stages. Mapping gene expression data to representative 3D models facilitates combinatorial analysis of multiple expression patterns but does not overcome the problems of sparsely sampled data in time and space. Here we describe software that allows 3D domains to be reconstructed by interpolating between sparse 2D gene expression patterns that have been mapped to 3D representative models of corresponding human developmental stages. A set of procedures are proposed to infer expression domains in these gaps. The procedures, which are connected in a serial way, include components clustering, components tracking, shape matching and points interpolation. Each procedure consists of a graphical user interface and a set of algorithms. Results on exemplar gene data are provided.     

Temporal learning in the giant tropical ant, Paraponera clavata

ABSTRACT. Our findings suggest that the capacity for temporal learning in ants may be associated with nectivory. We tested for the existence of time and location learning in the nectivorous tropical ant, Paraponera clavata (F.). Artificial nectar sources were provided at fixed times and locations to P. clavata workers for 5 days. Ants appeared at the feeding location at the appropriate time on the first day that no reward was provided. By the third day without reward there was no difference in visitation between the feeding site and a control site equidistant from the colony. Most ants arrived at the feeding sites within 30 min of the training time, indicating that the precision of ant time sense is similar to that previously documented for honeybees.     

Evolutionarily stable strategies in differential and difference equation models

Summary A definition for an evolutionarily stable strategy (ESS) is given which is applicable to a general differential equation population model and two difference equation analogs. With the introduction of a fitnessgenerating function, it is possible to develop necessary conditions for the determination of an ESS for each of these systems. In most situations, an ESS for one system will also be an ESS for the other. Necessary conditions for an ESS are obtained. Under certain restrictions, they are shown to be valid, even under an unstable equilibrium in population density. the results are illustrated with an example which has the same ESS solution whether a continuous or discrete model is used. The behavior of the ESS for the discrete model is then examined under unstable equilibrium conditions in population density.     

Cathepsin H activity in the human brain and human brain neoplasms

The human brain cathepsin H is shown to be a specific cysteine aminopeptidase with the optimum activity at pH 6.0. Human brain tumours of neuroectodermal (astrocytomas and glioblastomas) and epithelial (meningiomas) origin were used to study the cathepsin H activity in the malignant brain tissue. A significant increase in the aminopeptidase cathepsin H activity was found in malignant human brain tumours as compared to benign tumours and normal brain tissues.     

Generative models of rich clubs in Hebbian neuronal networks and large-scale human brain networks

Rich clubs arise when nodes that are ‘rich’ in connections also form an elite, densely connected ‘club’. In brain networks, rich clubs incur high physical connection costs but also appear to be especially valuable to brain function. However, little is known about the selection pressures that drive their formation. Here, we take two complementary approaches to this question: firstly we show, using generative modelling, that the emergence of rich clubs in large-scale human brain networks can be driven by an economic trade-off between connection costs and a second, competing topological term. Secondly we show, using simulated neural networks, that Hebbian learning rules also drive the emergence of rich clubs at the microscopic level, and that the prominence of these features increases with learning time. These results suggest that Hebbian learning may provide a neuronal mechanism for the selection of complex features such as rich clubs. The neural networks that we investigate are explicitly Hebbian, and we argue that the topological term in our model of large-scale brain connectivity may represent an analogous connection rule. This putative link between learning and rich clubs is also consistent with predictions that integrative aspects of brain network organization are especially important for adaptive behaviour.     

Biases in signal evolution: learning makes a difference

It is now well established that signal receivers have a key role in the evolution of animal communication: the suite of sensory and cognitive processes by which animals perceive and learn about their environment can have a significant impact on signal design. A crucial property of these information-processing mechanisms is the emergence of 'receiver bias' in the behavioural responses to signals. Whereas most research has focussed on receiver biases in the sensory system, more recent studies show that biases can also arise from learning about signals. Here, we highlight how learning-based biases can arise, and how these differ from biases emerging from sensory systems in their impact on signal evolution.     

Infant learning: music and the baby brain

When it comes to listening to music, infants literally have a more open mind than their parents. Studies which investigate listening behaviour of babies and adults have shown that, as we learn to discriminate the musical sounds in our own environment, we become less sensitive to those of other cultures.     

Temporal trends in the discovery of human viruses

On average, more than two new species of human virus are reported every year. We constructed the cumulative species discovery curve for human viruses going back to 1901. We fitted a statistical model to these data; the shape of the curve strongly suggests that the process of virus discovery is far from complete. We generated a 95% credible interval for the pool of as yet undiscovered virus species of 38-562. We extrapolated the curve and generated an estimate of 10-40 new species to be discovered by 2020. Although we cannot predict the level of health threat that these new viruses will present, we conclude that novel virus species must be anticipated in public health planning. More systematic virus discovery programmes, covering both humans and potential animal reservoirs of human viruses, should be considered.     

Temporal orientation in patients with brain disease.

A brief schedule of questions, based on empirically derived norms, disclosed significant inaccuracy in temporal orientation in 24% of a groups of nonaphasic patients with brain disease. Conventional neurological examination detected temporal disorientation in only 15%. Patients with bilateral lesions showed a higher frequency of temporal disorientation than did those with inilateral lesions. In a group of 15 aphasic patients, 5 (33%) showed evidence of temporal disorientation according to the test while only 3 (20%) were detected on clinical examination. It is suggested that a schedule of questions scored on the basis of empirically derived norms should be incorporated into the neurological examination to assess temporal orientation.