Experience-dependent modulation of tonotopic neural responses in human auditory cortex.

Experience-dependent plasticity of receptive fields in the auditory cortex has been demonstrated by electrophysiological experiments in animals. In the present study we used PET neuroimaging to measure regional brain activity in volunteer human subjects during discriminatory classical conditioning of high (8000 Hz) or low (200 Hz) frequency tones by an aversive 100 dB white noise burst. Conditioning-related, frequency-specific modulation of tonotopic neural responses in the auditory cortex was observed. The modulated regions of the auditory cortex positively covaried with activity in the amygdala, basal forebrain and orbitofrontal cortex, and showed context-specific functional interactions with the medial geniculate nucleus. These results accord with animal single-unit data and support neurobiological models of auditory conditioning and value-dependent neural selection.     

Asymmetric and adaptive reward coding via normalized reinforcement learning

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.     

Explicit neural signals reflecting reward uncertainty

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.     

Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology

Neurons in a small number of brain structures detect rewards and reward-predicting stimuli and are active during the expectation of predictable food and liquid rewards. These neurons code the reward information according to basic terms of various behavioural theories that seek to explain reward-directed learning, approach behaviour and decision-making. The involved brain structures include groups of dopamine neurons, the striatum including the nucleus accumbens, the orbitofrontal cortex and the amygdala. The reward information is fed to brain structures involved in decision-making and organisation of behaviour, such as the dorsolateral prefrontal cortex and possibly the parietal cortex. The neural coding of basic reward terms derived from formal theories puts the neurophysiological investigation of reward mechanisms on firm conceptual grounds and provides neural correlates for the function of rewards in learning, approach behaviour and decision-making.     

Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation

We monitored single-neuron activity in the orbitofrontal cortex of rats performing a time-discounting task in which the spatial location of the reward predicted whether the delay preceding reward delivery would be short or long. We found that rewards delivered after a short delay elicited a stronger neuronal response than those delivered after a long delay in most neurons. Activity in these neurons was not influenced by reward size when delays were held constant. This was also true for a minority of neurons that exhibited sustained increases in firing in anticipation of delayed reward. Thus, encoding of time-discounted rewards in orbitofrontal cortex is independent of the encoding of absolute reward value. These results are contrary to the proposal that orbitofrontal neurons signal the value of delayed rewards in a common currency and instead suggest alternative proposals for the role this region plays in guiding responses for delayed versus immediate rewards.     

Subliminal instrumental conditioning demonstrated in the human brain

How the brain uses success and failure to optimize future decisions is a long-standing question in neuroscience. One computational solution involves updating the values of context-action associations in proportion to a reward prediction error. Previous evidence suggests that such computations are expressed in the striatum and, as they are cognitively impenetrable, represent an unconscious learning mechanism. Here, we formally test this by studying instrumental conditioning in a situation where we masked contextual cues, such that they were not consciously perceived. Behavioral data showed that subjects nonetheless developed a significant propensity to choose cues associated with monetary rewards relative to punishments. Functional neuroimaging revealed that during conditioning cue values and prediction errors, generated from a computational model, both correlated with activity in ventral striatum. We conclude that, even without conscious processing of contextual cues, our brain can learn their reward value and use them to provide a bias on decision making.     

Neural ensemble coding of satiety states

The motivation to start or terminate a meal involves the continual updating of information on current body status by central gustatory and reward systems. Previous electrophysiological and neuroimaging investigations revealed region-specific decreases in activity as the subject's state transitions from hunger to satiety. By implanting bundles of microelectrodes in the lateral hypothalamus, orbitofrontal cortex, insular cortex, and amygdala of hungry rats that voluntarily eat to satiety, we have measured the behavior of neuronal populations through the different phases of a complete feeding cycle (hunger-satiety-hunger). Our data show that while most satiety-sensitive units preferentially responded to a unique hunger phase within a cycle, neuronal populations integrated single-unit information in order to reflect the animal's motivational state across the entire cycle, with higher activity levels during the hunger phases. This distributed population code might constitute a neural mechanism underlying meal initiation under different metabolic states.     

Birdsong: models and mechanisms

Recent studies have provided important information concerning the neural signals that subserve vocal learning in songbirds: advanced signal processing techniques are beginning to clarify the behavioral trajectories followed by developing birds; single-unit physiology in behaving animals is providing important clues about sensory and motor representations during learning; in vitro whole-cell recordings are revealing patterns of synaptic communication; and experimental alterations in song behavior have advanced our understanding of specific structure-function relationships. The construction of theoretical and computational models will be crucial in integrating such disparate experimental results.     

The neural basis of following advice

Learning by following explicit advice is fundamental for human cultural evolution, yet the neurobiology of adaptive social learning is largely unknown. Here, we used simulations to analyze the adaptive value of social learning mechanisms, computational modeling of behavioral data to describe cognitive mechanisms involved in social learning, and model-based functional magnetic resonance imaging (fMRI) to identify the neurobiological basis of following advice. One-time advice received before learning had a sustained influence on people's learning processes. This was best explained by social learning mechanisms implementing a more positive evaluation of the outcomes from recommended options. Computer simulations showed that this "outcome-bonus" accumulates more rewards than an alternative mechanism implementing higher initial reward expectation for recommended options. fMRI results revealed a neural outcome-bonus signal in the septal area and the left caudate. This neural signal coded rewards in the absence of advice, and crucially, it signaled greater positive rewards for positive and negative feedback after recommended rather than after non-recommended choices. Hence, our results indicate that following advice is intrinsically rewarding. A positive correlation between the model's outcome-bonus parameter and amygdala activity after positive feedback directly relates the computational model to brain activity. These results advance the understanding of social learning by providing a neurobiological account for adaptive learning from advice.     

Spectrally specific temporal analyses of spike-train responses to complex sounds: A unifying framework

Significant scientific and translational questions remain in auditory neuroscience surrounding the neural correlates of perception. Relating perceptual and neural data collected from humans can be useful; however, human-based neural data are typically limited to evoked far-field responses, which lack anatomical and physiological specificity. Laboratory-controlled preclinical animal models offer the advantage of comparing single-unit and evoked responses from the same animals. This ability provides opportunities to develop invaluable insight into proper interpretations of evoked responses, which benefits both basic-science studies of neural mechanisms and translational applications, e.g., diagnostic development. However, these comparisons have been limited by a disconnect between the types of spectrotemporal analyses used with single-unit spike trains and evoked responses, which results because these response types are fundamentally different (point-process versus continuous-valued signals) even though the responses themselves are related. Here, we describe a unifying framework to study temporal coding of complex sounds that allows spike-train and evoked-response data to be analyzed and compared using the same advanced signal-processing techniques. The framework uses a set of peristimulus-time histograms computed from single-unit spike trains in response to polarity-alternating stimuli to allow advanced spectral analyses of both slow (envelope) and rapid (temporal fine structure) response components. Demonstrated benefits include: (1) novel spectrally specific temporal-coding measures that are less confounded by distortions due to hair-cell transduction, synaptic rectification, and neural stochasticity compared to previous metrics, e.g., the correlogram peak-height, (2) spectrally specific analyses of spike-train modulation coding (magnitude and phase), which can be directly compared to modern perceptually based models of speech intelligibility (e.g., that depend on modulation filter banks), and (3) superior spectral resolution in analyzing the neural representation of nonstationary sounds, such as speech and music. This unifying framework significantly expands the potential of preclinical animal models to advance our understanding of the physiological correlates of perceptual deficits in real-world listening following sensorineural hearing loss.     

The ion channel inverse problem: neuroinformatics meets biophysics

Ion channels are the building blocks of the information processing capability of neurons: any realistic computational model of a neuron must include reliable and effective ion channel components. Sophisticated statistical and computational tools have been developed to study the ion channel structure–function relationship, but this work is rarely incorporated into the models used for single neurons or small networks. The disjunction is partly a matter of convention. Structure–function studies typically use a single Markov model for the whole channel whereas until recently whole-cell modeling software has focused on serial, independent, two-state subunits that can be represented by the Hodgkin–Huxley equations. More fundamentally, there is a difference in purpose that prevents models being easily reused. Biophysical models are typically developed to study one particular aspect of channel gating in detail, whereas neural modelers require broad coverage of the entire range of channel behavior that is often best achieved with approximate representations that omit structural features that cannot be adequately constrained. To bridge the gap so that more recent channel data can be used in neural models requires new computational infrastructure for bringing together diverse sources of data to arrive at best-fit models for whole-cell modeling. We review the current state of channel modeling and explore the developments needed for its conclusions to be integrated into whole-cell modeling.     

Frontal cortex and reward-guided learning and decision-making

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.     

The Role of Inhibition in a Computational Model of an Auditory Cortical Neuron during the Encoding of Temporal Information

In auditory cortex, temporal information within a sound is represented by two complementary neural codes: a temporal representation based on stimulus-locked firing and a rate representation, where discharge rate co-varies with the timing between acoustic events but lacks a stimulus-synchronized response. Using a computational neuronal model, we find that stimulus-locked responses are generated when sound-evoked excitation is combined with strong, delayed inhibition. In contrast to this, a non-synchronized rate representation is generated when the net excitation evoked by the sound is weak, which occurs when excitation is coincident and balanced with inhibition. Using single-unit recordings from awake marmosets (Callithrix jacchus), we validate several model predictions, including differences in the temporal fidelity, discharge rates and temporal dynamics of stimulus-evoked responses between neurons with rate and temporal representations. Together these data suggest that feedforward inhibition provides a parsimonious explanation of the neural coding dichotomy observed in auditory cortex.     

Accounting for Negative Automaintenance in Pigeons: A Dual Learning Systems Approach and Factored Representations

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.     

What Do I Want and When Do I Want It: Brain Correlates of Decisions Made for Self and Other

A number of recent functional Magnetic Resonance Imaging (fMRI) studies on intertemporal choice behavior have demonstrated that so-called emotion- and reward-related brain areas are preferentially activated by decisions involving immediately available (but smaller) rewards as compared to (larger) delayed rewards. This pattern of activation was not seen, however, when intertemporal choices were made for another (unknown) individual, which speaks to that activation having been triggered by self-relatedness. In the present fMRI study, we investigated the brain correlates of individuals who passively observed intertemporal choices being made either for themselves or for an unknown person. We found higher activation within the ventral striatum, medial prefrontal and orbitofrontal cortex, pregenual anterior cingulate cortex, and posterior cingulate cortex when an immediate reward was possible for the observer herself, which is in line with findings from studies in which individuals actively chose immediately available rewards. Additionally, activation in the dorsal anterior cingulate cortex, posterior cingulate cortex, and precuneus was higher for choices that included immediate options than for choices that offered only delayed options, irrespective of who was to be the beneficiary. These results indicate that (1) the activations found in active intertemporal decision making are also present when the same decisions are merely observed, thus supporting the assumption that a robust brain network is engaged in immediate gratification; and (2) with immediate rewards, certain brain areas are activated irrespective of whether the observer or another person is the beneficiary of a decision, suggesting that immediacy plays a more general role for neural activation. An explorative analysis of participants’ brain activation corresponding to chosen rewards, further indicates that activation in the aforementioned brain areas depends on the mere presence, availability, or actual reception of immediate rewards.     

A Computational Model Incorporating Neural Stem Cell Dynamics Reproduces Glioma Incidence across the Lifespan in the Human Population

Glioma is the most common form of primary brain tumor. Demographically, the risk of occurrence increases until old age. Here we present a novel computational model to reproduce the probability of glioma incidence across the lifespan. Previous mathematical models explaining glioma incidence are framed in a rather abstract way, and do not directly relate to empirical findings. To decrease this gap between theory and experimental observations, we incorporate recent data on cellular and molecular factors underlying gliomagenesis. Since evidence implicates the adult neural stem cell as the likely cell-of-origin of glioma, we have incorporated empirically-determined estimates of neural stem cell number, cell division rate, mutation rate and oncogenic potential into our model. We demonstrate that our model yields results which match actual demographic data in the human population. In particular, this model accounts for the observed peak incidence of glioma at approximately 80 years of age, without the need to assert differential susceptibility throughout the population. Overall, our model supports the hypothesis that glioma is caused by randomly-occurring oncogenic mutations within the neural stem cell population. Based on this model, we assess the influence of the (experimentally indicated) decrease in the number of neural stem cells and increase of cell division rate during aging. Our model provides multiple testable predictions, and suggests that different temporal sequences of oncogenic mutations can lead to tumorigenesis. Finally, we conclude that four or five oncogenic mutations are sufficient for the formation of glioma.     

OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural networks and its application to the COVID-19 pandemics in Germany

Mathematical models in epidemiology are an indispensable tool to determine the dynamics and important characteristics of infectious diseases. Apart from their scientific merit, these models are often used to inform political decisions and interventional measures during an ongoing outbreak. However, reliably inferring the epidemical dynamics by connecting complex models to real data is still hard and requires either laborious manual parameter fitting or expensive optimization methods which have to be repeated from scratch for every application of a given model. In this work, we address this problem with a novel combination of epidemiological modeling with specialized neural networks. Our approach entails two computational phases: In an initial training phase, a mathematical model describing the epidemic is used as a coach for a neural network, which acquires global knowledge about the full range of possible disease dynamics. In the subsequent inference phase, the trained neural network processes the observed data of an actual outbreak and infers the parameters of the model in order to realistically reproduce the observed dynamics and reliably predict future progression. With its flexible framework, our simulation-based approach is applicable to a variety of epidemiological models. Moreover, since our method is fully Bayesian, it is designed to incorporate all available prior knowledge about plausible parameter values and returns complete joint posterior distributions over these parameters. Application of our method to the early Covid-19 outbreak phase in Germany demonstrates that we are able to obtain reliable probabilistic estimates for important disease characteristics, such as generation time, fraction of undetected infections, likelihood of transmission before symptom onset, and reporting delays using a very moderate amount of real-world observations.     

Modular inverse reinforcement learning for visuomotor behavior

In a large variety of situations one would like to have an expressive and accurate model of observed animal or human behavior. While general purpose mathematical models may capture successfully properties of observed behavior, it is desirable to root models in biological facts. Because of ample empirical evidence for reward-based learning in visuomotor tasks, we use a computational model based on the assumption that the observed agent is balancing the costs and benefits of its behavior to meet its goals. This leads to using the framework of reinforcement learning, which additionally provides well-established algorithms for learning of visuomotor task solutions. To quantify the agent’s goals as rewards implicit in the observed behavior, we propose to use inverse reinforcement learning, which quantifies the agent’s goals as rewards implicit in the observed behavior. Based on the assumption of a modular cognitive architecture, we introduce a modular inverse reinforcement learning algorithm that estimates the relative reward contributions of the component tasks in navigation, consisting of following a path while avoiding obstacles and approaching targets. It is shown how to recover the component reward weights for individual tasks and that variability in observed trajectories can be explained succinctly through behavioral goals. It is demonstrated through simulations that good estimates can be obtained already with modest amounts of observation data, which in turn allows the prediction of behavior in novel configurations.     

Religious people discount the future less

The propensity for religious belief and behavior is a universal feature of human societies, but religious practice often imposes substantial costs upon its practitioners. This suggests that during human cultural evolution, the costs associated with religiosity might have been traded off for psychological or social benefits that redounded to fitness on average. One possible benefit of religious belief and behavior, which virtually every world religion extols, is delay of gratification—that is, the ability to forego small rewards available immediately in the interest of obtaining larger rewards that are available only after a time delay. In this study, we found that religious commitment was associated with a tendency to forgo immediate rewards in order to gain larger, future rewards. We also found that this relationship was partially mediated by future time orientation, which is a subjective sense that the future is very close in time and is approaching rapidly. Although the effect sizes of these associations were relatively small in magnitude, they were obtained even when controlling for sex and the Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism).