Model-based influences on humans' choices and striatal prediction errors

The mesostriatal dopamine system is prominently implicated in model-free reinforcement learning, with fMRI BOLD signals in ventral striatum notably covarying with model-free prediction errors. However, latent learning and devaluation studies show that behavior also shows hallmarks of model-based planning, and the interaction between model-based and model-free values, prediction errors, and preferences is underexplored. We designed a multistep decision task in which model-based and model-free influences on human choice behavior could be distinguished. By showing that choices reflected both influences we could then test the purity of the ventral striatal BOLD signal as a model-free report. Contrary to expectations, the signal reflected both model-free and model-based predictions in proportions matching those that best explained choice behavior. These results challenge the notion of a separate model-free learner and suggest a more integrated computational architecture for high-level human decision-making.     

Some capabilities for model-based and model-free linkage analysis using the program package S.A.G.E. (Statistical Analysis for Genetic Epidemiology)

For both model-free and model-based linkage analysis the S.A.G.E. (Statistical Analysis for Genetic Epidemiology) program package has some unique capabilities in analyzing both continuous traits and binary traits with variable age of onset. Here we highlight model-based linkage analysis of a quantitative trait (plasma dopamine β hydroxylase) that is known to be largely determined by monogenic inheritance, using a prior segregation analysis to produce the best fitting model for the trait. For a binary trait with variable age of onset (schizophrenia), we illustrate how using age of onset information to obtain a quantitative susceptibility trait leads to more statistically significant linkage signals, suggesting better power.     

Model-Based Reasoning in Humans Becomes Automatic with Training

Model-based and model-free reinforcement learning (RL) have been suggested as algorithmic realizations of goal-directed and habitual action strategies. Model-based RL is more flexible than model-free but requires sophisticated calculations using a learnt model of the world. This has led model-based RL to be identified with slow, deliberative processing, and model-free RL with fast, automatic processing. In support of this distinction, it has recently been shown that model-based reasoning is impaired by placing subjects under cognitive load—a hallmark of non-automaticity. Here, using the same task, we show that cognitive load does not impair model-based reasoning if subjects receive prior training on the task. This finding is replicated across two studies and a variety of analysis methods. Thus, task familiarity permits use of model-based reasoning in parallel with other cognitive demands. The ability to deploy model-based reasoning in an automatic, parallelizable fashion has widespread theoretical implications, particularly for the learning and execution of complex behaviors. It also suggests a range of important failure modes in psychiatric disorders.     

Simple Plans or Sophisticated Habits? State,Transition and Learning Interactions in the Two-Step Task

The recently developed ‘two-step’ behavioural task promises to differentiate model-based from model-free reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted its widespread adoption. Here, we analyse the interactions between a range of different strategies and the structure of transitions and outcomes in order to examine constraints on what can be learned from behavioural performance. The task involves a trade-off between the need for stochasticity, to allow strategies to be discriminated, and a need for determinism, so that it is worth subjects’ investment of effort to exploit the contingencies optimally. We show through simulation that under certain conditions model-free strategies can masquerade as being model-based. We first show that seemingly innocuous modifications to the task structure can induce correlations between action values at the start of the trial and the subsequent trial events in such a way that analysis based on comparing successive trials can lead to erroneous conclusions. We confirm the power of a suggested correction to the analysis that can alleviate this problem. We then consider model-free reinforcement learning strategies that exploit correlations between where rewards are obtained and which actions have high expected value. These generate behaviour that appears model-based under these, and also more sophisticated, analyses. Exploiting the full potential of the two-step task as a tool for behavioural neuroscience requires an understanding of these issues.     

The role of parametric linkage methods in complex trait analyses using microsatellites

Many investigators of complexly inherited familial traits bypass classical segregation analysis to perform model-free genome-wide linkage scans. Because model-based or parametric linkage analysis may be the most powerful means to localize genes when a model can be approximated, model-free statistics may result in a loss of power to detect linkage. We performed limited segregation analyses on the electrophysiological measurements that have been collected for the Collaborative Study on the Genetics of Alcoholism. The resulting models are used in whole-genome scans. Four genomic regions provided a model-based LOD > 2 and only 3 of these were detected (p < 0.05) by a model-free approach. We conclude that parametric methods, using even over-simplified models of complex phenotypes, may complement nonparametric methods and decrease false positives.     

Effects of subclinical depression on prefrontal–striatal model-based and model-free learning

Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal–striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.     

Linkage analysis in the presence of errors IV: joint pseudomarker analysis of linkage and/or linkage disequilibrium on a mixture of pedigrees and singletons when the mode of inheritance cannot be accurately specified

There is a lot of confusion in the literature about the "differences" between "model-based" and "model-free" methods and about which approach is better suited for detection of the genes predisposing to complex multifactorial phenotypes. By starting from first principles, we demonstrate that the differences between the two approaches have more to do with study design than statistical analysis. When simple data structures are repeatedly ascertained, no assumptions about the genotype-phenotype relationship need to be made for the analysis to be powerful, since simple data structures admit only a small number of df. When more complicated and/or heterogeneous data structures are ascertained, however, the number of df in the underlying probability model is too large to have a powerful, truly "model-free" test. So-called "model-free" methods typically simplify the underlying probability model by implicitly assuming that, in some sense, all meioses connecting two affected individuals are informative for linkage with identical probability and that the affected individuals in a pedigree share as many disease-predisposing alleles as possible. By contrast, "model-based" methods add structure to the underlying parameter space by making assumptions about the genotype-phenotype relationship, making it possible to probabilistically assign disease-locus genotypes to all individuals in the data set on the basis of the observed phenotypes. In this study, we demonstrate the equivalence of these two approaches in a variety of situations and exploit this equivalence to develop more powerful and efficient likelihood-based analogues of "model-free" tests of linkage and/or linkage disequilibrium. Through the use of a "pseudomarker" locus to structure the space of observations, sib-pairs, triads, and singletons can be analyzed jointly, which will lead to tests that are more well-behaved, efficient, and powerful than traditional "model-free" tests such as the affected sib-pair, transmission/disequilibrium, haplotype relative risk, and case-control tests. Also described is an extension of this approach to large pedigrees, which, in practice, is equivalent to affected relative-pair analysis. The proposed methods are equally applicable to two-point and multipoint analysis (using complex-valued recombination fractions).     

Model based planners reflect on their model-free propensities

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.     

Transcranial Direct Current Stimulation of Right Dorsolateral Prefrontal Cortex Does Not Affect Model-Based or Model-Free Reinforcement Learning in Humans

There is broad consensus that the prefrontal cortex supports goal-directed, model-based decision-making. Consistent with this, we have recently shown that model-based control can be impaired through transcranial magnetic stimulation of right dorsolateral prefrontal cortex in humans. We hypothesized that an enhancement of model-based control might be achieved by anodal transcranial direct current stimulation of the same region. We tested 22 healthy adult human participants in a within-subject, double-blind design in which participants were given Active or Sham stimulation over two sessions. We show Active stimulation had no effect on model-based control or on model-free (‘habitual’) control compared to Sham stimulation. These null effects are substantiated by a power analysis, which suggests that our study had at least 60% power to detect a true effect, and by a Bayesian model comparison, which favors a model of the data that assumes stimulation had no effect over models that assume stimulation had an effect on behavioral control. Although we cannot entirely exclude more trivial explanations for our null effect, for example related to (faults in) our experimental setup, these data suggest that anodal transcranial direct current stimulation over right dorsolateral prefrontal cortex does not improve model-based control, despite existing evidence that transcranial magnetic stimulation can disrupt such control in the same brain region.     

Rethinking motor learning and savings in adaptation paradigms: model-free memory for successful actions combines with internal models

Although motor learning is likely to involve multiple processes, phenomena observed in error-based motor learning paradigms tend to be conceptualized in terms of only a single process: adaptation, which occurs through updating an internal model. Here we argue that fundamental phenomena like movement direction biases, savings (faster relearning), and interference do not relate to adaptation but instead are attributable to two additional learning processes that can be characterized as model-free: use-dependent plasticity and operant reinforcement. Although usually "hidden" behind adaptation, we demonstrate, with modified visuomotor rotation paradigms, that these distinct model-based and model-free processes combine to learn an error-based motor task. (1) Adaptation of an internal model channels movements toward successful error reduction in visual space. (2) Repetition of the newly adapted movement induces directional biases toward the?repeated movement. (3) Operant reinforcement through association of the adapted movement with successful error reduction is responsible for savings.     

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.     

Impact of network topology on inference of synaptic connectivity from multi-neuronal spike data simulated by a large-scale cortical network model

Many mechanisms of neural processing rely critically upon the synaptic connectivity between neurons. As our ability to simultaneously record from large populations of neurons expands, the ability to infer network connectivity from this data has become a major goal of computational neuroscience. To address this issue, we employed several different methods to infer synaptic connections from simulated spike data from a realistic local cortical network model. This approach allowed us to directly compare the accuracy of different methods in predicting synaptic connectivity. We compared the performance of model-free (coherence measure and transfer entropy) and model-based (coupled escape rate model) methods of connectivity inference, applying those methods to the simulated spike data from the model networks with different network topologies. Our results indicate that the accuracy of the inferred connectivity was higher for highly clustered, near regular, or small-world networks, while accuracy was lower for random networks, irrespective of which analysis method was employed. Among the employed methods, the model-based method performed best. This model performed with higher accuracy, was less sensitive to threshold changes, and required less data to make an accurate assessment of connectivity. Given that cortical connectivity tends to be highly clustered, our results outline a powerful analytical tool for inferring local synaptic connectivity from observations of spontaneous activity.     

Set theory formulation of the model-free problem and the diffusion seeded model-free paradigm

Model-free analysis of NMR relaxation data, which describes the motion of individual atoms, is a problem intricately linked to the Brownian rotational diffusion of the macromolecule. The diffusion tensor parameters strongly influence the optimisation of the various model-free models and the subsequent model selection between them. Finding the optimal model of the dynamics of the system among the numerous diffusion and model-free models is hence quite complex. Using set theory, the entirety of this global problem has been encapsulated by the universal set Ll, and its resolution mathematically formulated as the universal solution Ll. Ever since the original Lipari and Szabo papers the model-free dynamics of a molecule has most often been solved by initially estimating the diffusion tensor. The model-free models which depend on the diffusion parameter values are then optimised and the best model is chosen to represent the dynamics of the residue. Finally, the global model of all diffusion and model-free parameters is optimised. These steps are repeated until convergence. For simplicity this approach to Ll will be labelled the diffusion seeded model-free paradigm. Although this technique suffers from a number of problems many have been solved. All aspects of the diffusion seeded paradigm and its consequences, together with a few alternatives to the paradigm, will be reviewed through the use of set notation.     

A novel model-free approach for reconstruction of time-delayed gene regulatory networks

Many important biological processes (e.g. cellular differentiation during development, aging, disease etiology etc.) are very unlikely controlled by a single gene instead by the underlying complex regulatory interactions between thousands of genes within …     

To Control False Positives in Gene-Gene Interaction Analysis: Two Novel Conditional Entropy-Based Approaches

Genome-wide analysis of gene-gene interactions has been recognized as a powerful avenue to identify the missing genetic components that can not be detected by using current single-point association analysis. Recently, several model-free methods (e.g. the commonly used information based metrics and several logistic regression-based metrics) were developed for detecting non-linear dependence between genetic loci, but they are potentially at the risk of inflated false positive error, in particular when the main effects at one or both loci are salient. In this study, we proposed two conditional entropy-based metrics to challenge this limitation. Extensive simulations demonstrated that the two proposed metrics, provided the disease is rare, could maintain consistently correct false positive rate. In the scenarios for a common disease, our proposed metrics achieved better or comparable control of false positive error, compared to four previously proposed model-free metrics. In terms of power, our methods outperformed several competing metrics in a range of common disease models. Furthermore, in real data analyses, both metrics succeeded in detecting interactions and were competitive with the originally reported results or the logistic regression approaches. In conclusion, the proposed conditional entropy-based metrics are promising as alternatives to current model-based approaches for detecting genuine epistatic effects.     

The algorithmic anatomy of model-based evaluation

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.     

基于单核苷酸多态性的基因互作分析方法学进展

基于单核苷酸多态性的关联分析已成为当前解析人类常见复杂疾病遗传机制的重要手段之一, 然而, 目前普遍使用的单位点分析策略仅能发现部分单独效应显著的易感SNP位点, 因此遗漏了重要的遗传力组分——基因上位效应或联合效应。识别全基因组多基因间复杂的互作关系已成为全面解析复杂疾病致病分子机制必不可少的一项任务。已有很多方法被应用于全基因组交互作用分析, 加深了人类对复杂疾病遗传机制的进一步认识。基于各类方法的理论基础及算法的异同, 文章对目前应用较为广泛的基于遗传互作模型的方法、不基于互作模型的方法和数据挖掘类算法3类方法进行了系统地评述, 着重介绍了这些方法的主要思想、实现过程及应用中的注意事项等, 并指出开展大规模全基因组范围互作检测面临的问题, 以期能为相关领域的研究者提供方法学参考。     

Model-free model elimination: A new step in the model-free dynamic analysis of NMR relaxation data

Model-free analysis is a technique commonly used within the field of NMR spectroscopy to extract atomic resolution, interpretable dynamic information on multiple timescales from the R ₁, R ₂, and steady state NOE. Model-free approaches employ two disparate areas of data analysis, the discipline of mathematical optimisation, specifically the minimisation of a χ² function, and the statistical field of model selection. By searching through a large number of model-free minimisations, which were setup using synthetic relaxation data whereby the true underlying dynamics is known, certain model-free models have been identified to, at times, fail. This has been characterised as either the internal correlation times, τ _e , τ _f , or τ _s , or the global correlation time parameter, local τ _m , heading towards infinity, the result being that the final parameter values are far from the true values. In a number of cases the minimised χ² value of the failed model is significantly lower than that of all other models and, hence, will be the model which is chosen by model selection techniques. If these models are not removed prior to model selection the final model-free results could be far from the truth. By implementing a series of empirical rules involving inequalities these models can be specifically isolated and removed. Model-free analysis should therefore consist of three distinct steps: model-free minimisation, model-free model elimination, and finally model-free model selection. Failure has also been identified to affect the individual Monte Carlo simulations used within error analysis. Each simulation involves an independent randomised relaxation data set and model-free minimisation, thus simulations suffer from exactly the same types of failure as model-free models. Therefore, to prevent these outliers from causing a significant overestimation of the errors the failed Monte Carlo simulations need to be culled prior to calculating the parameter standard deviations.