Active inference and agency: optimal control without cost functions

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.     

Enriching the analysis of genomewide association studies with hierarchical modeling

Genomewide association studies (GWAs) initially investigate hundreds of thousands of single-nucleotide polymorphisms (SNPs), and the most promising SNPs are further evaluated with additional subjects, for replication or a joint analysis. Deciding which SNPs merit follow-up is one of the most crucial aspects of these studies. We present here an approach for selecting the most-promising SNPs that incorporates into a hierarchical model both conventional results and other existing information about the SNPs. The model is developed for general use, its potential value is shown by application, and tools are provided for undertaking hierarchical modeling. By quantitatively harnessing all available information in GWAs, hierarchical modeling may more clearly distinguish true causal variants from noise.     

Sharing detection heterogeneity information among species in community models of occupancy and abundance can strengthen inference

1. The estimation of abundance and distribution and factors governing patterns in these parameters is central to the field of ecology. The continued development of hierarchical models that best utilize available information to inform these processes is a key goal of quantitative ecologists. However, much remains to be learned about simultaneously modeling true abundance, presence, and trajectories of ecological communities.
2. Simultaneous modeling of the population dynamics of multiple species provides an interesting mechanism to examine patterns in community processes and, as we emphasize herein, to improve species‐specific estimates by leveraging detection information among species. Here, we demonstrate a simple but effective approach to share information about observation parameters among species in hierarchical community abundance and occupancy models, where we use shared random effects among species to account for spatiotemporal heterogeneity in detection probability.
3. We demonstrate the efficacy of our modeling approach using simulated abundance data, where we recover well our simulated parameters using N‐mixture models. Our approach substantially increases precision in estimates of abundance compared with models that do not share detection information among species. We then expand this model and apply it to repeated detection/non‐detection data collected on six species of tits (Paridae) breeding at 119 1 km² sampling sites across a P. montanus hybrid zone in northern Switzerland (2004–2020). We find strong impacts of forest cover and elevation on population persistence and colonization in all species. We also demonstrate evidence for interspecific competition on population persistence and colonization probabilities, where the presence of marsh tits reduces population persistence and colonization probability of sympatric willow tits, potentially decreasing gene flow among willow tit subspecies.
4. While conceptually simple, our results have important implications for the future modeling of population abundance, colonization, persistence, and trajectories in community frameworks. We suggest potential extensions of our modeling in this paper and discuss how leveraging data from multiple species can improve model performance and sharpen ecological inference.

     

Observing the observer (II): deciding when to decide

In a companion paper [1], we have presented a generic approach for inferring how subjects make optimal decisions under uncertainty. From a Bayesian decision theoretic perspective, uncertain representations correspond to "posterior" beliefs, which result from integrating (sensory) information with subjective "prior" beliefs. Preferences and goals are encoded through a "loss" (or "utility") function, which measures the cost incurred by making any admissible decision for any given (hidden or unknown) state of the world. By assuming that subjects make optimal decisions on the basis of updated (posterior) beliefs and utility (loss) functions, one can evaluate the likelihood of observed behaviour. In this paper, we describe a concrete implementation of this meta-Bayesian approach (i.e. a Bayesian treatment of Bayesian decision theoretic predictions) and demonstrate its utility by applying it to both simulated and empirical reaction time data from an associative learning task. Here, inter-trial variability in reaction times is modelled as reflecting the dynamics of the subjects' internal recognition process, i.e. the updating of representations (posterior densities) of hidden states over trials while subjects learn probabilistic audio-visual associations. We use this paradigm to demonstrate that our meta-Bayesian framework allows for (i) probabilistic inference on the dynamics of the subject's representation of environmental states, and for (ii) model selection to disambiguate between alternative preferences (loss functions) human subjects could employ when dealing with trade-offs, such as between speed and accuracy. Finally, we illustrate how our approach can be used to quantify subjective beliefs and preferences that underlie inter-individual differences in behaviour.     

A hierarchical modeling framework for multiple observer transect surveys

Ecologists often use multiple observer transect surveys to census animal populations. In addition to animal counts, these surveys produce sequences of detections and non-detections for each observer. When combined with additional data (i.e. covariates such as distance from the transect line), these sequences provide the additional information to estimate absolute abundance when detectability on the transect line is less than one. Although existing analysis approaches for such data have proven extremely useful, they have some limitations. For instance, it is difficult to extrapolate from observed areas to unobserved areas unless a rigorous sampling design is adhered to; it is also difficult to share information across spatial and temporal domains or to accommodate habitat-abundance relationships. In this paper, we introduce a hierarchical modeling framework for multiple observer line transects that removes these limitations. In particular, abundance intensities can be modeled as a function of habitat covariates, making it easier to extrapolate to unsampled areas. Our approach relies on a complete data representation of the state space, where unobserved animals and their covariates are modeled using a reversible jump Markov chain Monte Carlo algorithm. Observer detections are modeled via a bivariate normal distribution on the probit scale, with dependence induced by a distance-dependent correlation parameter. We illustrate performance of our approach with simulated data and on a known population of golf tees. In both cases, we show that our hierarchical modeling approach yields accurate inference about abundance and related parameters. In addition, we obtain accurate inference about population-level covariates (e.g. group size). We recommend that ecologists consider using hierarchical models when analyzing multiple-observer transect data, especially when it is difficult to rigorously follow pre-specified sampling designs. We provide a new R package, hierarchicalDS, to facilitate the building and fitting of these models.     

Dynamic Alignment Models for Neural Coding

Recently, there have been remarkable advances in modeling the relationships between the sensory environment, neuronal responses, and behavior. However, most models cannot encompass variable stimulus-response relationships such as varying response latencies and state or context dependence of the neural code. Here, we consider response modeling as a dynamic alignment problem and model stimulus and response jointly by a mixed pair hidden Markov model (MPH). In MPHs, multiple stimulus-response relationships (e.g., receptive fields) are represented by different states or groups of states in a Markov chain. Each stimulus-response relationship features temporal flexibility, allowing modeling of variable response latencies, including noisy ones. We derive algorithms for learning of MPH parameters and for inference of spike response probabilities. We show that some linear-nonlinear Poisson cascade (LNP) models are a special case of MPHs. We demonstrate the efficiency and usefulness of MPHs in simulations of both jittered and switching spike responses to white noise and natural stimuli. Furthermore, we apply MPHs to extracellular single and multi-unit data recorded in cortical brain areas of singing birds to showcase a novel method for estimating response lag distributions. MPHs allow simultaneous estimation of receptive fields, latency statistics, and hidden state dynamics and so can help to uncover complex stimulus response relationships that are subject to variable timing and involve diverse neural codes.     

Understanding movement data and movement processes: current and emerging directions

Animal movement has been the focus on much theoretical and empirical work in ecology over the last 25 years. By studying the causes and consequences of individual movement, ecologists have gained greater insight into the behavior of individuals and the spatial dynamics of populations at increasingly higher levels of organization. In particular, ecologists have focused on the interaction between individuals and their environment in an effort to understand future impacts from habitat loss and climate change. Tools to examine this interaction have included: fractal analysis, first passage time, Lévy flights, multi‐behavioral analysis, hidden markov models, and state‐space models. Concurrent with the development of movement models has been an increase in the sophistication and availability of hierarchical bayesian models. In this review we bring these two threads together by using hierarchical structures as a framework for reviewing individual models. We synthesize emerging themes in movement ecology, and propose a new hierarchical model for animal movement that builds on these emerging themes. This model moves away from traditional random walks, and instead focuses inference on how moving animals with complex behavior interact with their landscape and make choices about its suitability.     

A Bayesian hierarchical model for analyzing methylated RNA immunoprecipitation sequencing data

Background: The recently emerged technology of methylated RNA immunoprecipitation sequencing (MeRIP-seq) sheds light on the study of RNA epigenetics. This new bioinformatics question calls for effective and robust peaking calling algorithms to detect mRNA methylation sites from MeRIP-seq data. Methods: We propose a Bayesian hierarchical model to detect methylation sites from MeRIP-seq data. Our modeling approach includes several important characteristics. First, it models the zero-inflated and over-dispersed counts by deploying a zero-inflated negative binomial model. Second, it incorporates a hidden Markov model (HMM) to account for the spatial dependency of neighboring read enrichment. Third, our Bayesian inference allows the proposed model to borrow strength in parameter estimation, which greatly improves the model stability when dealing with MeRIP-seq data with a small number of replicates. We use Markov chain Monte Carlo (MCMC) algorithms to simultaneously infer the model parameters in a de novo fashion. The R Shiny demo is available at the authors' website and the R/C++ code is available at https://github.com/liqiwei2000/BaySeqPeak. Results: In simulation studies, the proposed method outperformed the competing methods exomePeak and MeTPeak, especially when an excess of zeros were present in the data. In real MeRIP-seq data analysis, the proposed method identified methylation sites that were more consistent with biological knowledge, and had better spatial resolution compared to the other methods. Conclusions: In this study, we develop a Bayesian hierarchical model to identify methylation peaks in MeRIP-seq data. The proposed method has a competitive edge over existing methods in terms of accuracy, robustness and spatial resolution.     

An Accessible Method for Implementing Hierarchical Models with Spatio-Temporal Abundance Data

A common goal in ecology and wildlife management is to determine the causes of variation in population dynamics over long periods of time and across large spatial scales. Many assumptions must nevertheless be overcome to make appropriate inference about spatio-temporal variation in population dynamics, such as autocorrelation among data points, excess zeros, and observation error in count data. To address these issues, many scientists and statisticians have recommended the use of Bayesian hierarchical models. Unfortunately, hierarchical statistical models remain somewhat difficult to use because of the necessary quantitative background needed to implement them, or because of the computational demands of using Markov Chain Monte Carlo algorithms to estimate parameters. Fortunately, new tools have recently been developed that make it more feasible for wildlife biologists to fit sophisticated hierarchical Bayesian models (i.e., Integrated Nested Laplace Approximation, ‘INLA’). We present a case study using two important game species in North America, the lesser and greater scaup, to demonstrate how INLA can be used to estimate the parameters in a hierarchical model that decouples observation error from process variation, and accounts for unknown sources of excess zeros as well as spatial and temporal dependence in the data. Ultimately, our goal was to make unbiased inference about spatial variation in population trends over time.     

Flexible empirical Bayes models for differential gene expression

MOTIVATION: Inference about differential expression is a typical objective when analyzing gene expression data. Recently, Bayesian hierarchical models have become increasingly popular for this type of problem. The two most common hierarchical models are the hierarchical Gamma-Gamma (GG) and Lognormal-Normal (LNN) models. However, to facilitate inference, some unrealistic assumptions have been made. One such assumption is that of a common coefficient of variation across genes, which can adversely affect the resulting inference. RESULTS: In this paper, we extend both the GG and LNN modeling frameworks to allow for gene-specific variances and propose EM based algorithms for parameter estimation. The proposed methodology is evaluated on three experimental datasets: one cDNA microarray experiment and two Affymetrix spike-in experiments. The two extended models significantly reduce the false positive rate while keeping a high sensitivity when compared to the originals. Finally, using a simulation study we show that the new frameworks are also more robust to model misspecification. AVAILABILITY: The R code for implementing the proposed methodology can be downloaded at http://www.stat.ubc.ca/~c.lo/FEBarrays. SUPPLEMENTARY INFORMATION: The supplementary material is available at http://www.stat.ubc.ca/~c.lo/FEBarrays/supp.pdf.     

Are the numbers adding up? Exploiting discrepancies among complementary population models

Large carnivores are difficult to monitor because they tend to be sparsely distributed, sensitive to human activity, and associated with complex life histories. Consequently, understanding population trend and viability requires conservationists to cope with uncertainty and bias in population data. Joint analysis of combined data sets using multiple models (i.e., integrated population model) can improve inference about mechanisms (e.g., habitat heterogeneity and food distribution) affecting population dynamics. However, unobserved or unobservable processes can also introduce bias and can be difficult to quantify. We developed a Bayesian hierarchical modeling approach for inference on an integrated population model that reconciles annual population counts with recruitment and survival data (i.e., demographic processes). Our modeling framework is flexible and enables a realistic form of population dynamics by fitting separate density-dependent responses for each demographic process. Discrepancies estimated from shared parameters among different model components represent unobserved additions (i.e., recruitment or immigration) or removals (i.e., death or emigration) when annual population counts are reliable. In a case study of gray wolves in Wisconsin (1980–2011), concordant with policy changes, we estimated that a discrepancy of 0% (1980–1995), −2% (1996–2002), and 4% (2003–2011) in the annual mortality rate was needed to explain annual growth rate. Additional mortality in 2003–2011 may reflect density-dependent mechanisms, changes in illegal killing with shifts in wolf management, and nonindependent censoring in survival data. Integrated population models provide insights into unobserved or unobservable processes by quantifying discrepancies among data sets. Our modeling approach is generalizable to many population analysis needs and allows for identifying dynamic differences due to external drivers, such as management or policy changes.     

A Bayesian hierarchical approach for combining case-control and prospective studies

Motivated by the absolute risk predictions required in medical decision making and patient counseling, we propose an approach for the combined analysis of case-control and prospective studies of disease risk factors. The approach is hierarchical to account for parameter heterogeneity among studies and among sampling units of the same study. It is based on modeling the retrospective distribution of the covariates given the disease outcome, a strategy that greatly simplifies both the combination of prospective and retrospective studies and the computation of Bayesian predictions in the hierarchical case-control context. Retrospective modeling differentiates our approach from most current strategies for inference on risk factors, which are based on the assumption of a specific prospective model. To ensure modeling flexibility, we propose using a mixture model for the retrospective distributions of the covariates. This leads to a general nonlinear regression family for the implied prospective likelihood. After introducing and motivating our proposal, we present simple results that highlight its relationship with existing approaches, develop Markov chain Monte Carlo methods for inference and prediction, and present an illustration using ovarian cancer data.     

Complex sequencing rules of birdsong can be explained by simple hidden Markov processes

Complex sequencing rules observed in birdsongs provide an opportunity to investigate the neural mechanism for generating complex sequential behaviors. To relate the findings from studying birdsongs to other sequential behaviors such as human speech and musical performance, it is crucial to characterize the statistical properties of the sequencing rules in birdsongs. However, the properties of the sequencing rules in birdsongs have not yet been fully addressed. In this study, we investigate the statistical properties of the complex birdsong of the Bengalese finch (Lonchura striata var. domestica). Based on manual-annotated syllable labeles, we first show that there are significant higher-order context dependencies in Bengalese finch songs, that is, which syllable appears next depends on more than one previous syllable. We then analyze acoustic features of the song and show that higher-order context dependencies can be explained using first-order hidden state transition dynamics with redundant hidden states. This model corresponds to hidden Markov models (HMMs), well known statistical models with a large range of application for time series modeling. The song annotation with these models with first-order hidden state dynamics agreed well with manual annotation, the score was comparable to that of a second-order HMM, and surpassed the zeroth-order model (the Gaussian mixture model; GMM), which does not use context information. Our results imply that the hierarchical representation with hidden state dynamics may underlie the neural implementation for generating complex behavioral sequences with higher-order dependencies.     

Modeling disease incidence data with spatial and spatio temporal dirichlet process mixtures

Disease incidence or mortality data are typically available as rates or counts for specified regions, collected over time. We propose Bayesian nonparametric spatial modeling approaches to analyze such data. We develop a hierarchical specification using spatial random effects modeled with a Dirichlet process prior. The Dirichlet process is centered around a multivariate normal distribution. This latter distribution arises from a log-Gaussian process model that provides a latent incidence rate surface, followed by block averaging to the areal units determined by the regions in the study. With regard to the resulting posterior predictive inference, the modeling approach is shown to be equivalent to an approach based on block averaging of a spatial Dirichlet process to obtain a prior probability model for the finite dimensional distribution of the spatial random effects. We introduce a dynamic formulation for the spatial random effects to extend the model to spatio-temporal settings. Posterior inference is implemented through Gibbs sampling. We illustrate the methodology with simulated data as well as with a data set on lung cancer incidences for all 88 counties in the state of Ohio over an observation period of 21 years.     

Robust two-stage estimation in hierarchical nonlinear models

Hierarchical models encompass two sources of variation, namely within and among individuals in the population; thus, it is important to identify outliers that may arise at each sampling level. A two-stage approach to analyzing nonlinear repeated measurements naturally allows parametric modeling of the respective variance structure for the intraindividual random errors and interindividual random effects. We propose a robust two-stage procedure based on Huber's (1981, Robust Statistics) theory of M-estimation to accommodate separately aberrant responses within an experimental unit and subjects deviating from the study population when the usual assumptions of normality are violated. A toxicology study of chronic ozone exposure in rats illustrates the impact of outliers on the population inference and hence the advantage of adopting the robust methodology. The robust weights generated by the two-stage M-estimation process also serve as diagnostics for gauging the relative influence of outliers at each level of the hierarchical model. A practical appeal of our proposal is the computational simplicity since the estimation algorithm may be implemented using standard statistical software with a nonlinear least squares routine and iterative capability.     

A hidden Markov movement model for rapidly identifying behavioral states from animal tracks

Electronic telemetry is frequently used to document animal movement through time. Methods that can identify underlying behaviors driving specific movement patterns can help us understand how and why animals use available space, thereby aiding conservation and management efforts. For aquatic animal tracking data with significant measurement error, a Bayesian state‐space model called the first‐Difference Correlated Random Walk with Switching (DCRWS) has often been used for this purpose. However, for aquatic animals, highly accurate tracking data are now becoming more common. We developed a new hidden Markov model (HMM) for identifying behavioral states from animal tracks with negligible error, called the hidden Markov movement model (HMMM). We implemented as the basis for the HMMM the process equation of the DCRWS, but we used the method of maximum likelihood and the R package TMB for rapid model fitting. The HMMM was compared to a modified version of the DCRWS for highly accurate tracks, the DCRWS, and to a common HMM for animal tracks fitted with the R package moveHMM. We show that the HMMM is both accurate and suitable for multiple species by fitting it to real tracks from a grey seal, lake trout, and blue shark, as well as to simulated data. The HMMM is a fast and reliable tool for making meaningful inference from animal movement data that is ideally suited for ecologists who want to use the popular DCRWS implementation and have highly accurate tracking data. It additionally provides a groundwork for development of more complex modeling of animal movement with TMB. To facilitate its uptake, we make it available through the R package swim.     

Observing the observer (I): meta-bayesian models of learning and decision-making

In this paper, we present a generic approach that can be used to infer how subjects make optimal decisions under uncertainty. This approach induces a distinction between a subject's perceptual model, which underlies the representation of a hidden "state of affairs" and a response model, which predicts the ensuing behavioural (or neurophysiological) responses to those inputs. We start with the premise that subjects continuously update a probabilistic representation of the causes of their sensory inputs to optimise their behaviour. In addition, subjects have preferences or goals that guide decisions about actions given the above uncertain representation of these hidden causes or state of affairs. From a Bayesian decision theoretic perspective, uncertain representations are so-called "posterior" beliefs, which are influenced by subjective "prior" beliefs. Preferences and goals are encoded through a "loss" (or "utility") function, which measures the cost incurred by making any admissible decision for any given (hidden) state of affair. By assuming that subjects make optimal decisions on the basis of updated (posterior) beliefs and utility (loss) functions, one can evaluate the likelihood of observed behaviour. Critically, this enables one to "observe the observer", i.e. identify (context- or subject-dependent) prior beliefs and utility-functions using psychophysical or neurophysiological measures. In this paper, we describe the main theoretical components of this meta-Bayesian approach (i.e. a Bayesian treatment of Bayesian decision theoretic predictions). In a companion paper ('Observing the observer (II): deciding when to decide'), we describe a concrete implementation of it and demonstrate its utility by applying it to simulated and real reaction time data from an associative learning task.     

Model fitting and evaluation in climate reconstruction of tree-ring data: A comment on Steinschneider et al. (2017): Hierarchical regression models for dendroclimatic standardization and climate reconstruction

Steinschneider et al. (2017) investigate model choices made in the hierarchical climate reconstruction approach of Schofield et al. (2016). We identify two flaws in their approach. The first is the use of an unusual approximation to Bayesian inference that unnecessarily discards important information. The second is that they mischaracterize the robustness of their reconstructions due to overlooking important features of the out-of-sample predictions. We demonstrate how full Bayesian inference can be conducted with no additional effort, providing R/JAGS code. We also show how graphical visualization of the out-of-sample predictions can lead to better understanding and comparison of the models fitted.     

Towards a Mathematical Theory of Cortical Micro-circuits

The theoretical setting of hierarchical Bayesian inference is gaining acceptance as a framework for understanding cortical computation. In this paper, we describe how Bayesian belief propagation in a spatio-temporal hierarchical model, called Hierarchical Temporal Memory (HTM), can lead to a mathematical model for cortical circuits. An HTM node is abstracted using a coincidence detector and a mixture of Markov chains. Bayesian belief propagation equations for such an HTM node define a set of functional constraints for a neuronal implementation. Anatomical data provide a contrasting set of organizational constraints. The combination of these two constraints suggests a theoretically derived interpretation for many anatomical and physiological features and predicts several others. We describe the pattern recognition capabilities of HTM networks and demonstrate the application of the derived circuits for modeling the subjective contour effect. We also discuss how the theory and the circuit can be extended to explain cortical features that are not explained by the current model and describe testable predictions that can be derived from the model.     

When mechanism matters: Bayesian forecasting using models of ecological diffusion

Ecological diffusion is a theory that can be used to understand and forecast spatio‐temporal processes such as dispersal, invasion, and the spread of disease. Hierarchical Bayesian modelling provides a framework to make statistical inference and probabilistic forecasts, using mechanistic ecological models. To illustrate, we show how hierarchical Bayesian models of ecological diffusion can be implemented for large data sets that are distributed densely across space and time. The hierarchical Bayesian approach is used to understand and forecast the growth and geographic spread in the prevalence of chronic wasting disease in white‐tailed deer (Odocoileus virginianus). We compare statistical inference and forecasts from our hierarchical Bayesian model to phenomenological regression‐based methods that are commonly used to analyse spatial occurrence data. The mechanistic statistical model based on ecological diffusion led to important ecological insights, obviated a commonly ignored type of collinearity, and was the most accurate method for forecasting.