Graphs, random sums, and sojourn time distributions, with application to ion-channel modeling.

This paper considers the distribution of a sojourn time in a class of states of a stochastic process having finite discrete state space where sojourn times in any individual state are independent and identically distributed, and transitions between states follow a Markov chain. The state space and possible transitions of the process are represented by a graph. Class sojourn time distributions are derived by modifying this graph using 'composition' of states, defining a new Markov chain on the modified graph, and expressing the sojourn time in a composition state as a random sum. Appropriate compositions are chosen according to the possible "cores" of sojourns in the particular class, where a core describes the structure of a sojourn in terms of a single state or a chain in the original graph. Graph methods provide an algorithmic basis for the derivation, which can be simplified by using symmetry results. Models of ion-channel kinetics are used throughout for illustration; class sojourn time distributions are important in such models because individual states are often indistinguishable experimentally. Markov processes are the special case where sojourn times in individual states are exponentially distributed. In this case kinetic parameter estimation based on the observed class sojourn time distribution is briefly discussed; explicit estimating equations applicable to sequential models of nicotinic receptor kinetics are given.     

Statistical inference from single channel records: two-state Markov model with limited time resolution

Though stochastic models are widely used to describe single ion channel behaviour, statistical inference based on them has received little consideration. This paper describes techniques of statistical inference, in particular likelihood methods, suitable for Markov models incorporating limited time resolution by means of a discrete detection limit. To simplify the analysis, attention is restricted to two-state models, although the methods have more general applicability. Non-uniqueness of the mean open-time and mean closed-time estimators obtained by moment methods based on single exponential approximations to the apparent open-time and apparent closed-time distributions has been reported. The present study clarifies and extends this previous work by proving that, for such approximations, the likelihood equations as well as the moment equations (usually) have multiple solutions. Such non-uniqueness corresponds to non-identifiability of the statistical model for the apparent quantities. By contrast, higher-order approximations yield theoretically identifiable models. Likelihood-based estimation procedures are developed for both single exponential and bi-exponential approximations. The methods and results are illustrated by numerical examples based on literature and simulated data, with consideration given to empirical distributions and model control, likelihood plots, and point estimation and confidence regions.     

Covariate Adjustment of Event Histories Estimated from Markov Chains: The Additive Approach

Markov chain models are frequently used for studying event histories that include transitions between several states. An empirical transition matrix for nonhomogeneous Markov chains has previously been developed, including a detailed statistical theory based on counting processes and martingales. In this article, we show how to estimate transition probabilities dependent on covariates. This technique may, e.g., be used for making estimates of individual prognosis in epidemiological or clinical studies. The covariates are included through nonparametric additive models on the transition intensities of the Markov chain. The additive model allows for estimation of covariate-dependent transition intensities, and again a detailed theory exists based on counting processes. The martingale setting now allows for a very natural combination of the empirical transition matrix and the additive model, resulting in estimates that can be expressed as stochastic integrals, and hence their properties are easily evaluated. Two medical examples will be given. In the first example, we study how the lung cancer mortality of uranium miners depends on smoking and radon exposure. In the second example, we study how the probability of being in response depends on patient group and prophylactic treatment for leukemia patients who have had a bone marrow transplantation. A program in R and S-PLUS that can carry out the analyses described here has been developed and is freely available on the Internet.     

A queueing model for chronic recurrent conditions under panel observation

In many chronic conditions, subjects alternate between an active and an inactive state, and sojourns into the active state may involve multiple lesions, infections, or other recurrences with different times of onset and resolution. We present a biologically interpretable model of such chronic recurrent conditions based on a queueing process. The model has a birth-death process describing recurrences and a semi-Markov process describing the alternation between active and inactive states, and can be fit to panel data that provide only a binary assessment of the active or inactive state at a series of discrete time points using a hidden Markov approach. We accommodate individual heterogeneity and covariates using a random effects model, and simulate the posterior distribution of unknowns using a Markov chain Monte Carlo algorithm. Application to a clinical trial of genital herpes shows how the method can characterize the biology of the disease and estimate treatment efficacy.     

Applications of hidden hybrid Markov/semi‐Markov models: from stopover duration to breeding success dynamics

Usually in capture–recapture, a model parameter is time or time since first capture dependent. However, the case where the probability of staying in one state depends on the time spent in that particular state is not rare. Hidden Markov models are not appropriate to manage these situations. A more convenient approach would be to consider models that incorporate semi‐Markovian states which explicitly define the waiting time distribution and have been used in previous biologic studies as a convenient framework for modeling the time spent in a given physiological state. Here, we propose hidden Markovian models that combine several nonhomogeneous Markovian states with one semi‐Markovian state and which (i) are well adapted to imperfect and variable detection and (ii) allow us to consider time, time since first capture, and time spent in one state effects. Implementation details depending on the number of semi‐Markovian states are discussed. From a user's perspective, the present approach enhances the toolbox for analyzing capture–recapture data. We then show the potential of this framework by means of two ecological examples: (i) stopover duration and (ii) breeding success dynamics.     

Linking exponential components to kinetic states in Markov models for single-channel gating

Discrete state Markov models have proven useful for describing the gating of single ion channels. Such models predict that the dwell-time distributions of open and closed interval durations are described by mixtures of exponential components, with the number of exponential components equal to the number of states in the kinetic gating mechanism. Although the exponential components are readily calculated (Colquhoun and Hawkes, 1982, Phil. Trans. R. Soc. Lond. B. 300:1-59), there is little practical understanding of the relationship between components and states, as every rate constant in the gating mechanism contributes to each exponential component. We now resolve this problem for simple models. As a tutorial we first illustrate how the dwell-time distribution of all closed intervals arises from the sum of constituent distributions, each arising from a specific gating sequence. The contribution of constituent distributions to the exponential components is then determined, giving the relationship between components and states. Finally, the relationship between components and states is quantified by defining and calculating the linkage of components to states. The relationship between components and states is found to be both intuitive and paradoxical, depending on the ratios of the state lifetimes. Nevertheless, both the intuitive and paradoxical observations can be described within a consistent framework. The approach used here allows the exponential components to be interpreted in terms of underlying states for all possible values of the rate constants, something not previously possible.     

Markov and Semi‐Markov Switching Linear Mixed Models Used to Identify Forest Tree Growth Components

Summary Tree growth is assumed to be mainly the result of three components: (i) an endogenous component assumed to be structured as a succession of roughly stationary phases separated by marked change points that are asynchronous among individuals, (ii) a time‐varying environmental component assumed to take the form of synchronous fluctuations among individuals, and (iii) an individual component corresponding mainly to the local environment of each tree. To identify and characterize these three components, we propose to use semi‐Markov switching linear mixed models, i.e., models that combine linear mixed models in a semi‐Markovian manner. The underlying semi‐Markov chain represents the succession of growth phases and their lengths (endogenous component) whereas the linear mixed models attached to each state of the underlying semi‐Markov chain represent—in the corresponding growth phase—both the influence of time‐varying climatic covariates (environmental component) as fixed effects, and interindividual heterogeneity (individual component) as random effects. In this article, we address the estimation of Markov and semi‐Markov switching linear mixed models in a general framework. We propose a Monte Carlo expectation–maximization like algorithm whose iterations decompose into three steps: (i) sampling of state sequences given random effects, (ii) prediction of random effects given state sequences, and (iii) maximization. The proposed statistical modeling approach is illustrated by the analysis of successive annual shoots along Corsican pine trunks influenced by climatic covariates.     

Bayesian restoration of a hidden Markov chain with applications to DNA sequencing.

Hidden Markov models (HMMs) are a class of stochastic models that have proven to be powerful tools for the analysis of molecular sequence data. A hidden Markov model can be viewed as a black box that generates sequences of observations. The unobservable internal state of the box is stochastic and is determined by a finite state Markov chain. The observable output is stochastic with distribution determined by the state of the hidden Markov chain. We present a Bayesian solution to the problem of restoring the sequence of states visited by the hidden Markov chain from a given sequence of observed outputs. Our approach is based on a Monte Carlo Markov chain algorithm that allows us to draw samples from the full posterior distribution of the hidden Markov chain paths. The problem of estimating the probability of individual paths and the associated Monte Carlo error of these estimates is addressed. The method is illustrated by considering a problem of DNA sequence multiple alignment. The special structure for the hidden Markov model used in the sequence alignment problem is considered in detail. In conclusion, we discuss certain interesting aspects of biological sequence alignments that become accessible through the Bayesian approach to HMM restoration.     

Kinetic time constants independent of previous single-channel activity suggest Markov gating for a large conductance Ca-activated K channel

Models for the gating of ion channels usually assume that the rate constants for leaving any given kinetic state are independent of previous channel activity. Although such discrete Markov models have been successful in describing channel gating, there is little direct evidence for the Markov assumption of time-invariant rate constants for constant conditions. This paper tests the Markov assumption by determining whether the single-channel kinetics of the large conductance Ca-activated K channel in cultured rat skeletal muscle are independent of previous single-channel activity. The experimental approach is to examine dwell-time distributions conditional on adjacent interval durations. The time constants of the exponential components describing the distributions are found to be independent of adjacent interval duration, and hence, previous channel activity. In contrast, the areas of the different components can change. Since the observed time constants are a function of the underlying rate constants for transitions among the kinetic states, the observation of time constants independent of previous channel activity suggests that the rate constants are also independent of previous channel activity. Thus, the channel kinetics are consistent with Markov gating. An observed dependent (inverse) relationship between durations of adjacent open and shut intervals together with Markov gating indicates that there are two or more independent transition pathways connecting open and shut states. Finally, no evidence is found to suggest that gating is not at thermodynamic equilibrium: the inverse relationship was independent of the time direction of analysis.     

A general approach for population games with application to vaccination

Reconciling the interests of individuals with the interests of communities is a major challenge in designing and implementing health policies. In this paper, we present a technique based on a combination of mechanistic population-scale models, Markov decision process theory and game theory that facilitates the evaluation of game theoretic decisions at both individual and community scales. To illustrate our technique, we provide solutions to several variants of the simple vaccination game including imperfect vaccine efficacy and differential waning of natural and vaccine immunity. In addition, we show how path-integral approaches can be applied to the study of models in which strategies are fixed waiting times rather than exponential random variables. These methods can be applied to a wide variety of decision problems with population-dynamic feedbacks.     

Modeling nonhomogeneous Markov processes via time transformation

Summary .   Longitudinal studies are a powerful tool for characterizing the course of chronic disease. These studies are usually carried out with subjects observed at periodic visits giving rise to panel data. Under this observation scheme the exact times of disease state transitions and sequence of disease states visited are unknown and Markov process models are often used to describe disease progression. Most applications of Markov process models rely on the assumption of time homogeneity, that is, that the transition rates are constant over time. This assumption is not satisfied when transition rates depend on time from the process origin. However, limited statistical tools are available for dealing with nonhomogeneity. We propose models in which the time scale of a nonhomogeneous Markov process is transformed to an operational time scale on which the process is homogeneous. We develop a method for jointly estimating the time transformation and the transition intensity matrix for the time transformed homogeneous process. We assess maximum likelihood estimation using the Fisher scoring algorithm via simulation studies and compare performance of our method to homogeneous and piecewise homogeneous models. We apply our methodology to a study of delirium progression in a cohort of stem cell transplantation recipients and show that our method identifies temporal trends in delirium incidence and recovery.     

Sampling errors create bias in Markov models for community dynamics: the problem and a method for its solution

Repeated, spatially explicit sampling is widely used to characterize the dynamics of sessile communities in both terrestrial and aquatic systems, yet our understanding of the consequences of errors made in such sampling is limited. In particular, when Markov transition probabilities are calculated by tracking individual points over time, misidentification of the same spatial locations will result in biased estimates of transition probabilities, successional rates, and community trajectories. Nonetheless, to date, all published studies that use such data have implicitly assumed that resampling occurs without error when making estimates of transition rates. Here, we develop and test a straightforward maximum likelihood approach, based on simple field estimates of resampling errors, to arrive at corrected estimates of transition rates between species in a rocky intertidal community. We compare community Markov models based on raw and corrected transition estimates using data from Endocladia muricata-dominated plots in a California intertidal assemblage, finding that uncorrected predictions of succession consistently overestimate recovery time. We tested the precision and accuracy of the approach using simulated datasets and found good performance of our estimation method over a range of realistic sample sizes and error rates.     

The importance of individual developmental variation in stage‐structured population models

Population stage structure is fundamental to ecology, and models of this structure have proven useful in many different systems. Many ecological variables other than stage, such as habitat type, site occupancy and metapopulation status are also modelled using transitions among discrete states. Transitions among life stages can be characterised by the distribution of time spent in each stage, including the mean and variance of each stage duration and within‐individual correlations among multiple stage durations. Three modelling traditions represent stage durations differently. Matrix models can be derived as a long‐run approximation from any distribution of stage durations, but they are often interpreted directly as a Markov model for stage transitions. Statistical stage‐duration distribution models accommodate the variation typical of cohort development data, but such realism has rarely been incorporated in population theory or statistical population models. Delay‐differential equation models include lags but no variation, except in limited cases. We synthesise these models in one framework and illustrate how individual variation and correlations in development can impact population growth. Furthermore, different development models can yield the same long‐term matrix transition rates but different sensitivities and elasticities. Finally, we discuss future directions for estimating realistic stage duration models from data.     

Spatial multistate transitional models for longitudinal event data

Summary .   Follow-up medical studies often collect longitudinal data on patients. Multistate transitional models are useful for analysis in such studies where at any point in time, individuals may be said to occupy one of a discrete set of states and interest centers on the transition process between states. For example, states may refer to the number of recurrences of an event, or the stage of a disease. We develop a hierarchical modeling framework for the analysis of such longitudinal data when the processes corresponding to different subjects may be correlated spatially over a region. Continuous-time Markov chains incorporating spatially correlated random effects are introduced. Here, joint modeling of both spatial dependence as well as dependence between different transition rates is required and a multivariate spatial approach is employed. A proportional intensities frailty model is developed where baseline intensity functions are modeled using parametric Weibull forms, piecewise-exponential formulations, and flexible representations based on cubic B-splines. The methodology is developed within the context of a study examining invasive cardiac procedures in Quebec. We consider patients admitted for acute coronary syndrome throughout the 139 local health units of the province and examine readmission and mortality rates over a 4-year period.     

Hidden Markov models for settings with interval-censored transition times and uncertain time origin: application to HIV genetic analyses

The simplicity and flexibility of Markov models make them appealing for investigations of the acquisition of HIV drug-resistance mutations, whose presence can define specific Markov states. Because the exact time of acquiring a mutation is not observed during clinical research studies on HIV infection, it is important that methods for fitting such models accommodate interval-censored transition times. Furthermore, many such studies include patients with extensive treatment experience prior to the onset of the studies. Therefore, the virus in these patients may have already acquired resistance mutations by study entry. Retrospective data regarding the time on treatment, which is often known or known with error, provide information about the acquisition rates before the start of a study. Finally, variability in the genetic sequences of circulating HIV creates uncertainty in the Markov states. This paper considers approaches to fitting Markov models to data with interval-censored transition times when the time origin and the Markov states are known with error. The methods were applied to AIDS Clinical Trial Group protocol 398, a randomized comparison of mono- versus dual-protease inhibitor use in heavily pretreated patients. We found that the estimated rates of acquiring certain classes of mutations depended on the presence of others, and that the precision of these estimates can be considerably improved by inclusion of retrospective data.     

Forecasting and chronic illness

Biomedical data in the form of series of observations made on a single process at regular intervals constitute a discrete time series and are eligible for time series methods of analysis. The models yielded by this analysis provide the framework within which exponential smoothing methods may operate on the data to provide recurrent forecasts of future states of the process. Because the forecasts may be made on an individual basis and are sensitive to the past behavior of the individual process, the methods are presented as being potentially of great utility in the management of chronic and progressive illnesses. When incorporated into automated testing and diagnostic systems, the forecasting method will provide the capability of making prognoses for large numbers of individuals, quickly, routinely and reproducibly.     

Single ion channel models incorporating aggregation and time interval omission.

We present a general theoretical framework, incorporating both aggregation of states into classes and time interval omission, for stochastic modeling of the dynamic aspects of single channel behavior. Our semi-Markov models subsume the standard continuous-time Markov models, diffusion models and fractal models. In particular our models allow for quite general distributions of state sojourn times and arbitrary correlations between successive sojourn times. Another key feature is the invariance of our framework with respect to time interval omission: that is, properties of the aggregated process incorporating time interval omission can be derived directly from corresponding properties of the process without it. Even in the special case when the underlying process is Markov, this leads to considerable clarification of the effects of time interval omission. Among the properties considered are equilibrium behavior, sojourn time distributions and their moments, and auto-correlation and cross-correlation functions. The theory is motivated by ion channel mechanisms drawn from the literature, and illustrated by numerical examples based on these.     

Looking for a needle in a haystack: inference about individual fitness components in a heterogeneous population

Studies of wild vertebrates have provided evidence of substantial differences in lifetime reproduction among individuals and the sequences of life history ‘states’ during life (breeding, nonbreeding, etc.). Such differences may reflect ‘fixed’ differences in fitness components among individuals determined before, or at the onset of reproductive life. Many retrospective life history studies have translated this idea by assuming a ‘latent’ unobserved heterogeneity resulting in a fixed hierarchy among individuals in fitness components. Alternatively, fixed differences among individuals are not necessarily needed to account for observed levels of individual heterogeneity in life histories. Individuals with identical fitness traits may stochastically experience different outcomes for breeding and survival through life that lead to a diversity of ‘state’ sequences with some individuals living longer and being more productive than others, by chance alone. The question is whether individuals differ in their underlying fitness components in ways that cannot be explained by observable ‘states’ such as age, previous breeding success, etc. Here, we compare statistical models that represent these opposing hypotheses, and mixtures of them, using data from kittiwakes. We constructed models that accounted for observed covariates, individual random effects (unobserved heterogeneity), first‐order Markovian transitions between observed states, or combinations of these features. We show that individual sequences of states are better accounted for by models incorporating unobserved heterogeneity than by models including first‐order Markov processes alone, or a combination of both. If we had not considered individual heterogeneity, models including Markovian transitions would have been the best performing ones. We also show that inference about age‐related changes in fitness components is sensitive to incorporation of underlying individual heterogeneity in models. Our approach provides insight into the sources of individual heterogeneity in life histories, and can be applied to other data sets to examine the ubiquity of our results across the tree of life.     

Non-homogeneous Markov process models with informative observations with an application to Alzheimer's disease

Identifying risk factors for transition rates among normal cognition, mildly cognitive impairment, dementia and death in an Alzheimer's disease study is very important. It is known that transition rates among these states are strongly time dependent. While Markov process models are often used to describe these disease progressions, the literature mainly focuses on time homogeneous processes, and limited tools are available for dealing with non-homogeneity. Further, patients may choose when they want to visit the clinics, which creates informative observations. In this paper, we develop methods to deal with non-homogeneous Markov processes through time scale transformation when observation times are pre-planned with some observations missing. Maximum likelihood estimation via the EM algorithm is derived for parameter estimation. Simulation studies demonstrate that the proposed method works well under a variety of situations. An application to the Alzheimer's disease study identifies that there is a significant increase in transition rates as a function of time. Furthermore, our models reveal that the non-ignorable missing mechanism is perhaps reasonable.     

Combining phylogenetic and hidden Markov models in biosequence analysis.

A few models have appeared in recent years that consider not only the way substitutions occur through evolutionary history at each site of a genome, but also the way the process changes from one site to the next. These models combine phylogenetic models of molecular evolution, which apply to individual sites, and hidden Markov models, which allow for changes from site to site. Besides improving the realism of ordinary phylogenetic models, they are potentially very powerful tools for inference and prediction--for example, for gene finding or prediction of secondary structure. In this paper, we review progress on combined phylogenetic and hidden Markov models and present some extensions to previous work. Our main result is a simple and efficient method for accommodating higher-order states in the HMM, which allows for context-dependent models of substitution--that is, models that consider the effects of neighboring bases on the pattern of substitution. We present experimental results indicating that higher-order states, autocorrelated rates, and multiple functional categories all lead to significant improvements in the fit of a combined phylogenetic and hidden Markov model, with the effect of higher-order states being particularly pronounced.