Semi‐Markov Models with Phase‐Type Sojourn Distributions

Summary Continuous‐time multistate models are widely used for categorical response data, particularly in the modeling of chronic diseases. However, inference is difficult when the process is only observed at discrete time points, with no information about the times or types of events between observation times, unless a Markov assumption is made. This assumption can be limiting as rates of transition between disease states might instead depend on the time since entry into the current state. Such a formulation results in a semi‐Markov model. We show that the computational problems associated with fitting semi‐Markov models to panel‐observed data can be alleviated by considering a class of semi‐Markov models with phase‐type sojourn distributions. This allows methods for hidden Markov models to be applied. In addition, extensions to models where observed states are subject to classification error are given. The methodology is demonstrated on a dataset relating to development of bronchiolitis obliterans syndrome in post‐lung‐transplantation patients.     

Modeling nonhomogeneous Markov processes via time transformation

Summary .   Longitudinal studies are a powerful tool for characterizing the course of chronic disease. These studies are usually carried out with subjects observed at periodic visits giving rise to panel data. Under this observation scheme the exact times of disease state transitions and sequence of disease states visited are unknown and Markov process models are often used to describe disease progression. Most applications of Markov process models rely on the assumption of time homogeneity, that is, that the transition rates are constant over time. This assumption is not satisfied when transition rates depend on time from the process origin. However, limited statistical tools are available for dealing with nonhomogeneity. We propose models in which the time scale of a nonhomogeneous Markov process is transformed to an operational time scale on which the process is homogeneous. We develop a method for jointly estimating the time transformation and the transition intensity matrix for the time transformed homogeneous process. We assess maximum likelihood estimation using the Fisher scoring algorithm via simulation studies and compare performance of our method to homogeneous and piecewise homogeneous models. We apply our methodology to a study of delirium progression in a cohort of stem cell transplantation recipients and show that our method identifies temporal trends in delirium incidence and recovery.     

Evaluation and Estimation of Various Markov Models with Applications to Membrane Channel Kinetics

Hidden Markov modelling is a powerful and efficient digital signal processing strategy for extracting the maximum likelihood model from a finite length sample of noisy data. Assuming the number of states in the model is known, then the state levels, transition probabilities, initial state distribution and the noise variance can be estimated. We investigate the applicability of this technique in membrane channel kinetics not only as a parameter estimator, but also as an aid to discriminating between various model types according to their statistical likelihood. We survey three representative classes of channel dynamics, namely: aggregated Markov models, semi-Markov models (with asymptotically convergent transition probabilities), and coupled Markov models; reformulating each within a discrete-time hidden Markov model framework. We then provide numerical evidence of the effectiveness of the procedure using simulated channel data and hence show that the correct model, as well as the model parameters, can be discerned. We also demonstrate that the model likelihood can be used to indicate the approximate number of states in the model.     

A biological marker model for predicting disease transitions

For patients with chronic myelogenous leukemia (CML), the effect of elevated blood levels of adenosine deaminase (ADA) is studied as a marker for transitions from stable disease to blast crisis and then to death. Data in the form of snapshots over time, with day, state of disease, and ADA level, are analyzed for 55 patients. A simple three-state Markov model with one-way transition probabilities dependent on ADA is used to determine if the marker has a significant effect on the prediction of changes from stable disease to blast crisis.     

A stochastic Markov chain model to describe lung cancer growth and metastasis

A stochastic Markov chain model for metastatic progression is developed for primary lung cancer based on a network construction of metastatic sites with dynamics modeled as an ensemble of random walkers on the network. We calculate a transition matrix, with entries (transition probabilities) interpreted as random variables, and use it to construct a circular bi-directional network of primary and metastatic locations based on postmortem tissue analysis of 3827 autopsies on untreated patients documenting all primary tumor locations and metastatic sites from this population. The resulting 50 potential metastatic sites are connected by directed edges with distributed weightings, where the site connections and weightings are obtained by calculating the entries of an ensemble of transition matrices so that the steady-state distribution obtained from the long-time limit of the Markov chain dynamical system corresponds to the ensemble metastatic distribution obtained from the autopsy data set. We condition our search for a transition matrix on an initial distribution of metastatic tumors obtained from the data set. Through an iterative numerical search procedure, we adjust the entries of a sequence of approximations until a transition matrix with the correct steady-state is found (up to a numerical threshold). Since this constrained linear optimization problem is underdetermined, we characterize the statistical variance of the ensemble of transition matrices calculated using the means and variances of their singular value distributions as a diagnostic tool. We interpret the ensemble averaged transition probabilities as (approximately) normally distributed random variables. The model allows us to simulate and quantify disease progression pathways and timescales of progression from the lung position to other sites and we highlight several key findings based on the model.     

Markov models for covariate dependence of binary sequences

Suppose that a heterogeneous group of individuals is followed over time and that each individual can be in state 0 or state 1 at each time point. The sequence of states is assumed to follow a binary Markov chain. In this paper we model the transition probabilities for the 0 to 0 and 1 to 0 transitions by two logistic regressions, thus showing how the covariates relate to changes in state. With p covariates, there are 2(p + 1) parameters including intercepts, which we estimate by maximum likelihood. We show how to use transition probability estimates to test hypotheses about the probability of occupying state 0 at time i (i = 2, ..., T) and the equilibrium probability of state 0. These probabilities depend on the covariates. A recursive algorithm is suggested to estimate regression coefficients when some responses are missing. Extensions of the basic model which allow time-dependent covariates and nonstationary or second-order Markov chains are presented. An example shows the model applied to a study of the psychological impact of breast cancer in which women did or did not manifest distress at four time points in the year following surgery.     

Sampling errors create bias in Markov models for community dynamics: the problem and a method for its solution

Repeated, spatially explicit sampling is widely used to characterize the dynamics of sessile communities in both terrestrial and aquatic systems, yet our understanding of the consequences of errors made in such sampling is limited. In particular, when Markov transition probabilities are calculated by tracking individual points over time, misidentification of the same spatial locations will result in biased estimates of transition probabilities, successional rates, and community trajectories. Nonetheless, to date, all published studies that use such data have implicitly assumed that resampling occurs without error when making estimates of transition rates. Here, we develop and test a straightforward maximum likelihood approach, based on simple field estimates of resampling errors, to arrive at corrected estimates of transition rates between species in a rocky intertidal community. We compare community Markov models based on raw and corrected transition estimates using data from Endocladia muricata-dominated plots in a California intertidal assemblage, finding that uncorrected predictions of succession consistently overestimate recovery time. We tested the precision and accuracy of the approach using simulated datasets and found good performance of our estimation method over a range of realistic sample sizes and error rates.     

Analysis of cause of death: Competing risks or progressive illness‐death model?

The analysis of cause of death is increasingly becoming a topic in oncology. It is usually distinguished between disease‐related and disease‐unrelated death. A frequently used approach is to define death as disease‐related when a progression to advanced phases has occurred before, otherwise as disease‐unrelated. The data are often analyzed as competing risks, while a progressive illness‐death model might in fact describe the situation more precisely. In this study, we investigated under which circumstances this misspecification leads to biased estimations of the state occupation probabilities. We simulated data according to the progressive illness‐death model in various settings, analyzed them with a competing risks model and with a progressive illness‐death model and compared them to the true state occupation probabilities. Censoring was either added independently of the status or based on the patients' status. The simulations showed that the censoring mechanism was decisive for the bias while neither the progression hazard nor the Markov property was important. Further, we found a slightly increased standard deviation for the competing risk estimator when censoring was independent of the patients' status. For illustration, both methods were applied to two practical examples of chronic myeloid leukemia (CML): one randomized controlled trial and one registry data set. While in the first case both estimators yielded almost identical results, in the latter case, visible differences were found between both methods.     

Estimation in a semi-Markov transformation model

Semi-Markov and modulated renewal processes provide a large class of multi-state models which can be used for analysis of longitudinal failure time data. In biomedical applications, models of this kind are often used to describe evolution of a disease and assume that patient may move among a finite number of states representing different phases in the disease progression. Several authors proposed extensions of the proportional hazard model for regression analysis of these processes. In this paper, we consider a general class of censored semi-Markov and modulated renewal processes and propose use of transformation models for their analysis. Special cases include modulated renewal processes with interarrival times specified using transformation models, and semi-Markov processes with with one-step transition probabilities defined using copula-transformation models. We discuss estimation of finite and infinite dimensional parameters and develop an extension of the Gaussian multiplier method for setting confidence bands for transition probabilities and related parameters. A transplant outcome data set from the Center for International Blood and Marrow Transplant Research is used for illustrative purposes.     

Generalizing HMMs to Continuous Time for Fast Kinetics: Hidden Markov Jump Processes

The hidden Markov model (HMM) is a framework for time series analysis widely applied to single-molecule experiments. Although initially developed for applications outside the natural sciences, the HMM has traditionally been used to interpret signals generated by physical systems, such as single molecules, evolving in a discrete state space observed at discrete time levels dictated by the data acquisition rate. Within the HMM framework, transitions between states are modeled as occurring at the end of each data acquisition period and are described using transition probabilities. Yet, whereas measurements are often performed at discrete time levels in the natural sciences, physical systems evolve in continuous time according to transition rates. It then follows that the modeling assumptions underlying the HMM are justified if the transition rates of a physical process from state to state are small as compared to the data acquisition rate. In other words, HMMs apply to slow kinetics. The problem is, because the transition rates are unknown in principle, it is unclear, a priori, whether the HMM applies to a particular system. For this reason, we must generalize HMMs for physical systems, such as single molecules, because these switch between discrete states in “continuous time”. We do so by exploiting recent mathematical tools developed in the context of inferring Markov jump processes and propose the hidden Markov jump process. We explicitly show in what limit the hidden Markov jump process reduces to the HMM. Resolving the discrete time discrepancy of the HMM has clear implications: we no longer need to assume that processes, such as molecular events, must occur on timescales slower than data acquisition and can learn transition rates even if these are on the same timescale or otherwise exceed data acquisition rates.     

Exchangeability in multivariate Markov chain models.

Time-homogeneous Markov chain models with state space [0, 1]k are useful in analysis of binary follow-up data on k individuals that interact. The number of parameters increases exponentially with k so more restrictive models are imperative for statistical inference. The hypothesis that the matrix of transition probabilities is invariant under permutation of individuals is discussed. It is shown that if individuals are exchangeable, then the process counting the number of individuals occupying a given state is a Markov chain. This reduction of data is sufficient if either at most a single individual may change state between two consecutive time points or if a state is absorbing. Similar results are obtained for exchangeability within two subgroups. Inference in the multivariate process reduces to a univariate problem if individuals are independent given the group's previous response. It is shown how conditional independence could be tested assuming exchangeability. The different hypotheses re examined in an analysis of the occurrence of bacteria in milk samples of Danish dairy cattle.     

Multi-Stage Markov Analysis of Progressive Disease Applied to Melanoma

A fundamental research goal in clinical studies of progressive, multi-stage disease is to understand its natural history and its relationship with prognostic factors. Our current understanding of this topic is based on the use of two-stage methods for event-time analysis which neglect intermediate transition information. In contrast, a multi-stage model utilizes all available data and provides more accurate insight into disease progression. We specify a forward-flowing multi-stage Markov model based on the discrete clinical stages of disease. By assuming the process to be Markovian, we avoid unnecessary complications to our numerical estimation procedure. Due to noncontinuous patient monitoring and the chronic nature of progressive disease, heavy right- and interval-censoring exists in the transition data. We develop a modified ECM algorithm to numerically carry out the otherwise complicated parameter estimation for this process. We also identify significant prognostic factors relevant to each transition, along with the relative importance of each prognostic factor. The numerical estimation is stable, and the parameter estimates are maximum likelihood estimates (Meng, 1990). In general our forward-flowing multi-stage models provide a flexible framework for the study of the effects of prognostic factors on progression among several stages. We apply our Markov model to a dataset of malignant melanoma patients, and present an inferential discussion. Results from our multi-stage Markov model provide an improved understanding of melanoma progression.     

The effect of errors of diagnosis and frequency of examination on reported rates of disease

Follow-up studies of certain diseases such as cervical dysplasia and carcinoma in situ may employ repeated screening to determine disease incidence in selected cohorts. Where errors of diagnosis occur, the true cohort experience will be distorted by an amount dependent upon the frequency of screening and the magnitude of the probabilities of incorrect diagnoses. These effects are investigated through a simple model of follow-up which employs three basic assumptions: that the probability of false positive and false negative diagnoses are constant; that individuals diagnosed as positive are treated and removed from the study, irrespective of whether or not they actually have the disease; and that fixed intervals occur between follow-up exams. The analysis permits one to assess the amount of bias introduced, and numerical examples are provided.     

Making use of multiple surveys: Estimating breeding probability using a multievent‐robust design capture–recapture model

Increased environmental stochasticity due to climate change will intensify temporal variance in the life‐history traits, and especially breeding probabilities, of long‐lived iteroparous species. These changes may decrease individual fitness and population viability and is therefore important to monitor. In wild animal populations with imperfect individual detection, breeding probabilities are best estimated using capture–recapture methods. However, in many vertebrate species (e.g., amphibians, turtles, seabirds), nonbreeders are unobservable because they are not tied to a territory or breeding location. Although unobservable states can be used to model temporary emigration of nonbreeders, there are disadvantages to having unobservable states in capture–recapture models. The best solution to deal with unobservable life‐history states is therefore to eliminate them altogether. Here, we achieve this objective by fitting novel multievent‐robust design models which utilize information obtained from multiple surveys conducted throughout the year. We use this approach to estimate annual breeding probabilities of capital breeding female elephant seals (Mirounga leonina). Conceptually, our approach parallels a multistate version of the Barker/robust design in that it combines robust design capture data collected during discrete breeding seasons with observations made at other times of the year. A substantial advantage of our approach is that the nonbreeder state became “observable” when multiple data sources were analyzed together. This allowed us to test for the existence of state‐dependent survival (with some support found for lower survival in breeders compared to nonbreeders), and to estimate annual breeding transitions to and from the nonbreeder state with greater precision (where current breeders tended to have higher future breeding probabilities than nonbreeders). We used program E‐SURGE (2.1.2) to fit the multievent‐robust design models, with uncertainty in breeding state assignment (breeder, nonbreeder) being incorporated via a hidden Markov process. This flexible modeling approach can easily be adapted to suit sampling designs from numerous species which may be encountered during and outside of discrete breeding seasons.     

Estimation of integrated transition hazards and stage occupation probabilities for non-Markov systems under dependent censoring

We propose nonparametric estimators of the stage occupation probabilities and transition hazards for a multistage system that is not necessarily Markovian, using data that are subject to dependent right censoring. We assume that the hazard of being censored at a given instant depends on a possibly time-dependent covariate process as opposed to assuming a fixed censoring hazard (independent censoring). The estimator of the integrated transition hazard matrix has a Nelson-Aalen form where each of the counting processes counting the number of transitions between states and the risk sets for leaving each stage have an IPCW (inverse probability of censoring weighted) form. We estimate these weights using Aalen's linear hazard model. Finally, the stage occupation probabilities are obtained from the estimated integrated transition hazard matrix via product integration. Consistency of these estimators under the general paradigm of non-Markov models is established and asymptotic variance formulas are provided. Simulation results show satisfactory performance of these estimators. An analysis of data on graft-versus-host disease for bone marrow transplant patients is used as an illustration.     

Analysis of Longitudinal Data of Epileptic Seizure Counts – A Two‐State Hidden Markov Regression Approach

This paper discusses a two‐state hidden Markov Poisson regression (MPR) model for analyzing longitudinal data of epileptic seizure counts, which allows for the rate of the Poisson process to depend on covariates through an exponential link function and to change according to the states of a two‐state Markov chain with its transition probabilities associated with covariates through a logit link function. This paper also considers a two‐state hidden Markov negative binomial regression (MNBR) model, as an alternative, by using the negative binomial instead of Poisson distribution in the proposed MPR model when there exists extra‐Poisson variation conditional on the states of the Markov chain. The two proposed models in this paper relax the stationary requirement of the Markov chain, allow for overdispersion relative to the usual Poisson regression model and for correlation between repeated observations. The proposed methodology provides a plausible analysis for the longitudinal data of epileptic seizure counts, and the MNBR model fits the data much better than the MPR model. Maximum likelihood estimation using the EM and quasi‐Newton algorithms is discussed. A Monte Carlo study for the proposed MPR model investigates the reliability of the estimation method, the choice of probabilities for the initial states of the Markov chain, and some finite sample behaviors of the maximum likelihood estimates, suggesting that (1) the estimation method is accurate and reliable as long as the total number of observations is reasonably large, and (2) the choice of probabilities for the initial states of the Markov process has little impact on the parameter estimates.     

A compact statistical model of the song syntax in Bengalese finch

Songs of many songbird species consist of variable sequences of a finite number of syllables. A common approach for characterizing the syntax of these complex syllable sequences is to use transition probabilities between the syllables. This is equivalent to the Markov model, in which each syllable is associated with one state, and the transition probabilities between the states do not depend on the state transition history. Here we analyze the song syntax in Bengalese finch. We show that the Markov model fails to capture the statistical properties of the syllable sequences. Instead, a state transition model that accurately describes the statistics of the syllable sequences includes adaptation of the self-transition probabilities when states are revisited consecutively, and allows associations of more than one state to a given syllable. Such a model does not increase the model complexity significantly. Mathematically, the model is a partially observable Markov model with adaptation (POMMA). The success of the POMMA supports the branching chain network model of how syntax is controlled within the premotor song nucleus HVC, but also suggests that adaptation and many-to-one mapping from the syllable-encoding chain networks in HVC to syllables should be included in the network model.     

Cumulative disease progression models for cross‐sectional data: A review and comparison

A better understanding of disease progression is beneficial for early diagnosis and appropriate individual therapy. Many different approaches for statistical modelling of cumulative disease progression have been proposed in the literature, including simple path models up to complex restricted Bayesian networks. Important fields of application are diseases such as cancer and HIV. Tumour progression is measured by means of chromosome aberrations, whereas people infected with HIV develop drug resistances because of genetic changes of the HI‐virus. These two very different diseases have typical courses of disease progression, which can be modelled partly by consecutive and partly by independent steps. This paper gives an overview of the different progression models and points out their advantages and drawbacks. Different models are compared via simulations to analyse how they work if some of their assumptions are violated. In a simulation study, we evaluate how models perform in terms of fitting induced multivariate probability distributions and topological relationships. We often find that the true model class used for generating data is outperformed by either a less or a more complex model class. The more flexible conjunctive Bayesian networks can be used to fit oncogenetic trees, whereas mixtures of oncogenetic trees with three tree components can be well fitted by mixture models with only two tree components.     

transition priors for protein hidden Markov models: an empirical study towards maximum discrimination.

Insertions and deletions in a profile hidden Markov model (HMM) are modeled by transition probabilities between insert, delete and match states. These are estimated by combining observed data and prior probabilities. The transition prior probabilities can be defined either ad hoc or by maximum likelihood (ML) estimation. We show that the choice of transition prior greatly affects the HMM's ability to discriminate between true and false hits. HMM discrimination was measured using the HMMER 2.2 package applied to 373 families from Pfam. We measured the discrimination between true members and noise sequences employing various ML transition priors and also systematically scanned the parameter space of ad hoc transition priors. Our results indicate that ML priors produce far from optimal discrimination, and we present an empirically derived prior that considerably decreases the number of misclassifications compared to ML. Most of the difference stems from the probabilities for exiting a delete state. The ML prior, which is unaware of noise sequences, estimates a delete-to-delete probability that is relatively high and does not penalize noise sequences enough for optimal discrimination.     

Modeling animals' behavioral response by Markov chain models for capture-recapture experiments

A bivariate Markov chain approach that includes both enduring (long-term) and ephemeral (short-term) behavioral effects in models for capture-recapture experiments is proposed. The capture history of each animal is modeled as a Markov chain with a bivariate state space with states determined by the capture status (capture/noncapture) and marking status (marked/unmarked). In this framework, a conditional-likelihood method is used to estimate the population size and the transition probabilities. The classical behavioral model that assumes only an enduring behavioral effect is included as a special case of the bivariate Markovian model. Another special case that assumes only an ephemeral behavioral effect reduces to a univariate Markov chain based on capture/noncapture status. The model with the ephemeral behavioral effect is extended to incorporate time effects; in this model, in contrast to extensions of the classical behavioral model, all parameters are identifiable. A data set is analyzed to illustrate the use of the Markovian models in interpreting animals' behavioral response. Simulation results are reported to examine the performance of the estimators.