Non-homogeneous Markov processes for biomedical data analysis

Some time ago, the Markov processes were introduced in biomedical sciences in order to study disease history events. Homogeneous and Non-homogeneous Markov processes are an important field of research into stochastic processes, especially when exact transition times are unknown and interval-censored observations are present in the analysis. Non-homogeneous Markov process should be used when the homogeneous assumption is too strong. However these sorts of models increase the complexity of the analysis and standard software is limited. In this paper, some methods for fitting non-homogeneous Markov models are reviewed and an algorithm is proposed for biomedical data analysis. The method has been applied to analyse breast cancer data. Specific software for this purpose has been implemented.     

Modeling nonhomogeneous Markov processes via time transformation

Summary .   Longitudinal studies are a powerful tool for characterizing the course of chronic disease. These studies are usually carried out with subjects observed at periodic visits giving rise to panel data. Under this observation scheme the exact times of disease state transitions and sequence of disease states visited are unknown and Markov process models are often used to describe disease progression. Most applications of Markov process models rely on the assumption of time homogeneity, that is, that the transition rates are constant over time. This assumption is not satisfied when transition rates depend on time from the process origin. However, limited statistical tools are available for dealing with nonhomogeneity. We propose models in which the time scale of a nonhomogeneous Markov process is transformed to an operational time scale on which the process is homogeneous. We develop a method for jointly estimating the time transformation and the transition intensity matrix for the time transformed homogeneous process. We assess maximum likelihood estimation using the Fisher scoring algorithm via simulation studies and compare performance of our method to homogeneous and piecewise homogeneous models. We apply our methodology to a study of delirium progression in a cohort of stem cell transplantation recipients and show that our method identifies temporal trends in delirium incidence and recovery.     

Maximum likelihood estimation in random effects cure rate models with nonignorable missing covariates

We introduce a method of parameter estimation for a random effects cure rate model. We also propose a methodology that allows us to account for nonignorable missing covariates in this class of models. The proposed method corrects for possible bias introduced by complete case analysis when missing data are not missing completely at random and is motivated by data from a pair of melanoma studies conducted by the Eastern Cooperative Oncology Group in which clustering by cohort or time of study entry was suspected. In addition, these models allow estimation of cure rates, which is desirable when we do not wish to assume that all subjects remain at risk of death or relapse from disease after sufficient follow-up. We develop an EM algorithm for the model and provide an efficient Gibbs sampling scheme for carrying out the E-step of the algorithm.     

Stochastic algorithms for Markov models estimation with intermittent missing data

Multistate Markov models are frequently used to characterize disease processes, but their estimation from longitudinal data is often hampered by complex patterns of incompleteness. Two algorithms for estimating Markov chain models in the case of intermittent missing data in longitudinal studies, a stochastic EM algorithm and the Gibbs sampler, are described. The first can be viewed as a random perturbation of the EM algorithm and is appropriate when the M step is straightforward but the E step is computationally burdensome. It leads to a good approximation of the maximum likelihood estimates. The Gibbs sampler is used for a full Bayesian inference. The performances of the two algorithms are illustrated on two simulated data sets. A motivating example concerned with the modelling of the evolution of parasitemia by Plasmodium falciparum (malaria) in a cohort of 105 young children in Cameroon is described and briefly analyzed.     

Hidden Markov models for settings with interval-censored transition times and uncertain time origin: application to HIV genetic analyses

The simplicity and flexibility of Markov models make them appealing for investigations of the acquisition of HIV drug-resistance mutations, whose presence can define specific Markov states. Because the exact time of acquiring a mutation is not observed during clinical research studies on HIV infection, it is important that methods for fitting such models accommodate interval-censored transition times. Furthermore, many such studies include patients with extensive treatment experience prior to the onset of the studies. Therefore, the virus in these patients may have already acquired resistance mutations by study entry. Retrospective data regarding the time on treatment, which is often known or known with error, provide information about the acquisition rates before the start of a study. Finally, variability in the genetic sequences of circulating HIV creates uncertainty in the Markov states. This paper considers approaches to fitting Markov models to data with interval-censored transition times when the time origin and the Markov states are known with error. The methods were applied to AIDS Clinical Trial Group protocol 398, a randomized comparison of mono- versus dual-protease inhibitor use in heavily pretreated patients. We found that the estimated rates of acquiring certain classes of mutations depended on the presence of others, and that the precision of these estimates can be considerably improved by inclusion of retrospective data.     

Signal extraction from long-term ecological data using Bayesian and non-Bayesian state-space models

Emerging ecological time series from long-term ecological studies and remote sensing provide excellent opportunities for ecologists to study the dynamic patterns and governing processes of ecological systems. However, signal extraction from long-term time series often requires system learning (e.g., estimation of true system state) to process the large amount of information, to reconstruct system state, to account for measurement error, and to handle missing data. State-space models (SSMs) are a natural choice for these tasks and thus have received increasing attention in ecological and environmental studies. Data-based learning using SSMs that connect ecological processes to the measurement of system state becomes a useful technique in the ecological informatics toolkit. The present study illustrates the use of the Kalman filter (KF), an estimator of SSMs, with case studies of population dynamics. The examples of the SSM applications include the reconstruction of system state using the KF method and Markov chain Monte Carlo methods, estimation of measurement-error variances in the estimates of animal population abundance using basic structural models (BSMs), and estimation of missing values using the KF and Kalman smoother. Estimation of measurement-error variances by BSMs does not require knowledge of the functional form that generates the time series data. Instead, BSMs approximate the trajectory or deterministic skeleton of a system dynamics in a semi-parametric fashion, and provide a robust estimator of measurement-error variances. The present study also compares Bayesian SSMs with non-Bayesian SSMs. The joint use of the KF method or its extensions and Markov chain Monte Carlo (MCMC) methods is a promising approach to the parameter estimation of SSMs.     

Semi‐Markov Models with Phase‐Type Sojourn Distributions

Summary Continuous‐time multistate models are widely used for categorical response data, particularly in the modeling of chronic diseases. However, inference is difficult when the process is only observed at discrete time points, with no information about the times or types of events between observation times, unless a Markov assumption is made. This assumption can be limiting as rates of transition between disease states might instead depend on the time since entry into the current state. Such a formulation results in a semi‐Markov model. We show that the computational problems associated with fitting semi‐Markov models to panel‐observed data can be alleviated by considering a class of semi‐Markov models with phase‐type sojourn distributions. This allows methods for hidden Markov models to be applied. In addition, extensions to models where observed states are subject to classification error are given. The methodology is demonstrated on a dataset relating to development of bronchiolitis obliterans syndrome in post‐lung‐transplantation patients.     

Estimating transition rates in aggregated Markov models of ion channel gating with loops and with nearly equal dwell times

A typical task in the application of aggregated Markov models to ion channel data is the estimation of the transition rates between the states. Realistic models for ion channel data often have one or more loops. We show that the transition rates of a model with loops are not identifiable if the model has either equal open or closed dwell times. This non-identifiability of the transition rates also has an effect on the estimation of the transition rates for models which are not subject to the constraint of either equal open or closed dwell times. If a model with loops has nearly equal dwell times, the Hessian matrix of its likelihood function will be ill-conditioned and the standard deviations of the estimated transition rates become extraordinarily large for a number of data points which are typically recorded in experiments.     

Stochastic modelling of a single ion channel: an alternating renewal approach with application to limited time resolution

Stochastic models of ion channels have been based largely on Markov theory where individual states and transition rates must be specified, and sojourn-time densities for each state are constrained to be exponential. This study presents an approach based on random-sum methods and alternating-renewal theory, allowing individual states to be grouped into classes provided the successive sojourn times in a given class are independent and identically distributed. Under these conditions Markov models form a special case. The utility of the approach is illustrated by considering the effects of limited time resolution (modelled by using a discrete detection limit, xi) on the properties of observable events, with emphasis on the observed open-time (xi-open-time). The cumulants and Laplace transform for a xi-open-time are derived for a range of Markov and non-Markov models; several useful approximations to the xi-open-time density function are presented. Numerical studies show that the effects of limited time resolution can be extreme, and also highlight the relative importance of the various model parameters. The theory could form a basis for future inferential studies in which parameter estimation takes account of limited time resolution in single channel records. Appendixes include relevant results concerning random sums and a discussion of the role of exponential distributions in Markov models.     

Bayesian latent multi‐state modeling for nonequidistant longitudinal electronic health records

Large amounts of longitudinal health records are now available for dynamic monitoring of the underlying processes governing the observations. However, the health status progression across time is not typically observed directly: records are observed only when a subject interacts with the system, yielding irregular and often sparse observations. This suggests that the observed trajectories should be modeled via a latent continuous‐time process potentially as a function of time‐varying covariates. We develop a continuous‐time hidden Markov model to analyze longitudinal data accounting for irregular visits and different types of observations. By employing a specific missing data likelihood formulation, we can construct an efficient computational algorithm. We focus on Bayesian inference for the model: this is facilitated by an expectation‐maximization algorithm and Markov chain Monte Carlo methods. Simulation studies demonstrate that these approaches can be implemented efficiently for large data sets in a fully Bayesian setting. We apply this model to a real cohort where patients suffer from chronic obstructive pulmonary disease with the outcome being the number of drugs taken, using health care utilization indicators and patient characteristics as covariates.     

A continuous time version and a generalization of a Markov-recapture model for trapping experiments

Wileyto et al. [E.P. Wileyto, W.J. Ewens, M.A. Mullen, Markov-recapture population estimates: a tool for improving interpretation of trapping experiments, Ecology 75 (1994) 1109] propose a four-state discrete time Markov process, which describes the structure of a marking-capture experiment as a method of population estimation. They propose this method primarily for estimation of closed insect populations. Their method provides a mark-recapture estimate from a single trap observation by allowing subjects to mark themselves. The estimate of the unknown population size is based on the assumption of a closed population and a simple Markov model in which the rates of marking, capture, and recapture are assumed to be equal. Using the one step transition probability matrix of their model, we illustrate how to go from an embedded discrete time Markov process to a continuous time Markov process assuming exponentially distributed holding times. We also compute the transition probabilities after time t for the continuous time case and compare the limiting behavior of the continuous and discrete time processes. Finally, we generalize their model by relaxing the assumption of equal per capita rates for marking, capture, and recapture. Other questions about how their results change when using a continuous time Markov process are examined.     

Sampling errors create bias in Markov models for community dynamics: the problem and a method for its solution

Repeated, spatially explicit sampling is widely used to characterize the dynamics of sessile communities in both terrestrial and aquatic systems, yet our understanding of the consequences of errors made in such sampling is limited. In particular, when Markov transition probabilities are calculated by tracking individual points over time, misidentification of the same spatial locations will result in biased estimates of transition probabilities, successional rates, and community trajectories. Nonetheless, to date, all published studies that use such data have implicitly assumed that resampling occurs without error when making estimates of transition rates. Here, we develop and test a straightforward maximum likelihood approach, based on simple field estimates of resampling errors, to arrive at corrected estimates of transition rates between species in a rocky intertidal community. We compare community Markov models based on raw and corrected transition estimates using data from Endocladia muricata-dominated plots in a California intertidal assemblage, finding that uncorrected predictions of succession consistently overestimate recovery time. We tested the precision and accuracy of the approach using simulated datasets and found good performance of our estimation method over a range of realistic sample sizes and error rates.     

Estimation in a semi-Markov transformation model

Semi-Markov and modulated renewal processes provide a large class of multi-state models which can be used for analysis of longitudinal failure time data. In biomedical applications, models of this kind are often used to describe evolution of a disease and assume that patient may move among a finite number of states representing different phases in the disease progression. Several authors proposed extensions of the proportional hazard model for regression analysis of these processes. In this paper, we consider a general class of censored semi-Markov and modulated renewal processes and propose use of transformation models for their analysis. Special cases include modulated renewal processes with interarrival times specified using transformation models, and semi-Markov processes with with one-step transition probabilities defined using copula-transformation models. We discuss estimation of finite and infinite dimensional parameters and develop an extension of the Gaussian multiplier method for setting confidence bands for transition probabilities and related parameters. A transplant outcome data set from the Center for International Blood and Marrow Transplant Research is used for illustrative purposes.     

Bias in Markov models of disease

We examine bias in Markov models of diseases, including both chronic and infectious diseases. We consider two common types of Markov disease models: ones where disease progression changes by severity of disease, and ones where progression of disease changes in time or by age. We find sufficient conditions for bias to exist in models with aggregated transition probabilities when compared to models with state/time dependent transition probabilities. We also find that when aggregating data to compute transition probabilities, bias increases with the degree of data aggregation. We illustrate by examining bias in Markov models of Hepatitis C, Alzheimer’s disease, and lung cancer using medical data and find that the bias is significant depending on the method used to aggregate the data. A key implication is that by not incorporating state/time dependent transition probabilities, studies that use Markov models of diseases may be significantly overestimating or underestimating disease progression. This could potentially result in incorrect recommendations from cost-effectiveness studies and incorrect disease burden forecasts.     

Markov population processes as models of primate social and population dynamics

Thanks to recently developed theory of Markov population processes, models of how an individual primate migrates from one casual social group to another or from one breeding troop to another can now deal exactly with transition rates which depend nonlinearly on the sizes of both the group (or troop) left and the group (or troop) entered. Examples of such models presented here are consistent with existing observations of primate social and population dynamics and are more plausible as explanations of these data than previous linear models.     

Multi-Stage Markov Analysis of Progressive Disease Applied to Melanoma

A fundamental research goal in clinical studies of progressive, multi-stage disease is to understand its natural history and its relationship with prognostic factors. Our current understanding of this topic is based on the use of two-stage methods for event-time analysis which neglect intermediate transition information. In contrast, a multi-stage model utilizes all available data and provides more accurate insight into disease progression. We specify a forward-flowing multi-stage Markov model based on the discrete clinical stages of disease. By assuming the process to be Markovian, we avoid unnecessary complications to our numerical estimation procedure. Due to noncontinuous patient monitoring and the chronic nature of progressive disease, heavy right- and interval-censoring exists in the transition data. We develop a modified ECM algorithm to numerically carry out the otherwise complicated parameter estimation for this process. We also identify significant prognostic factors relevant to each transition, along with the relative importance of each prognostic factor. The numerical estimation is stable, and the parameter estimates are maximum likelihood estimates (Meng, 1990). In general our forward-flowing multi-stage models provide a flexible framework for the study of the effects of prognostic factors on progression among several stages. We apply our Markov model to a dataset of malignant melanoma patients, and present an inferential discussion. Results from our multi-stage Markov model provide an improved understanding of melanoma progression.     

Flexible nonhomogeneous markov models for panel observed data

Methods for fitting nonhomogeneous Markov models to panel-observed data using direct numerical solution to the Kolmogorov Forward equations are developed. Nonhomogeneous Markov models occur most commonly when baseline transition intensities depend on calendar time, but may also occur with deterministic time-dependent covariates such as age. We propose transition intensities based on B-splines as a smooth alternative to piecewise constant intensities and also as a generalization of time transformation models. An expansion of the system of differential equations allows first derivatives of the likelihood to be obtained, which can be used in a Fisher scoring algorithm for maximum likelihood estimation. The method is evaluated through a small simulation study and demonstrated on data relating to the development of cardiac allograft vasculopathy in posttransplantation patients.     

Modelling larval movement data from individual bioassays

We consider modelling the movements of larvae using individual bioassays in which data are collected at a high‐frequency rate of five observations per second. The aim is to characterize the behaviour of the larvae when exposed to attractant and repellent compounds. Mixtures of diffusion processes, as well as Hidden Markov models, are proposed as models of larval movement. These models account for directed and localized movements, and successfully distinguish between the behaviour of larvae exposed to attractant and repellent compounds. A simulation study illustrates the advantage of using a Hidden Markov model rather than a simpler mixture model. Practical aspects of model estimation and inference are considered on extensive data collected in a study of novel approaches for the management of cabbage root fly.     

Infinite hidden Markov models for multiple multivariate time series with missing data

Exposure to air pollution is associated with increased morbidity and mortality. Recent technological advancements permit the collection of time-resolved personal exposure data. Such data are often incomplete with missing observations and exposures below the limit of detection, which limit their use in health effects studies. In this paper, we develop an infinite hidden Markov model for multiple asynchronous multivariate time series with missing data. Our model is designed to include covariates that can inform transitions among hidden states. We implement beam sampling, a combination of slice sampling and dynamic programming, to sample the hidden states, and a Bayesian multiple imputation algorithm to impute missing data. In simulation studies, our model excels in estimating hidden states and state-specific means and imputing observations that are missing at random or below the limit of detection. We validate our imputation approach on data from the Fort Collins Commuter Study. We show that the estimated hidden states improve imputations for data that are missing at random compared to existing approaches. In a case study of the Fort Collins Commuter Study, we describe the inferential gains obtained from our model including improved imputation of missing data and the ability to identify shared patterns in activity and exposure among repeated sampling days for individuals and among distinct individuals.     

Missing covariates in longitudinal data with informative dropouts: bias analysis and inference

We consider estimation in generalized linear mixed models (GLMM) for longitudinal data with informative dropouts. At the time a unit drops out, time-varying covariates are often unobserved in addition to the missing outcome. However, existing informative dropout models typically require covariates to be completely observed. This assumption is not realistic in the presence of time-varying covariates. In this article, we first study the asymptotic bias that would result from applying existing methods, where missing time-varying covariates are handled using naive approaches, which include: (1) using only baseline values; (2) carrying forward the last observation; and (3) assuming the missing data are ignorable. Our asymptotic bias analysis shows that these naive approaches yield inconsistent estimators of model parameters. We next propose a selection/transition model that allows covariates to be missing in addition to the outcome variable at the time of dropout. The EM algorithm is used for inference in the proposed model. Data from a longitudinal study of human immunodeficiency virus (HIV)-infected women are used to illustrate the methodology.