Robustness of ANCOVA in randomized trials with unequal randomization

Randomized trials with continuous outcomes are often analyzed using analysis of covariance (ANCOVA), with adjustment for prognostic baseline covariates. The ANCOVA estimator of the treatment effect is consistent under arbitrary model misspecification. In an article recently published in the journal, Wang et al proved the model-based variance estimator for the treatment effect is also consistent under outcome model misspecification, assuming the probability of randomization to each treatment is 1/2. In this reader reaction, we derive explicit expressions which show that when randomization is unequal, the model-based variance estimator can be biased upwards or downwards. In contrast, robust sandwich variance estimators can provide asymptotically valid inferences under arbitrary misspecification, even when randomization probabilities are not equal.     

Analysis of covariance in parallel-group clinical trials with pretreatment baselines

Analysis of covariance (ANCOVA) techniques are often employed in the analysis of clinical trials to try to account for the effects of varying pretreatment baseline values of an outcome variable on posttreatment measurements of the same variable. Baseline measurements of outcome variables are typically random variables, which violates the usual ANCOVA assumption that covariate values are fixed. Therefore, the usual ANCOVA hypothesis tests of treatment effects may be invalid, and the ANCOVA slope parameter estimator biased, for this application. We show, however, that if the pretreatment - posttreatment measurements have a bivariate normal distribution, then (i) the ANCOVA model with residual error independent of the covariate is a valid expression of the relationship between pretreatment and posttreatment measurements; (ii) the usual (fixed-covariate analysis) ANCOVA estimates of the slope parameter and treatment effect contrasts are unbiased; and (iii) the usual ANCOVA treatment effect contrast t-tests are valid significance tests for treatment effects. Moreover, as long as the magnitudes of the treatment effects do not depend on the "true" pretreatment value of the outcome variable, the true slope parameter must lie in the interval (0, 1) and the ANCOVA model has a clear interpretation as an adjustment (based on between- and within-subject variability) to an analysis of variance model applied to the posttreatment-pretreatment differences.     

Analysis of covariance with pre-treatment measurements in randomized trials under the cases that covariances and post-treatment variances differ between groups

When primary endpoints of randomized trials are continuous variables, the analysis of covariance (ANCOVA) with pre-treatment measurements as a covariate is often used to compare two treatment groups. In the ANCOVA, equal slopes (coefficients of pre-treatment measurements) and equal residual variances are commonly assumed. However, random allocation guarantees only equal variances of pre-treatment measurements. Unequal covariances and variances of post-treatment measurements indicate unequal slopes and, usually, unequal residual variances. For non-normal data with unequal covariances and variances of post-treatment measurements, it is known that the ANCOVA with equal slopes and equal variances using an ordinary least-squares method provides an asymptotically normal estimator for the treatment effect. However, the asymptotic variance of the estimator differs from the variance estimated from a standard formula, and its property is unclear. Furthermore, the asymptotic properties of the ANCOVA with equal slopes and unequal variances using a generalized least-squares method are unclear. In this paper, we consider non-normal data with unequal covariances and variances of post-treatment measurements, and examine the asymptotic properties of the ANCOVA with equal slopes using the variance estimated from a standard formula. Analytically, we show that the actual type I error rate, thus the coverage, of the ANCOVA with equal variances is asymptotically at a nominal level under equal sample sizes. That of the ANCOVA with unequal variances using a generalized least-squares method is asymptotically at a nominal level, even under unequal sample sizes. In conclusion, the ANCOVA with equal slopes can be asymptotically justified under random allocation.     

On the covariate-adjusted estimation for an overall treatment difference with data from a randomized comparative clinical trial

To estimate an overall treatment difference with data from a randomized comparative clinical study, baseline covariates are often utilized to increase the estimation precision. Using the standard analysis of covariance technique for making inferences about such an average treatment difference may not be appropriate, especially when the fitted model is nonlinear. On the other hand, the novel augmentation procedure recently studied, for example, by Zhang and others (2008. Improving efficiency of inferences in randomized clinical trials using auxiliary covariates. Biometrics 64, 707-715) is quite flexible. However, in general, it is not clear how to select covariates for augmentation effectively. An overly adjusted estimator may inflate the variance and in some cases be biased. Furthermore, the results from the standard inference procedure by ignoring the sampling variation from the variable selection process may not be valid. In this paper, we first propose an estimation procedure, which augments the simple treatment contrast estimator directly with covariates. The new proposal is asymptotically equivalent to the aforementioned augmentation method. To select covariates, we utilize the standard lasso procedure. Furthermore, to make valid inference from the resulting lasso-type estimator, a cross validation method is used. The validity of the new proposal is justified theoretically and empirically. We illustrate the procedure extensively with a well-known primary biliary cirrhosis clinical trial data set.     

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.     

Longitudinal analysis of pre- and post-treatment measurements with equal baseline assumptions in randomized trials

For continuous variables of randomized controlled trials, recently, longitudinal analysis of pre- and posttreatment measurements as bivariate responses is one of analytical methods to compare two treatment groups. Under random allocation, means and variances of pretreatment measurements are expected to be equal between groups, but covariances and posttreatment variances are not. Under random allocation with unequal covariances and posttreatment variances, we compared asymptotic variances of the treatment effect estimators in three longitudinal models. The data-generating model has equal baseline means and variances, and unequal covariances and posttreatment variances. The model with equal baseline means and unequal variance–covariance matrices has a redundant parameter. In large sample sizes, these two models keep a nominal type I error rate and have high efficiency. The model with equal baseline means and equal variance–covariance matrices wrongly assumes equal covariances and posttreatment variances. Only under equal sample sizes, this model keeps a nominal type I error rate. This model has the same high efficiency with the data-generating model under equal sample sizes. In conclusion, longitudinal analysis with equal baseline means performed well in large sample sizes. We also compared asymptotic properties of longitudinal models with those of the analysis of covariance (ANCOVA) and t-test.     

Sample Size Considerations for GEE Analyses of Three‐Level Cluster Randomized Trials

Summary Cluster randomized trials in health care may involve three instead of two levels, for instance, in trials where different interventions to improve quality of care are compared. In such trials, the intervention is implemented in health care units (“clusters”) and aims at changing the behavior of health care professionals working in this unit (“subjects”), while the effects are measured at the patient level (“evaluations”). Within the generalized estimating equations approach, we derive a sample size formula that accounts for two levels of clustering: that of subjects within clusters and that of evaluations within subjects. The formula reveals that sample size is inflated, relative to a design with completely independent evaluations, by a multiplicative term that can be expressed as a product of two variance inflation factors, one that quantifies the impact of within‐subject correlation of evaluations on the variance of subject‐level means and the other that quantifies the impact of the correlation between subject‐level means on the variance of the cluster means. Power levels as predicted by the sample size formula agreed well with the simulated power for more than 10 clusters in total, when data were analyzed using bias‐corrected estimating equations for the correlation parameters in combination with the model‐based covariance estimator or the sandwich estimator with a finite sample correction.     

Efficient nonparametric estimation of causal effects in randomized trials with noncompliance

Causal approaches based on the potential outcome framework providea useful tool for addressing noncompliance problems in randomizedtrials. We propose a new estimator of causal treatment effectsin randomized clinical trials with noncompliance. We use theempirical likelihood approach to construct a profile randomsieve likelihood and take into account the mixture structurein outcome distributions, so that our estimator is robust toparametric distribution assumptions and provides substantialfinite-sample efficiency gains over the standard instrumentalvariable estimator. Our estimator is asymptotically equivalentto the standard instrumental variable estimator, and it canbe applied to outcome variables with a continuous, ordinal orbinary scale. We apply our method to data from a randomizedtrial of an intervention to improve the treatment of depressionamong depressed elderly patients in primary care practices.     

Identification of causal effects using instrumental variables in randomized trials with stochastic compliance

In randomized trials with imperfect compliance, it is sometimes recommended to supplement the intention‐to‐treat estimate with an instrumental variable (IV) estimate, which is consistent for the effect of treatment administration in those subjects who would get treated if randomized to treatment and would not get treated if randomized to control. The IV estimation however has been criticized for its reliance on simultaneous existence of complementary “fatalistic” compliance states. The objective of the present paper is to identify some sufficient conditions for consistent estimation of treatment effects in randomized trials with stochastic compliance. It is shown that in the stochastic framework, the classical IV estimator is generally inconsistent for the population‐averaged treatment effect. However, even under stochastic compliance, with certain common experimental designs the IV estimator and a simple alternative estimator can be used for consistent estimation of the effect of treatment administration in well‐defined and identifiable subsets of the study population.     

Quantifying the impact of fixed effects modeling of clusters in multiple imputation for cluster randomized trials

In cluster randomized trials (CRTs), identifiable clusters rather than individuals are randomized to study groups. Resulting data often consist of a small number of clusters with correlated observations within a treatment group. Missing data often present a problem in the analysis of such trials, and multiple imputation (MI) has been used to create complete data sets, enabling subsequent analysis with well-established analysis methods for CRTs. We discuss strategies for accounting for clustering when multiply imputing a missing continuous outcome, focusing on estimation of the variance of group means as used in an adjusted t-test or ANOVA. These analysis procedures are congenial to (can be derived from) a mixed effects imputation model; however, this imputation procedure is not yet available in commercial statistical software. An alternative approach that is readily available and has been used in recent studies is to include fixed effects for cluster, but the impact of using this convenient method has not been studied. We show that under this imputation model the MI variance estimator is positively biased and that smaller intraclass correlations (ICCs) lead to larger overestimation of the MI variance. Analytical expressions for the bias of the variance estimator are derived in the case of data missing completely at random, and cases in which data are missing at random are illustrated through simulation. Finally, various imputation methods are applied to data from the Detroit Middle School Asthma Project, a recent school-based CRT, and differences in inference are compared.     

Relative efficiency of unequal versus equal cluster sizes in cluster randomized trials using generalized estimating equation models

There is growing interest in conducting cluster randomized trials (CRTs). For simplicity in sample size calculation, the cluster sizes are assumed to be identical across all clusters. However, equal cluster sizes are not guaranteed in practice. Therefore, the relative efficiency (RE) of unequal versus equal cluster sizes has been investigated when testing the treatment effect. One of the most important approaches to analyze a set of correlated data is the generalized estimating equation (GEE) proposed by Liang and Zeger, in which the “working correlation structure” is introduced and the association pattern depends on a vector of association parameters denoted by ρ. In this paper, we utilize GEE models to test the treatment effect in a two‐group comparison for continuous, binary, or count data in CRTs. The variances of the estimator of the treatment effect are derived for the different types of outcome. RE is defined as the ratio of variance of the estimator of the treatment effect for equal to unequal cluster sizes. We discuss a commonly used structure in CRTs—exchangeable, and derive the simpler formula of RE with continuous, binary, and count outcomes. Finally, REs are investigated for several scenarios of cluster size distributions through simulation studies. We propose an adjusted sample size due to efficiency loss. Additionally, we also propose an optimal sample size estimation based on the GEE models under a fixed budget for known and unknown association parameter (ρ) in the working correlation structure within the cluster.     

Propensity Score Matching in Randomized Clinical Trials

Summary Cluster randomization trials with relatively few clusters have been widely used in recent years for evaluation of health‐care strategies. On average, randomized treatment assignment achieves balance in both known and unknown confounding factors between treatment groups, however, in practice investigators can only introduce a small amount of stratification and cannot balance on all the important variables simultaneously. The limitation arises especially when there are many confounding variables in small studies. Such is the case in the INSTINCT trial designed to investigate the effectiveness of an education program in enhancing the tPA use in stroke patients. In this article, we introduce a new randomization design, the balance match weighted (BMW) design, which applies the optimal matching with constraints technique to a prospective randomized design and aims to minimize the mean squared error (MSE) of the treatment effect estimator. A simulation study shows that, under various confounding scenarios, the BMW design can yield substantial reductions in the MSE for the treatment effect estimator compared to a completely randomized or matched‐pair design. The BMW design is also compared with a model‐based approach adjusting for the estimated propensity score and Robins‐Mark‐Newey E‐estimation procedure in terms of efficiency and robustness of the treatment effect estimator. These investigations suggest that the BMW design is more robust and usually, although not always, more efficient than either of the approaches. The design is also seen to be robust against heterogeneous error. We illustrate these methods in proposing a design for the INSTINCT trial.     

Generalizing causal inferences from individuals in randomized trials to all trial‐eligible individuals

We consider methods for causal inference in randomized trials nested within cohorts of trial‐eligible individuals, including those who are not randomized. We show how baseline covariate data from the entire cohort, and treatment and outcome data only from randomized individuals, can be used to identify potential (counterfactual) outcome means and average treatment effects in the target population of all eligible individuals. We review identifiability conditions, propose estimators, and assess the estimators' finite‐sample performance in simulation studies. As an illustration, we apply the estimators in a trial nested within a cohort of trial‐eligible individuals to compare coronary artery bypass grafting surgery plus medical therapy vs. medical therapy alone for chronic coronary artery disease.     

Using Regression Models to Analyze Randomized Trials: Asymptotically Valid Hypothesis Tests Despite Incorrectly Specified Models

Summary .  Regression models are often used to test for cause-effect relationships from data collected in randomized trials or experiments. This practice has deservedly come under heavy scrutiny, because commonly used models such as linear and logistic regression will often not capture the actual relationships between variables, and incorrectly specified models potentially lead to incorrect conclusions. In this article, we focus on hypothesis tests of whether the treatment given in a randomized trial has any effect on the mean of the primary outcome, within strata of baseline variables such as age, sex, and health status. Our primary concern is ensuring that such hypothesis tests have correct type I error for large samples. Our main result is that for a surprisingly large class of commonly used regression models, standard regression-based hypothesis tests (but using robust variance estimators) are guaranteed to have correct type I error for large samples, even when the models are incorrectly specified. To the best of our knowledge, this robustness of such model-based hypothesis tests to incorrectly specified models was previously unknown for Poisson regression models and for other commonly used models we consider. Our results have practical implications for understanding the reliability of commonly used, model-based tests for analyzing randomized trials.     

Model misspecification in proportional hazards regression

The proportional hazards model is frequently used to evaluatethe effect of treatment on failure time events in randomisedclinical trials. Concomitant variables are usually availableand may be considered for use in the primary analyses underthe assumption that incorporating them may reduce bias or improveefficiency. In this paper we consider two approaches to includingcovariate information: regression modelling and stratification.We focus on the setting where covariate effects are nonproportionaland we compare the bias, efficiency and coverage propertiesof these approaches. These results indicate that our intuitionbased on linear model analysis of covariance is misleading.Covariate adjustment in proportional hazards models has littleeffect on the variance but may significantly improve the accuracyof the treatment effect estimator.     

Improving efficiency of inferences in randomized clinical trials using auxiliary covariates

Summary .   The primary goal of a randomized clinical trial is to make comparisons among two or more treatments. For example, in a two-arm trial with continuous response, the focus may be on the difference in treatment means; with more than two treatments, the comparison may be based on pairwise differences. With binary outcomes, pairwise odds ratios or log odds ratios may be used. In general, comparisons may be based on meaningful parameters in a relevant statistical model. Standard analyses for estimation and testing in this context typically are based on the data collected on response and treatment assignment only. In many trials, auxiliary baseline covariate information may also be available, and it is of interest to exploit these data to improve the efficiency of inferences. Taking a semiparametric theory perspective, we propose a broadly applicable approach to adjustment for auxiliary covariates to achieve more efficient estimators and tests for treatment parameters in the analysis of randomized clinical trials. Simulations and applications demonstrate the performance of the methods.     

Identification of subpopulations with distinct treatment benefit rate using the Bayesian tree

Characterization of a subpopulation by the difference in marginal means of the outcome under the intervention and control may not be sufficient to provide informative guidance for individual decision and public policy making. Specifically, often we are interested in the treatment benefit rate (TBR), that is, the probability of benefitting an intervention in a meaningful way. For binary outcomes, TBR is the proportion that has “unfavorable” outcome under the control and “favorable” outcome under the intervention. Identification of subpopulations with distinct TBR by baseline characteristics will have significant implications in clinical setting where a medical intervention with potential negative health impact is under consideration for a given patient. In addition, these subpopulations with unique TBR set the basis for guidance in implementing the intervention toward a more personalized scheme of treatment. In this article, we propose a Bayesian tree based latent variable model to seek subpopulations with distinct TBR. Our method offers a nonparametric Bayesian framework that accounts for the uncertainty in estimating potential outcomes and allows more exhaustive search of the partitions of the baseline covariates space. The method is evaluated through a simulation study and applied to a randomized clinical trial of implantable cardioverter defibrillators to reduce mortality.     

Nonparametric analysis of covariance for comparing change in randomized studies with baseline values subject to error

Change from baseline to a follow-up examination can be compared among two or more randomly assigned treatment groups by using analysis of variance on the change scores. However, a generally more sensitive (powerful) test can be performed using analysis of covariance (ANOVA) on the follow-up data with the baseline data as a covariate. This approach is not without potential problems, though. The assumption of ordinary ANCOVA of normally distributed errors is speculative for many variables employed in biomedical research. Furthermore, the baseline values are inevitably random variables and often are measured with error. This report investigates, in this situation, the validity and relative power of the ordinary ANCOVA test and two asymptotically distribution-free alternative tests, one based on the rank transformation and the other based on the normal scores transformation. The procedures are illustrated with data from a clinical trial. Normal and several nonnormal distributions, as well as varying degree of variable error, are studied by Monte Carlo methods. The normal scores test is generally recommended for statistical practice.     

Improved confidence intervals for a difference of two cause-specific cumulative incidence functions estimated in the presence of competing risks and random censoring

A cause-specific cumulative incidence function (CIF) is the probability of failure from a specific cause as a function of time. In randomized trials, a difference of cause-specific CIFs (treatment minus control) represents a treatment effect. Cause-specific CIF in each intervention arm can be estimated based on the usual non-parametric Aalen–Johansen estimator which generalizes the Kaplan–Meier estimator of CIF in the presence of competing risks. Under random censoring, asymptotically valid Wald-type confidence intervals (CIs) for a difference of cause-specific CIFs at a specific time point can be constructed using one of the published variance estimators. Unfortunately, these intervals can suffer from substantial under-coverage when the outcome of interest is a rare event, as may be the case for example in the analysis of uncommon adverse events. We propose two new approximate interval estimators for a difference of cause-specific CIFs estimated in the presence of competing risks and random censoring. Theoretical analysis and simulations indicate that the new interval estimators are superior to the Wald CIs in the sense of avoiding substantial under-coverage with rare events, while being equivalent to the Wald CIs asymptotically. In the absence of censoring, one of the two proposed interval estimators reduces to the well-known Agresti–Caffo CI for a difference of two binomial parameters. The new methods can be easily implemented with any software package producing point and variance estimates for the Aalen–Johansen estimator, as illustrated in a real data example.