Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review,reanalysis, and simulation study

BackgroundCluster randomised trials (CRTs) are commonly analysed using mixed-effects models or generalised estimating equations (GEEs). However, these analyses do not always perform well with the small number of clusters typical of most CRTs. They can lead to increased risk of a type I error (finding a statistically significant treatment effect when it does not exist) if appropriate corrections are not used.MethodsWe conducted a small simulation study to evaluate the impact of using small-sample corrections for mixed-effects models or GEEs in CRTs with a small number of clusters. We then reanalysed data from TRIGGER, a CRT with six clusters, to determine the effect of using an inappropriate analysis method in practice. Finally, we reviewed 100 CRTs previously identified by a search on PubMed in order to assess whether trials were using appropriate methods of analysis. Trials were classified as at risk of an increased type I error rate if they did not report using an analysis method which accounted for clustering, or if they had fewer than 40 clusters and performed an individual-level analysis without reporting the use of an appropriate small-sample correction.ResultsOur simulation study found that using mixed-effects models or GEEs without an appropriate correction led to inflated type I error rates, even for as many as 70 clusters. Conversely, using small-sample corrections provided correct type I error rates across all scenarios. Reanalysis of the TRIGGER trial found that inappropriate methods of analysis gave much smaller P values (P ≤ 0.01) than appropriate methods (P = 0.04–0.15). In our review, of the 99 trials that reported the number of clusters, 64 (65 %) were at risk of an increased type I error rate; 14 trials did not report using an analysis method which accounted for clustering, and 50 trials with fewer than 40 clusters performed an individual-level analysis without reporting the use of an appropriate correction.ConclusionsCRTs with a small or medium number of clusters are at risk of an inflated type I error rate unless appropriate analysis methods are used. Investigators should consider using small-sample corrections with mixed-effects models or GEEs to ensure valid results.     

Power calculation for cross-sectional stepped wedge cluster randomized trials with variable cluster sizes

Standard sample size calculation formulas for stepped wedge cluster randomized trials (SW-CRTs) assume that cluster sizes are equal. When cluster sizes vary substantially, ignoring this variation may lead to an under-powered study. We investigate the relative efficiency of a SW-CRT with varying cluster sizes to equal cluster sizes, and derive variance estimators for the intervention effect that account for this variation under a mixed effects model—a commonly used approach for analyzing data from cluster randomized trials. When cluster sizes vary, the power of a SW-CRT depends on the order in which clusters receive the intervention, which is determined through randomization. We first derive a variance formula that corresponds to any particular realization of the randomized sequence and propose efficient algorithms to identify upper and lower bounds of the power. We then obtain an “expected” power based on a first-order approximation to the variance formula, where the expectation is taken with respect to all possible randomization sequences. Finally, we provide a variance formula for more general settings where only the cluster size arithmetic mean and coefficient of variation, instead of exact cluster sizes, are known in the design stage. We evaluate our methods through simulations and illustrate that the average power of a SW-CRT decreases as the variation in cluster sizes increases, and the impact is largest when the number of clusters is small.     

Quantifying the impact of fixed effects modeling of clusters in multiple imputation for cluster randomized trials

In cluster randomized trials (CRTs), identifiable clusters rather than individuals are randomized to study groups. Resulting data often consist of a small number of clusters with correlated observations within a treatment group. Missing data often present a problem in the analysis of such trials, and multiple imputation (MI) has been used to create complete data sets, enabling subsequent analysis with well-established analysis methods for CRTs. We discuss strategies for accounting for clustering when multiply imputing a missing continuous outcome, focusing on estimation of the variance of group means as used in an adjusted t-test or ANOVA. These analysis procedures are congenial to (can be derived from) a mixed effects imputation model; however, this imputation procedure is not yet available in commercial statistical software. An alternative approach that is readily available and has been used in recent studies is to include fixed effects for cluster, but the impact of using this convenient method has not been studied. We show that under this imputation model the MI variance estimator is positively biased and that smaller intraclass correlations (ICCs) lead to larger overestimation of the MI variance. Analytical expressions for the bias of the variance estimator are derived in the case of data missing completely at random, and cases in which data are missing at random are illustrated through simulation. Finally, various imputation methods are applied to data from the Detroit Middle School Asthma Project, a recent school-based CRT, and differences in inference are compared.     

A mixed model-based variance estimator for marginal model analyses of cluster randomized trials

Generalized estimating equations (GEE) are used in the analysis of cluster randomized trials (CRTs) because: 1) the resulting intervention effect estimate has the desired marginal or population-averaged interpretation, and 2) most statistical packages contain programs for GEE. However, GEE tends to underestimate the standard error of the intervention effect estimate in CRTs. In contrast, penalized quasi-likelihood (PQL) estimates the standard error of the intervention effect in CRTs much better than GEE but is used less frequently because: 1) it generates an intervention effect estimate with a conditional, or cluster-specific, interpretation, and 2) PQL is not a part of most statistical packages. We propose taking the variance estimator from PQL and re-expressing it as a sandwich-type estimator that could be easily incorporated into existing GEE packages, thereby making GEE useful for the analysis of CRTs. Using numerical examples and data from an actual CRT, we compare the performance of this variance estimator to others proposed in the literature, and we find that our variance estimator performs as well as or better than its competitors.     

Power analysis for cluster randomized trials with multiple binary co-primary endpoints

Cluster randomized trials (CRTs) are widely used in different areas of medicine and public health. Recently, with increasing complexity of medical therapies and technological advances in monitoring multiple outcomes, many clinical trials attempt to evaluate multiple co-primary endpoints. In this study, we present a power analysis method for CRTs with $K\ge 2$ binary co-primary endpoints. It is developed based on the GEE (generalized estimating equation) approach, and three types of correlations are considered: inter-subject correlation within each endpoint, intra-subject correlation across endpoints, and inter-subject correlation across endpoints. A closed-form joint distribution of the K test statistics is derived, which facilitates the evaluation of power and type I error for arbitrarily constructed hypotheses. We further present a theorem that characterizes the relationship between various correlations and testing power. We assess the performance of the proposed power analysis method based on extensive simulation studies. An application example to a real clinical trial is presented.     

Improving sandwich variance estimation for marginal Cox analysis of cluster randomized trials

Cluster randomized trials (CRTs) frequently recruit a small number of clusters, therefore necessitating the application of small-sample corrections for valid inference. A recent systematic review indicated that CRTs reporting right-censored, time-to-event outcomes are not uncommon and that the marginal Cox proportional hazards model is one of the common approaches used for primary analysis. While small-sample corrections have been studied under marginal models with continuous, binary, and count outcomes, no prior research has been devoted to the development and evaluation of bias-corrected sandwich variance estimators when clustered time-to-event outcomes are analyzed by the marginal Cox model. To improve current practice, we propose nine bias-corrected sandwich variance estimators for the analysis of CRTs using the marginal Cox model and report on a simulation study to evaluate their small-sample properties. Our results indicate that the optimal choice of bias-corrected sandwich variance estimator for CRTs with survival outcomes can depend on the variability of cluster sizes and can also slightly differ whether it is evaluated according to relative bias or type I error rate. Finally, we illustrate the new variance estimators in a real-world CRT where the conclusion about intervention effectiveness differs depending on the use of small-sample bias corrections. The proposed sandwich variance estimators are implemented in an R package CoxBcv .     

On small‐sample inference in group randomized trials with binary outcomes and cluster‐level covariates

Group randomized trials (GRTs) randomize groups, or clusters, of people to intervention or control arms. To test for the effectiveness of the intervention when subject‐level outcomes are binary, and while fitting a marginal model that adjusts for cluster‐level covariates and utilizes a logistic link, we develop a pseudo‐Wald statistic to improve inference. Alternative Wald statistics could employ bias‐corrected empirical sandwich standard error estimates, which have received limited attention in the GRT literature despite their broad utility and applicability in our settings of interest. The test could also be carried out using popular approaches based upon cluster‐level summary outcomes. A simulation study covering a variety of realistic GRT settings is used to compare the accuracy of these methods in terms of producing nominal test sizes. Tests based upon the pseudo‐Wald statistic and a cluster‐level summary approach utilizing the natural log of observed cluster‐level odds worked best. Due to weighting, some popular cluster‐level summary approaches were found to lead to invalid inference in many settings. Finally, although use of bias‐corrected empirical sandwich standard error estimates did not consistently result in nominal sizes, they did work well, thus supporting the applicability of marginal models in GRT settings.     

Sample size estimation for survival outcomes in cluster-randomized studies with small cluster sizes

We present a method for computing sample size for cluster-randomized studies involving a large number of clusters with relatively small numbers of observations within each cluster. For multivariate survival data, only the marginal bivariate distribution is assumed to be known. The validity of this assumption is also discussed.     

A powerful and robust test statistic for randomization inference in group-randomized trials with matched pairs of groups

For group-randomized trials, randomization inference based on rank statistics provides robust, exact inference against nonnormal distributions. However, in a matched-pair design, the currently available rank-based statistics lose significant power compared to normal linear mixed model (LMM) test statistics when the LMM is true. In this article, we investigate and develop an optimal test statistic over all statistics in the form of the weighted sum of signed Mann-Whitney-Wilcoxon statistics under certain assumptions. This test is almost as powerful as the LMM even when the LMM is true, but it is much more powerful for heavy tailed distributions. A simulation study is conducted to examine the power.     

Comparing biomarker measurements to a normal range: when to use standard error of the mean (SEM) or standard deviation (SD) confidence intervals tests

This commentary is the second of a series outlining one specific concept in interpreting biomarkers data. In the first, an observational method was presented for assessing the distribution of measurements before making parametric calculations. Here, the discussion revolves around the next step, the choice of using standard error of the mean or the calculated standard deviation to compare or predict measurement results     

The impact of a knowledge translation intervention employing educational outreach and a point-of-care reminder tool vs standard lay health worker training on tuberculosis treatment completion rates: study protocol for a cluster randomized controlled trial

BackgroundDespite availability of effective treatment, tuberculosis (TB) remains an important cause of morbidity and mortality globally, with low- and middle-income countries most affected. In many such settings, including Malawi, the high burden of disease and severe shortage of skilled healthcare workers has led to task-shifting of outpatient TB care to lay health workers (LHWs). LHWs improve access to healthcare and some outcomes, including TB completion rates, but lack of training and supervision limit their impact. The goals of this study are to improve TB care provided by LHWs in Malawi by refining, implementing, and evaluating a knowledge translation strategy designed to address a recognized gap in LHWs’ TB and job-specific knowledge and, through this, to improve patient outcomes.Methods/designWe are employing a mixed-methods design that includes a pragmatic cluster randomized controlled trial and a process evaluation using qualitative methods. Trial participants will include all health centers providing TB care in four districts in the South East Zone of Malawi. The intervention employs educational outreach, a point-of-care reminder tool, and a peer support network. The primary outcome is proportion of treatment successes, defined as the total of TB patients cured or completing treatment, with outcomes taken from Ministry of Health treatment records. With an alpha of 0.05, power of 0.80, a baseline treatment success of 0.80, intraclass correlation coefficient of 0.1 based on our pilot study, and an estimated 100 clusters (health centers providing TB care), a minimum of 6 patients per cluster is required to detect a clinically significant 0.10 increase in the proportion of treatment successes. Our process evaluation will include interviews with LHWs and patients, and a document analysis of LHW training logs, quarterly peer trainer meetings, and mentorship meeting notes. An estimated 10–15 LHWs and 10–15 patients will be required to reach saturation in each of 2 planned interview periods, for a total of 40–60 interview participants.DiscussionThis study will directly inform the efforts of knowledge users within TB care and, through extension of the approach, other areas of care provided by LHWs in Malawi and other low- and middle-income countries.

Trial registration

ClinicalTrials.gov {"type":"clinical-trial","attrs":{"text":"NCT02533089","term_id":"NCT02533089"}}NCT02533089. Registered 20 August 2015. Protocol Date/Version 29 May 2016/Version 2.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-016-1563-2) contains supplementary material, which is available to authorized users.     

Inference for the effect of treatment on survival probability in randomized trials with noncompliance and administrative censoring

In many clinical studies with a survival outcome, administrative censoring occurs when follow-up ends at a prespecified date and many subjects are still alive. An additional complication in some trials is that there is noncompliance with the assigned treatment. For this setting, we study the estimation of the causal effect of treatment on survival probability up to a given time point among those subjects who would comply with the assignment to both treatment and control. We first discuss the standard instrumental variable (IV) method for survival outcomes and parametric maximum likelihood methods, and then develop an efficient plug-in nonparametric empirical maximum likelihood estimation (PNEMLE) approach. The PNEMLE method does not make any assumptions on outcome distributions, and makes use of the mixture structure in the data to gain efficiency over the standard IV method. Theoretical results of the PNEMLE are derived and the method is illustrated by an analysis of data from a breast cancer screening trial. From our limited mortality analysis with administrative censoring times 10 years into the follow-up, we find a significant benefit of screening is present after 4 years (at the 5% level) and this persists at 10 years follow-up.     

A causal proportional hazards estimator for the effect of treatment actually received in a randomized trial with all-or-nothing compliance

Survival data from randomized trials are most often analyzed in a proportional hazards (PH) framework that follows the intention-to-treat (ITT) principle. When not all the patients on the experimental arm actually receive the assigned treatment, the ITT-estimator mixes its effect on treatment compliers with its absence of effect on noncompliers. The structural accelerated failure time (SAFT) models of Robins and Tsiatis are designed to consistently estimate causal effects on the treated, without direct assumptions about the compliance selection mechanism. The traditional PH-model, however, has not yet led to such causal interpretation. In this article, we examine a PH-model of treatment effect on the treated subgroup. While potential treatment compliance is unobserved in the control arm, we derive an estimating equation for the Compliers PROPortional Hazards Effect of Treatment (C-PROPHET). The jackknife is used for bias correction and variance estimation. The method is applied to data from a recently finished clinical trial in cancer patients with liver metastases.     

Presbyopia and heat: changes associated with aging of the human lens suggest a functional role for the small heat shock protein, alpha-crystallin, in maintaining lens flexibility

Presbyopia, the inability to focus up close, affects everyone by age 50 and is the most common eye condition. It is thought to result from changes to the lens over time making it less flexible. We present evidence that presbyopia may be the result of age-related changes to the proteins of the lens fibre cells. Specifically, we show that there is a progressive decrease in the concentration of the chaperone, α-crystallin, in human lens nuclei with age, as it becomes incorporated into high molecular weight aggregates and insoluble protein. This is accompanied by a large increase in lens stiffness. Stiffness increases even more dramatically after middle age following the disappearance of free soluble α-crystallin from the centre of the lens. These alterations in α-crystallin and aggregated protein in human lenses can be reproduced simply by exposing intact pig lenses to elevated temperatures, for example, 50 °C. In this model system, the same protein changes are also associated with a progressive increase in lens stiffness. These data suggest a functional role for α-crystallin in the human lens acting as a small heat shock protein and helping to maintain lens flexibility. Presbyopia may be the result of a loss of α-crystallin coupled with progressive heat-induced denaturation of structural proteins in the lens during the first five decades of life.