A homogeneity test in overviews with group sequentially monitored clinical trials

It is known that using statistical stopping rules in clinical trials can create an artificial heterogeneity of treatment effects in overviews of related trials (Hughes, Freedman, and Pocock, 1992, Biometrics 48, 41-53). If the true treatment effect being tested is small, as is often the case, the homogeneity test by DerSimonian and Laird (1986, Controlled Clinical Trials 7, 177-188) violates the size of the test very severely. This paper provides a new homogeneity test, which preserves the size of the test more accurately. The operating characteristics of the new test are examined through simulations.     

Flexible implementations of group sequential stopping rules using constrained boundaries

Group sequential stopping rules are often used during the conduct of clinical trials in order to attain more ethical treatment of patients and to better address efficiency concerns. Because the use of such stopping rules materially affects the frequentist operating characteristics of the hypothesis test, it is necessary to choose an appropriate stopping rule during the planning of the study. It is often the case, however, that the number and timing of interim analyses are not precisely known at the time of trial design, and thus the implementation of a particular stopping rule must allow for flexible determination of the schedule of interim analyses. In this article, we consider the use of constrained stopping boundaries in the implementation of stopping rules. We compare this approach when used on various scales for the test statistic. When implemented on the scale of boundary crossing probabilities, this approach is identical to the error spending function approach of Lan and DeMets (1983).     

The ‘standardized search’: An improved way to conduct bird surveys

Abstract Bird surveys are among the most widely used biodiversity inventories and serve as the basis for an increasing proportion of pure and applied ecological research. It is rarely possible to conduct exhaustive censuses of all individuals present at a particular site, so stopping rules are routinely used to determine when sampling should finish. Most bird survey methods use (implicit) effort‐based stopping rules, either fixed times, fixed sampling areas (quadrats) or both, to standardize samples of different sites. If between‐site variation is high, however, a fixed sampling effort will generate samples of variable completeness with samples from smaller, less complex sites being more representative and complete than samples from larger, more complex sites. More importantly, quadrat‐based methods shift the scope of the overall study from bird occurrence in sites to bird occurrence in quadrats within sites, diminishing the impact of the research given that results cannot be extrapolated to relevant biological and management scales. Here I advocate an alternative means of conducting bird surveys, whereby the entire site is sampled and a results‐based stopping rule is used to ensure sample completeness is uniform across all sites. For example, a researcher may decide to continue sampling each site until two or fewer previously unencountered species are recorded in a 40‐min period. Samples of different sites will vary in both area and duration but will all be equivalently accurate estimates of species richness. This approach allows the avifauna of entire sites (whether territories, woodland remnants or catchments) to be sampled and compared directly, generating results and implications at the appropriate scale. In addition to yielding reliable measures of species richness, data collected this way can be used to calculate estimates of sample completeness and species incidence, two valuable metrics for ecological studies. This paper includes detailed worked examples of how to conduct a ‘standardized search’ and calculate sample completeness and species incidence estimates. I encourage further research on bird survey methods, and suggest that most current methods are insufficient, inconsistent and unreliable.     

The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

Objective To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres.Design Systematic review.Selection criteria Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect.Data sources Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals, personal communication with methodologists, and the reference lists of relevant articles.Main outcome measures Relation between randomisation and estimates of effect.Results Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight for evaluations of the same intervention and three across different interventions), two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified. Failure to use random allocation and concealment of allocation were associated with relative increases in estimates of effects of 150% or more, relative decreases of up to 90%, inversion of the estimated effect and, in some cases, no difference. On average, failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups.Conclusions Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as or larger than the size of the effects that are to be detected.

Key messages

• Empirical studies support using random allocation in clinical trials and ensuring that the allocation process is concealed—that is, that assignment is impervious to any influence by the people making the allocation
• The effect of not using concealed random allocation can be as large or larger than the effects of worthwhile interventions
• On average, failure to use concealed random allocation results in overestimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups, but it can result in underestimates of effect, reverse the direction of effect, mask an effect, or give similar estimates of effect
• The adequacy of allocation concealment may be a more sensitive measure of bias in clinical trials than scales used to assess the quality of clinical trials
• It is a paradox that the unpredictability of randomisation is the best protection against the unpredictability of the extent and direction of bias in clinical trials that are not properly randomised

     

Estimating Extracellular Concentrations of Dopamine and 3,4-Dihydroxyphenylacetic Acid in Nucleus Accumbens and Striatum Using Microdialysis: Relationships Between In Vitro and In Vivo Recoveries

Abstract: It is common practice in microdialysis studies for probes to be “calibrated” in artificial CSF and in vitro recoveries determined for all substances to be measured in vivo. Dialysate concentrations of such substances are then “corrected” for in vitro recoveries to provide “estimates” of extracellular concentrations. At least for dopamine, in vitro and in vivo recoveries are significantly different and, therefore, an estimate of extracellular dopamine based on correction for in vitro recovery is likely to be erroneous. Generally, however, the relative relationships of such estimates among animals are of interest rather than the “true” extracellular values. Such relationships would be valid to the extent that estimated values are correlated with or predictive of true values. Using the “no net flux” procedure, the present study sought to determine, for both dopamine and its metabolite 3,4-dihydroxy-phenylacetic acid (DOPAC), whether in vitro and in vivo recoveries would correlate with each other as well as whether respective estimated and true (no net flux) values of these substances would correlate with each other. Probes (3 mm; BAS/CMed MF-5393), previously calibrated, were lowered into both the nucleus accumbens and striatum of freely moving rats the day before sample collection was begun. In vitro and in vivo recoveries were not significantly correlated (r= 0.1–0.3), for either dopamine or DOPAC. For both dopamine and DOPAC, however, there were significant correlations (r= 0.7–0.8) between estimated and true values. Surprisingly, when using these commercial probes, absolute dialysate levels for both substances were even better correlated (r = 0.9–0.95) with true values. This suggests that, with these probes, a direct comparison of dialysate concentrations can be used to determine relative changes in basal extracellular levels of dopamine and DOPAC when it is not practical to do no net flux studies (e.g., because of the time required to characterize a drug effect). The use of in vitro calibrations adjusts the values closer to the true values but also adds noise to each value and therefore should be avoided.     

Ascertainment-adjusted parameter estimates revisited

Ascertainment-adjusted parameter estimates from a genetic analysis are typically assumed to reflect the parameter values in the original population from which the ascertained data were collected. Burton et al. (2000) recently showed that, given unmodeled parameter heterogeneity, the standard ascertainment adjustment leads to biased parameter estimates of the population-based values. This finding has important implications in complex genetic studies, because of the potential existence of unmodeled genetic parameter heterogeneity. The authors further stated the important point that, given unmodeled heterogeneity, the ascertainment-adjusted parameter estimates reflect the true parameter values in the ascertained subpopulation. They illustrated these statements with two examples. By revisiting these examples, we demonstrate that if the ascertainment scheme and the nature of the data can be correctly modeled, then an ascertainment-adjusted analysis returns population-based parameter estimates. We further demonstrate that if the ascertainment scheme and data cannot be modeled properly, then the resulting ascertainment-adjusted analysis produces parameter estimates that generally do not reflect the true values in either the original population or the ascertained subpopulation.     

Robustness of an odds-ratio test in a stratified group sequential trial with a binary outcome measure

Many clinical trials compare two or more treatment groups by using a binary outcome measure. For example, the goal could be to determine whether the frequency of pain episodes is significantly reduced in the treatment group (arm A) as compared to the control group (arm B). However, for ethical or regulatory reasons, group sequential designs are commonly employed. Then, based on a binomial distribution, the stopping boundaries for the interim analyses are constructed for assessing the difference in the response probabilities between the two groups. This is easily accomplished by using any of the standard procedures, e.g., those discussed by Jennison and Turnbull (2000), and using one of the most commonly used software packages, East (2000). Several factors are known to often affect the primary outcome of interest, but their true distributions are not known in advance. In addition, these factors may cause heterogeneous treatment responses among individuals in a group, and their exact effect size may be unknown. To limit the effect of such factors on the comparison of the two arms, stratified randomization is used in the actual conduct of the trial. Then, a stratified analysis based on the odds ratio proposed in Jennison and Turnbull (2000, pages 251-252) and consistent with the stratified design is undertaken. However, the stopping rules used for the interim analyses are those obtained for determining the differences in response rates in a design that was not stratified. The purpose of this paper is to assess the robustness of such an approach on the performance of the odds ratio test when the underlying distribution and effect size of the factors that influence the outcome may vary. The simulation studies indicate that, in general, the stratified approach offers consistently better results than does the unstratified approach, as long as the difference in the weighted average of the response probabilities across strata between the two groups remains closer to the hypothesized values, irrespective of the differences in the (allocation) distributions and heterogeneous response rate. However, if the response probabilities deviate significantly from the hypothesized values so that the difference in the weighted average is less than the hypothesized value, then the proposed study could be significantly underpowered.     

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Objective To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome.Design Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses.Data sources 146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes.Main outcome measures Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome. A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates.Results In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)). In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding. There was little evidence for a difference between trials of drug and non-drug interventions. Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses.Conclusions The average bias associated with defects in the conduct of randomised trials varies with the type of outcome. Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.     

Statistical issues in a metaregression analysis of randomized trials: impact on the dietary sodium intake and blood pressure relationship

In a meta-analysis of randomized trials of the effects of dietary sodium interventions on blood pressure, we found substantial heterogeneity among the studies. We were interested in evaluating whether measurement error, known to be a problem for dietary sodium measures, publication bias, or confounding factors could be responsible for the heterogeneity. A measurement error correction was developed that corrects both the slope and the intercept and takes into account the sample size of each study and the number of measurements taken on an individual. The measurement error correction had a minimal effect on the estimates, although it performed well in simulated data. A smoothed scatter plot was used to assess publication bias. Metaregressions provide a convenient way to jointly assess the effects of several factors, but care must be taken to fit an appropriate model.     

One-sided sequential stopping boundaries for clinical trials: a decision-theoretic approach

We address one-sided stopping rules for clinical trials, or more generally, drug development programs, from a decision-theoretic point of view. If efficacy results are sufficiently negative then the trial will be stopped. But regardless of how positive the efficacy results are, the trial will continue in order to demonstrate safety. We show how sequential decisions should be made by a pharmaceutical company attempting to maximize its expected profits.     

Cyclosporin A: a powerful immunosuppressant.

Cyclosporin A (CyA) is a powerful immunosuppressive agent whose lack of myelotoxicity makes it unique among nonsteroidal drugs currently given for immunosuppression. It has been used with initial success in recipients of kidney, liver, bone marrow and pancreas transplants, and it may also have clinical application in the treatment of autoimmune disorders. In regard to its use in transplant recipients, there are many remaining questions about its mechanism of action, the optimum dose, whether it should be used alone or with other immunosuppressants, whether it can suppress chronic rejection and what its long-term side effects may be. These questions can only be answered by further careful laboratory investigation and controlled clinical trials. Until then, CyA should only be administered in centres experienced in its use.     

Satisfaction of the uncertainty principle in cancer clinical trials: retrospective cohort analysis

Objective To assess whether publicly funded adult cancer trials satisfy the uncertainty principle, which states that physicians should enrol a patient in a trial only if they are substantially uncertain which of the treatments in the trial is most appropriate for the patient. This principle is violated if trials systematically favour either the experimental or the standard treatment.Design Retrospective cohort study of completed cancer trials, with randomisation as the unit of analysis.Setting Two cooperative research groups in the United States.Studies included 93 phase III randomised trials (103 randomisations) that completed recruitment of patients between 1981 and 1995.Main outcome measures Whether the randomisation favoured the experimental treatment, the standard treatment, or neither treatment; effect size (outcome of the experimental treatment compared with outcome of the standard treatment) for each randomisation.Results Three randomisations (3%) favoured the standard treatment, 70 (68%) found no significant difference between treatments, and 30 (29%) favoured the experimental treatment. The average effect size was 1.20 (95% confidence interval 1.13 to 1.28), reflecting a slight advantage for the experimental treatment.Conclusions In cooperative group trials in adults with cancer, there is a measurable average improvement in disease control associated with assignment to the experimental rather than the standard arm. However, the heterogeneity of outcomes and the small magnitude of the advantage suggest that, as a group, these trials satisfy the uncertainty principle.     

Study duration for clinical trials with survival response and early stopping rule

A comparative clinical trial with built-in sequential stopping rules allows earlier-than-scheduled stopping, should there be a significant indication of treatment difference. In a clinical trial where the major outcome is time (survival time or response) to a certain event such as failure, the design of the study should determine how long one needs to accrue patients and follow through until there is a sufficient number of events observed during the entire study duration. This paper proposes a unified design procedure for group sequential clinical trials with survival response. The time to event is assumed to be exponentially distributed, but the arguments extend naturally to the proportional hazards model after suitable transformation on the time scale. An example from the Eastern Cooperative Oncology Group (ECOG) is given to illustrate how this procedure can be implemented. The same example is used to explore the overall operating characteristics and the robustness of the proposed group sequential design.     

Reappraising the variability of effects of antipsychotic medication in schizophrenia: a meta‐analysis

It is common experience for practising psychiatrists that individuals with schizophrenia vary markedly in their symptomatic response to antipsychotic medication. What is not clear, however, is whether this variation reflects variability of medication‐specific effects (also called “treatment effect heterogeneity”), as opposed to variability of non‐specific effects such as natural symptom fluctuation or placebo response. Previous meta‐analyses found no evidence of treatment effect heterogeneity, suggesting that a “one size fits all” approach may be appropriate and that efforts at developing personalized treatment strategies for schizophrenia are unlikely to succeed. Recent advances indicate, however, that earlier approaches may have been unable to accurately quantify treatment effect heterogeneity due to their neglect of a key parameter: the correlation between placebo response and medication‐specific effects. In the present paper, we address this shortcoming by using individual patient data and study‐level data to estimate that correlation and quantitatively characterize antipsychotic treatment effect heterogeneity in schizophrenia. Individual patient data (on 384 individuals who were administered antipsychotic treatment and 88 who received placebo) were obtained from the Yale University Open Data Access (YODA) database. Study‐level data were obtained from a meta‐analysis of 66 clinical trials including 17,202 patients. Both individual patient and study‐level analyses yielded a negative correlation between placebo response and treatment effect for the total score on the Positive and Negative Syndrome Scale (PANSS) (ρ=–0.32, p=0.002 and ρ=–0.39, p<0.001, respectively). Using the most conservative of these estimates, a meta‐analysis of treatment effect heterogeneity provided evidence of a marked variability in antipsychotic‐specific effects between individuals with schizophrenia, with the top quartile of patients experiencing beneficial treatment effects of 17.7 points or more on the PANSS total score, while the bottom quartile presented a detrimental effect of treatment relative to placebo. This evidence of clinically meaningful treatment effect heterogeneity suggests that efforts to personalize antipsychotic treatment of schizophrenia have potential for success.     

Iterative kriging for removing spatial autocorrelation in analysis of forest genetic trials

Conventional analysis of spatially correlated data in inadequately blocked field genetic trials may give erroneous results that would seriously affect breeding decisions. Forest genetic trials are commonly very large and strongly heterogeneous, so adjustments for micro-environmental heterogeneity become indispensable. This study explores the use of geostatistics to account for the spatial autocorrelation in four Pinus pinaster Ait. progeny trials established on hilly and irregular terrains with a randomized complete block design and large blocks. Data of five different traits assessed at age 8 were adjusted using an iterative method based on semivariograms and kriging, and the effects on estimates of variance components, heritability, and family effects were evaluated in relation to conventional analysis. Almost all studied traits showed nonrandom spatial structures. Therefore, after the adjustments for spatial autocorrelation, the block and family × block variance components, which were extremely high in the conventional analysis, almost disappeared. The reduction of the interaction variance was recovered by the family variance component, resulting in higher heritability estimates. The removal of the spatial autocorrelation also affected the estimation of family effects, resulting in important changes in family ranks after the spatial adjustments. Comparison among families was also greatly improved due to higher accuracy of the family effect estimations. The analysis improvement was larger for growth traits, which showed the strongest spatial heterogeneity, but was also evident for other traits such as straightness or number of whorls. The present paper demonstrates how spatial autocorrelation can drastically affect the analysis of forest genetic trials with large blocks. The iterative kriging procedure presented in this paper is a promising tool to account for this spatial heterogeneity.     

Assessing exposure-time treatment effect heterogeneity in stepped-wedge cluster randomized trials

A stepped-wedge cluster randomized trial (CRT) is a unidirectional crossover study in which timings of treatment initiation for clusters are randomized. Because the timing of treatment initiation is different for each cluster, an emerging question is whether the treatment effect depends on the exposure time, namely, the time duration since the initiation of treatment. Existing approaches for assessing exposure-time treatment effect heterogeneity either assume a parametric functional form of exposure time or model the exposure time as a categorical variable, in which case the number of parameters increases with the number of exposure-time periods, leading to a potential loss in efficiency. In this article, we propose a new model formulation for assessing treatment effect heterogeneity over exposure time. Rather than a categorical term for each level of exposure time, the proposed model includes a random effect to represent varying treatment effects by exposure time. This allows for pooling information across exposure-time periods and may result in more precise average and exposure-time-specific treatment effect estimates. In addition, we develop an accompanying permutation test for the variance component of the heterogeneous treatment effect parameters. We conduct simulation studies to compare the proposed model and permutation test to alternative methods to elucidate their finite-sample operating characteristics, and to generate practical guidance on model choices for assessing exposure-time treatment effect heterogeneity in stepped-wedge CRTs.     

A Method of Screening for Genes of Major Effect

This paper describes a method for screening animal populations on an index of calculated probabilities of genotype status at an unknown single locus. Animals selected by such a method might then be candidates in test matings and genetic marker analyses for major gene detection. The method relies on phenotypic measures for a continuous trait plus identification of sire and dam. Some missing phenotypes and missing pedigree information are permitted. The method is an iterative two-step procedure, the first step estimates genotype probabilities and the second step estimates genotypic effects by regressing phenotypes on genotype probabilities, modeled as true genotype status plus error. Prior knowledge or choice of major locus-free heritability for the trait of interest is required, plus initial starting estimates of the effect on phenotype of carrying one and two copies of the unknown gene. Gene frequency can be estimated by this method, but it is demonstrated that the consequences of using an incorrect fixed prior for gene frequency are not particularly adverse where true frequency of the allele with major effect is low. Simulations involving deterministic sampling from the normal distribution lead to convergence for estimates of genotype effects at the true values, for a reasonable range of starting values, illustrating that estimation of major gene effects has a rational basis. In the absence of polygenic effects, stochastic simulations of 600 animals in five generations resulted in estimates of genotypic effects close to the true values. However, stochastic simulations involving generation and fitting of both major genotype and animal polygenic effects showed upward bias in estimates of major genotype effects. This can be partially overcome by not using information from relatives when calculating genotype probabilities-a result which suggests a route to a modified method which is unbiased and yet does use this information.     

The relation between treatment benefit and underlying risk in meta-analysis.

In meta-analyses of clinical trials comparing a treated group with a control group it has been common to ask whether the treatment benefit varies according to the underlying risk of the patients in the different trials, with the hope of defining which patients would benefit most and which least from medical interventions. The usual analysis used to investigate this issue, however, which uses the observed proportions of events in the control groups of the trials as a measure of the underlying risk, is flawed and produces seriously misleading results. This arises through a bias due to regression to the mean and will be particularly acute in meta-analyses which include some small trials or in which the variability in the true underlying risks across trials is small. Approaches which previously have been thought to be more appropriate are to substitute the average proportion of events in the control and treated groups as the measure of underlying risk or to plot the proportion of events in the treated group against that in the control group (L''Abbé plot). However, these are still subject to bias in most circumstances. Because of the potentially seriously flawed conclusions that can result from such analyses, they should be replaced either by statistically appropriate (but more complex) approaches or, preferably, by analyses which investigate the dependence of the treatment effect on measured baseline characteristics of the patients in each trial.     

Generalized least squares for the synthesis of correlated information

This paper deals with the synthesis of information from different studies when there is lack of independence in some of the contrasts to be combined. This problem can arise in several different situations in both case-control studies and clinical trials. For efficient estimation we appeal to the method of generalized least squares to estimate the summary effect and its standard error. This method requires estimates of the covariances between those contrasts that are not independent. Although it is not possible to estimate the covariance between effects that have been adjusted for confounding factors we present a method for finding upper and lower bounds for this covariance. In the simplest discussion homogeneity of the relative risks is assumed but the method is then extended to allow for heterogeneity in an overall estimate. We then illustrate the method with several examples from an analysis in which case-control studies of cervical cancer and oral contraceptive use are synthesized.     

Influence of trial duration on the bias of the estimated treatment effect in clinical trials when individual heterogeneity is ignored

In randomized clinical trials where the times to event of two treatment groups are compared under a proportional hazards assumption, it has been established that omitting prognostic factors from the model entails an underestimation of the hazards ratio. Heterogeneity due to unobserved covariates in cancer patient populations is a concern since genomic investigations have revealed molecular and clinical heterogeneity in these populations. In HIV prevention trials, heterogeneity is unavoidable and has been shown to decrease the treatment effect over time. This article assesses the influence of trial duration on the bias of the estimated hazards ratio resulting from omitting covariates from the Cox analysis. The true model is defined by including an unobserved random frailty term in the individual hazard that reflects the omitted covariate. Three frailty distributions are investigated: gamma, log‐normal, and binary, and the asymptotic bias of the hazards ratio estimator is calculated. We show that the attenuation of the treatment effect resulting from unobserved heterogeneity strongly increases with trial duration, especially for continuous frailties that are likely to reflect omitted covariates, as they are often encountered in practice. The possibility of interpreting the long‐term decrease in treatment effects as a bias induced by heterogeneity and trial duration is illustrated by a trial in oncology where adjuvant chemotherapy in stage 1B NSCLC was investigated.