A comparison of methods for constructing confidence intervals after phase II/III clinical trials

Recently, in order to accelerate drug development, trials that use adaptive seamless designs such as phase II/III clinical trials have been proposed. Phase II/III clinical trials combine traditional phases II and III into a single trial that is conducted in two stages. Using stage 1 data, an interim analysis is performed to answer phase II objectives and after collection of stage 2 data, a final confirmatory analysis is performed to answer phase III objectives. In this paper we consider phase II/III clinical trials in which, at stage 1, several experimental treatments are compared to a control and the apparently most effective experimental treatment is selected to continue to stage 2. Although these trials are attractive because the confirmatory analysis includes phase II data from stage 1, the inference methods used for trials that compare a single experimental treatment to a control and do not have an interim analysis are no longer appropriate. Several methods for analysing phase II/III clinical trials have been developed. These methods are recent and so there is little literature on extensive comparisons of their characteristics. In this paper we review and compare the various methods available for constructing confidence intervals after phase II/III clinical trials.     

Internal pilots for observational studies

Study planning often involves selecting an appropriate sample size. Power calculations require specifying an effect size and estimating “nuisance” parameters, e.g. the overall incidence of the outcome. For observational studies, an additional source of randomness must be estimated: the rate of the exposure. A poor estimate of any of these parameters will produce an erroneous sample size. Internal pilot (IP) designs reduce the risk of this error ‐ leading to better resource utilization ‐ by using revised estimates of the nuisance parameters at an interim stage to adjust the final sample size. In the clinical trials setting, where allocation to treatment groups is pre‐determined, IP designs have been shown to achieve the targeted power without introducing substantial inflation of the type I error rate. It has not been demonstrated whether the same general conclusions hold in observational studies, where exposure‐group membership cannot be controlled by the investigator. We extend the IP to observational settings. We demonstrate through simulations that implementing an IP, in which prevalence of the exposure can be re‐estimated at an interim stage, helps ensure optimal power for observational research with little inflation of the type I error associated with the final data analysis.     

Conditional Rejection Probabilities of Student's t‐test and Design Adaptations

For clinical trials with interim analyses conditional rejection probabilities play an important role when stochastic curtailment or design adaptations are performed. The conditional rejection probability gives the conditional probability to finally reject the null hypothesis given the interim data. It is computed either under the null or the alternative hypothesis. We investigate the properties of the conditional rejection probability for the one sided, one sample t‐test and show that it can be non monotone in the interim mean of the data and non monotone in the non‐centrality parameter for the alternative. We give several proposals how to implement design adaptations (that are based on the conditional rejection probability) for the t‐test and give a numerical example. Additionally, the conditional rejection probability given the interim t‐statistic is investigated. It does not depend on the unknown σ and can be used in stochastic curtailment procedures. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

One‐sided Simultaneous Confidence Intervals for Effective Dose Steps in Unbalanced Designs

An important issue in dose finding is whether a further dose increment leads to a relevant increase in efficacy. Clinical efficacy should not be considered by point zero null hypotheses. Instead, shifted hypotheses for the difference or the ratio can be used. Because the a priori definition of a relevance threshold is frequently difficult, confidence intervals should be used for a posteriori interpretation. Sample size estimation – a‐priori or by adaptive interim analysis‐ is inherent, because the effective dose steps are arbitrary in un‐designed studies. For simultaneous confidence intervals without order restriction the exact distributions under the null and the alternative hypothesis is proposed for the general unbalanced one‐way design.     

A Bootstrap Procedure for Adaptive Selection of the Test Statistic in Flexible Two‐Stage Designs

Adaptive two‐stage designs allow a data‐driven change of design characteristics during the ongoing trial. One of the available options is an adaptive choice of the test statistic for the second stage of the trial based on the results of the interim analysis. Since there is often only a vague knowledge of the distribution shape of the primary endpoint in the planning phase of a study, a change of the test statistic may then be considered if the data indicate that the assumptions underlying the initial choice of the test are not correct. Collings and Hamilton proposed a bootstrap method for the estimation of the power of the two‐sample Wilcoxon test for shift alternatives. We use this approach for the selection of the test statistic. By means of a simulation study, we show that the gain in terms of power may be considerable when the initial assumption about the underlying distribution was wrong, whereas the loss is relatively small when in the first instance the optimal test statistic was chosen. The results also hold true for comparison with a one‐stage design. Application of the method is illustrated by a clinical trial example.     

Repeated confidence intervals in self-designing clinical trials and switching between noninferiority and superiority

In self-designing clinical trials, repeated confidence intervals are derived for the parameter of interest where the results of the independent study stages are combined using the generalized inverse chi-square-method. The confidence intervals can be calculated at each interim analysis and always hold the predefined overall nominal confidence level. Moreover, the confidence intervals calculated during the course of the trial are nested in the sense that a calculated interval is completely contained in all the previously calculated intervals. During the course of the self-designing trial the sample sizes as well as the number of study stages can be determined simultaneously in a completely adaptive way. The adaptive procedure allows an early stop for significance. The clinical trial may be originally designed either to show noninferiority or superiority. However, in each interim analysis, it is possible to change the planning from showing superiority to showing noninferiority or vice versa. Since the repeated confidence intervals are nested, there is no risk to loose the noninferiority once showed when, after an interim analysis, the trial is continued in an attempt to reach superiority. A simulation study investigates the behavior of the considered confidence intervals. The performance of the derived nested repeated confidence intervals is also demonstrated in examples showing both kinds of switching during an ongoing trial.     

Inference on Multiple Endpoints in Clinical Trials with Adaptive Interim Analyses

Planned interim analyses which permit early stopping or sample size adaption of a trial are desirable for ethical and scientific reasons. Multiple test procedures allow inference about several hypotheses within a single clinical trial. In this paper, a method which combines multiple testing with adaptive interim analyses whilst controlling the experimentwise error rate is proposed. The general closed testing principle, the situation of a priori ordered hypotheses, and application of the Bonferroni-Holm method are considered. The practical application of the method is demonstrated by an example.     

Sequential tests for noninferiority and superiority

The problem of simultaneous sequential tests for noninferiority and superiority of a treatment, as compared to an active control, is considered in terms of continuous hierarchical families of one-sided null hypotheses, in the framework of group sequential and adaptive two-stage designs. The crucial point is that the decision boundaries for the individual null hypotheses may vary over the parameter space. This allows one to construct designs where, e.g., a rigid stopping criterion is chosen, rejecting or accepting all individual null hypotheses simultaneously. Another possibility is to use monitoring type stopping boundaries, which leave some flexibility to the experimenter: he can decide, at the interim analysis, whether he is satisfied with the noninferiority margin achieved at this stage, or wants to go for more at the second stage. In the case where he proceeds to the second stage, he may perform midtrial design modifications (e.g., reassess the sample size). The proposed approach allows one to "spend," e.g., less of alpha for an early proof of noninferiority than for an early proof of superiority, and is illustrated by typical examples.     

Stopping Rules for Clinical Trials Implicitly Incorporating Safety Information

Clinical trials research is mainly conducted for the purpose of evaluating the relative efficacy of two or more treatments. However, a positive response due to treatment is not sufficient to put forward a new product because one must also demonstrate safety. In such cases, clinical trials which show a positive effect would need to accrue enough patients to also demonstrate that the new treatment is safe. It is our purpose to show how the efficacy and safety problems can be combined to yield a more practical clinical trial design. In this paper we propose an asymmetric stopping rule which allows the experimenter to terminate a clinical trial early for a sufficiently negative result and to continue to a specified number of patients otherwise. As it turns out, a few interim tests will have negligible effects on the overall significance level.     

Adaptive Two Stage Designs and the Conditional Error Function

Proschan and Hunsberger (1995) suggest the use of a conditional error function to construct a two stage test that meets the α level and allows a very flexible reassessment of the sample size after the interim analysis. In this note we show that several adaptive designs can be formulated in terms of such an error function. The conditional power function defined similarly provides a simple method for sample size reassessment in adaptive two stage designs.     

Implementation of group sequential logrank tests in a maximum duration trial

To control the Type I error probability in a group sequential procedure using the logrank test, it is important to know the information times (fractions) at the times of interim analyses conducted for purposes of data monitoring. For the logrank test, the information time at an interim analysis is the fraction of the total number of events to be accrued in the entire trial. In a maximum information trial design, the trial is concluded when a prespecified total number of events has been accrued. For such a design, therefore, the information time at each interim analysis is known. However, many trials are designed to accrue data over a fixed duration of follow-up on a specified number of patients. This is termed a maximum duration trial design. Under such a design, the total number of events to be accrued is unknown at the time of an interim analysis. For a maximum duration trial design, therefore, these information times need to be estimated. A common practice is to assume that a fixed fraction of information will be accrued between any two consecutive interim analyses, and then employ a Pocock or O'Brien-Fleming boundary. In this article, we describe an estimate of the information time based on the fraction of total patient exposure, which tends to be slightly negatively biased (i.e., conservative) if survival is exponentially distributed. We then present a numerical exploration of the robustness of this estimate when nonexponential survival applies. We also show that the Lan-DeMets (1983, Biometrika 70, 659-663) procedure for constructing group sequential boundaries with the desired level of Type I error control can be computed using the estimated information fraction, even though it may be biased. Finally, we discuss the implications of employing a biased estimate of study information for a group sequential procedure.     

Conditionally unbiased and near unbiased estimation of the selected treatment mean for multistage drop‐the‐losers trials

The two‐stage drop‐the‐loser design provides a framework for selecting the most promising of K experimental treatments in stage one, in order to test it against a control in a confirmatory analysis at stage two. The multistage drop‐the‐losers design is both a natural extension of the original two‐stage design, and a special case of the more general framework of Stallard & Friede ( 2008 ) (Stat. Med. 27 , 6209–6227). It may be a useful strategy if deselecting all but the best performing treatment after one interim analysis is thought to pose an unacceptable risk of dropping the truly best treatment. However, estimation has yet to be considered for this design. Building on the work of Cohen & Sackrowitz ( 1989 ) (Stat. Prob. Lett. 8 , 273–278), we derive unbiased and near‐unbiased estimates in the multistage setting. Complications caused by the multistage selection process are shown to hinder a simple identification of the multistage uniform minimum variance conditionally unbiased estimate (UMVCUE); two separate but related estimators are therefore proposed, each containing some of the UMVCUEs theoretical characteristics. For a specific example of a three‐stage drop‐the‐losers trial, we compare their performance against several alternative estimators in terms of bias, mean squared error, confidence interval width and coverage.     

Adaptive group sequential designs for clinical trials: combining the advantages of adaptive and of classical group sequential approaches

A general method is presented integrating the concept of adaptive interim analyses into classical group sequential testing. This allows the researcher to represent every group sequential plan as an adaptive trial design and to make design changes during the course of the trial after every interim analysis in the same way as with adaptive designs. The concept of adaptive trial designing is thereby generalized to a large variety of possible sequential plans.     

Monitoring the rates of composite events with censored data in phase II clinical trials

In many phase II clinical trials, interim monitoring is based on the probability of a binary event, response, defined in terms of one or more time-to-event variables within a time period of fixed length. Such outcome-adaptive methods may require repeated interim suspension of accrual in order to follow each patient for the time period required to evaluate response. This may increase trial duration, and eligible patients arriving during such delays either must wait for accrual to reopen or be treated outside the trial. Alternatively, monitoring may be done continuously by ignoring censored data each time the stopping rule is applied, which wastes information. We propose an adaptive Bayesian method that eliminates these problems. At each patient's accrual time, an approximate posterior for the response probability based on all of the event-time data is used to compute an early stopping criterion. Application to a leukemia trial with a composite event shows that the method can reduce trial duration substantially while maintaining the reliability of interim decisions.     

Population-Enrichment Adaptive Design Strategy for an Event-Driven Vaccine Efficacy Trial

A population-enrichment adaptive design allows a prospective use for study population selection. It has the flexibility allowing pre-specified modifications to an ongoing trial to mitigate the potential risk associated with the assumptions made at design stage. In this way, the trial can potentially encompass a broader target patient population, and move forward only with the subpopulations that appear to be benefiting from the treatment. Our work is motivated by a Phase III event-driven vaccine efficacy trial. Two target patient subpopulations were enrolled with the assumption that vaccine efficacy can be demonstrated based on the combined population. It is recognized due to the nature of patients’ underlying conditions, one subpopulation might respond to the treatment better than the other. To maximize the probability of demonstrating vaccine efficacy in at least one patient population while taking advantage of combining two subpopulations in one single trial, an adaptive design strategy with potential population enrichment is developed. Specifically, if the observed vaccine efficacy at interim for one subpopulation is not promising to warrant carrying forward, the population may be enriched with the other subpopulation with better performance. Simulations were conducted to evaluate the operational characteristics from a selection of interim analysis plans. This population-enrichment design provides a more efficient way as compared to the conventional approaches when targeting multiple subpopulations. If executed and planned with caution, this strategy can provide a greater chance of success of the trial and help maintain scientific and regulatory rigors.     

Combining adaptive designs with control of the false discovery rate--a generalized definition for a global p-value

When combining adaptive designs with control of the False Discovery Rate one has to keep in mind that the most frequently used procedure for controlling the False Discovery Rate--the explorative Simes procedure--is a stepwise multiple testing procedure. At the interim analysis of an adaptive design it is however not yet known what the boundaries for rejection of hypotheses in the final analysis will be as these boundaries depend on the size of the final p-values. Therefore classical adaptive designs with a predefined stopping criterion for early rejection of hypotheses are not well suited. We propose a generalized definition of a global p-value for a two-stage adaptive design permitting a flexible decision for stopping at the interim analysis. By means of a simulation study in the field of genetic epidemiology we illustrate how applying such a two-stage design can reduce costs.     

Improved Repeated Confidence Bounds in Trials with a Maximal Goal

Repeated confidence intervals can be computed at every interim analysis of a flexible group sequential design without the need to stop the trial with a pre‐planned stopping rule. Often, however, there is a maximal goal such that the trial is surely stopped if this goal is reached. This induces a maximal stopping rule, and repeated confidence intervals are strictly conservative, when adhering to it. A modification is proposed which uniformly improves the one sided repeated confidence interval in such a situation. It preserves the monitoring character, and leads to uniformly smaller intervals, when reaching the maximal goal at an interim analysis. The modification is worked out for two stage designs and is indicated for multi‐stage trials. The extent of the improvement is quantified for two simple scenarios.     

Multiple Comparisons of Treatments with Stable Multivariate Tests in a Two‐Stage Adaptive Design,Including a Test for Non‐Inferiority

The application of stabilized multivariate tests is demonstrated in the analysis of a two‐stage adaptive clinical trial with three treatment arms. Due to the clinical problem, the multiple comparisons include tests of superiority as well as a test for non‐inferiority, where non‐inferiority is (because of missing absolute tolerance limits) expressed as linear contrast of the three treatments. Special emphasis is paid to the combination of the three sources of multiplicity – multiple endpoints, multiple treatments, and two stages of the adaptive design. Particularly, the adaptation after the first stage comprises a change of the a‐priori order of hypotheses.     

Impacts on type I error rate with inappropriate use of learn and confirm in confirmatory adaptive design trials

A two-stage adaptive design trial is a single trial that combines the learning data from stage 1 (or phase II) and the confirming data in stage 2 (or phase III) for formal statistical testing. We call it a "Learn and Confirm" trial. The studywise type I error rate remains to be at issue in a "Learn and Confirm" trial. For studying multiple doses or multiple enpdoints, a "Learn and Confirm" adaptive design can be more attractive than a fixed design approach. This is because intuitively the learning data in stage 1 should not be subjected to type I error scrutiny if there is no formal interim analysis performed and only an adaptive selection of design parameters is made at stage 1. In this work, we conclude from extensive simulation studies that the intuition is most often misleading. That is, regardless of whether or not there is a formal interim analysis for making an adaptive selection, the type I error rates are always at risk of inflation. Inappropriate use of any "Learn and Confirm" strategy should not be overlooked.