Room for improvement in conducting and reporting non-inferiority randomized controlled trials on drugs: a systematic review

Background

A non-inferiority (NI) trial is intended to show that the effect of a new treatment is not worse than the comparator. We conducted a review to identify how NI trials were conducted and reported, and whether the standard requirements from the guidelines were followed.

Methodology and Principal Findings

From 300 randomly selected articles on NI trials registered in PubMed at 5 February 2009, we included 227 NI articles that referred to 232 trials. We excluded studies on bioequivalence, trials on healthy volunteers, non-drug trials, and articles of which the full-text version could not be retrieved. A large proportion of trials (34.0%) did not use blinding. The NI margin was reported in 97.8% of the trials, but only 45.7% of the trials reported the method to determine the margin. Most of the trials used either intention to treat (ITT) (34.9%) or per-protocol (PP) analysis (19.4%), while 41.8% of the trials used both methods. Less than 10% of the trials included a placebo arm to confirm the efficacy of the new drug and active comparator against placebo, and less than 5.0% were reporting the similarity of the current trial with the previous comparator''s trials. In general, no difference was seen in the quality of reporting before and after the release of the CONSORT statement extension 2006 or between the high-impact and low-impact journals.

Conclusion

The conduct and reporting of NI trials can be improved, particularly in terms of maximizing the use of blinding, the use of both ITT and PP analysis, reporting the similarity with the previous comparator''s trials to guarantee a valid constancy assumption, and most importantly reporting the method to determine the NI margin.     

Challenges and regulatory experiences with non‐inferiority trial design without placebo arm

For a non‐inferiority trial without a placebo arm, the direct comparison between the test treatment and the selected positive control is in principle the only basis for statistical inference. Therefore, evaluating the test treatment relative to the non‐existent placebo presents extreme challenges and requires some kind of bridging from the past to the present with no current placebo data. For such inference based partly on an indirect bridging manipulation, fixed margin method and synthesis method are the two widely discussed methods in the recent literature. There are major differences in statistical inference paradigm between the two methods. The fixed margin method employs the historical data that assess the performances of the active control versus a placebo to guide the selection of the non‐inferiority margin. Such guidance is not part of the ultimate statistical inference in the non‐inferiority trial. In contrast, the synthesis method connects the historical data to the non‐inferiority trial data for making broader inferences relating the test treatment to the non‐existent current placebo. On the other hand, the type I error rate associated with the direct comparison between the test treatment and the active control cannot shed any light on the appropriateness of the indirect inference for faring the test treatment against the non‐existent placebo. This work explores an approach for assessing the impact of potential bias due to violation of a key statistical assumption to guide determination of the non‐inferiority margin.     

Testing superiority and non-inferiority hypotheses in active controlled clinical trials

Switching between testing for superiority and non-inferiority has been an important statistical issue in the design and analysis of active controlled clinical trial. In practice, it is often conducted with a two-stage testing procedure. It has been assumed that there is no type I error rate adjustment required when either switching to test for non-inferiority once the data fail to support the superiority claim or switching to test for superiority once the null hypothesis of non-inferiority is rejected with a pre-specified non-inferiority margin in a generalized historical control approach. However, when using a cross-trial comparison approach for non-inferiority testing, controlling the type I error rate sometimes becomes an issue with the conventional two-stage procedure. We propose to adopt a single-stage simultaneous testing concept as proposed by Ng (2003) to test both non-inferiority and superiority hypotheses simultaneously. The proposed procedure is based on Fieller's confidence interval procedure as proposed by Hauschke et al. (1999).     

Noninferiority trials

Noninferiority trials are intended to show that the effect of a new treatment is not worse than that of an active control by more than a specified margin. These trials have a number of inherent weaknesses that superiority trials do not: no internal demonstration of assay sensitivity, no single conservative analysis approach, lack of protection from bias by blinding, and difficulty in specifying the noninferiority margin. Noninferiority trials may sometimes be necessary when a placebo group can not be ethically included, but it should be recognized that the results of such trials are not as credible as those from a superiority trial.     

Monitoring clinical trials with multiple arms

In order to benefit from the substantial overhead expenses of a large group sequential clinical trial, the simultaneous investigation of several competing treatments becomes more popular. If at some interim analysis any treatment arm reveals itself to be inferior to any other treatment under investigation, this inferior arm may be or may even need to be dropped for ethical and/or economic reasons. Recently proposed methods for monitoring and analysis of group sequential clinical trials with multiple treatment arms are compared and discussed. The main focus of the article is on the application and extension of (adaptive) closed testing procedures in the group sequential setting that strongly control the familywise error rate. A numerical example is given for illustration.     

Non-inferiority testing with a variable margin

There has been growing interest, when comparing an experimental treatment with an active control with respect to a binary outcome, in allowing the non-inferiority margin to depend on the unknown success rate in the control group. It does not seem universally recognized, however, that the statistical test should appropriately adjust for the uncertainty surrounding the non-inferiority margin. In this paper, we inspect a naive procedure that treats an "observed margin" as if it were fixed a priori, and explain why it might not be valid. We then derive a class of tests based on the delta method, including the Wald test and the score test, for a smooth margin. An alternative derivation is given for the asymptotic distribution of the likelihood ratio statistic, again for a smooth margin. We discuss the asymptotic behavior of these tests when applied to a piecewise smooth margin. A simple condition on the margin function is given which allows the likelihood ratio test to carry over to a piecewise smooth margin using the same critical value as for a smooth margin. Simulation experiments are conducted, under a smooth margin and a piecewise linear margin, to evaluate the finite-sample performance of the asymptotic tests studied.     

A mixed approach for proving non-inferiority in clinical trials with binary endpoints

When a new treatment is compared to an established one in a randomized clinical trial, it is standard practice to statistically test for non-inferiority rather than for superiority. When the endpoint is binary, one usually compares two treatments using either an odds-ratio or a difference of proportions. In this paper, we propose a mixed approach which uses both concepts. One first defines the non-inferiority margin using an odds-ratio and one ultimately proves non-inferiority statistically using a difference of proportions. The mixed approach is shown to be more powerful than the conventional odds-ratio approach when the efficacy of the established treatment is known (with good precision) and high (e.g. with more than 56% of success). The gain of power achieved may lead in turn to a substantial reduction in the sample size needed to prove non-inferiority. The mixed approach can be generalized to ordinal endpoints.     

The placebo response in clinical trials: more questions than answers

Meta-analyses and re-analyses of trial data have not been able to answer some of the essential questions that would allow prediction of placebo responses in clinical trials. We will confront these questions with current empirical evidence. The most important question asks whether the placebo response rates in the drug arm and in the placebo arm are equal. This 'additive model' is a general assumption in almost all placebo-controlled drug trials but has rarely been tested. Secondly, we would like to address whether the placebo response is a function of the likelihood of receiving drug/placebo. Evidence suggests that the number of study arms in a trial may determine the size of the placebo and the drug response. Thirdly, we ask what the size of the placebo response is in 'comparator' studies with a direct comparison of a (novel) drug against another drug. Meta-analytic and experimental evidence suggests that comparator studies may produce higher placebo response rates when compared with placebo-controlled trials. Finally, we address the placebo response rate outside the laboratory and outside of trials in clinical routine. This question poses a serious challenge whether the drug response in trials can be taken as evidence of drug effects in clinical routine.     

On testing simultaneously non-inferiority in two multiple primary endpoints and superiority in at least one of them

In a clinical trial with an active treatment and a placebo the situation may occur that two (or even more) primary endpoints may be necessary to describe the active treatment's benefit. The focus of our interest is a more specific situation with two primary endpoints in which superiority in one of them would suffice given that non-inferiority is observed in the other. Several proposals exist in the literature for dealing with this or similar problems, but prove insufficient or inadequate at a closer look (e.g. Bloch et al. (2001, 2006) or Tamhane and Logan (2002, 2004)). For example, we were unable to find a good reason why a bootstrap p-value for superiority should depend on the initially selected non-inferiority margins or on the initially selected type I error alpha. We propose a hierarchical three step procedure, where non-inferiority in both variables must be proven in the first step, superiority has to be shown by a bivariate test (e.g. Holm (1979), O'Brien (1984), Hochberg (1988), a bootstrap (Wang (1998)), or L?uter (1996)) in the second step, and then superiority in at least one variable has to be verified in the third step by a corresponding univariate test. All statistical tests are performed at the same one-sided significance level alpha. From the above mentioned bivariate superiority tests we preferred L?uter's SS test and the Holm procedure for the reason that these have been proven to control the type I error strictly, irrespective of the correlation structure among the primary variables and the sample size applied. A simulation study reveals that the performance regarding power of the bivariate test depends to a considerable degree on the correlation and on the magnitude of the expected effects of the two primary endpoints. Therefore, the recommendation of which test to choose depends on knowledge of the possible correlation between the two primary endpoints. In general, L?uter's SS procedure in step 2 shows the best overall properties, whereas Holm's procedure shows an advantage if both a positive correlation between the two variables and a considerable difference between their standardized effect sizes can be expected.     

Non-inferiority of new procedure to standard procedure in stratified matched-pair design

We consider the statistical testing for non-inferiority of a new treatment compared with the standard one under matched-pair setting in a stratified study or in several trials. A non-inferiority test based on the efficient scores and a Mantel-Haenszel (M-H) like procedure with restricted maximum likelihood estimators (RMLEs) of nuisance parameters and their corresponding sample size formulae are presented. We evaluate the above tests and the M-H type Wald test in level and power. The stratified score test is conservative and provides the best power. The M-H like procedure with RMLEs gives an accurate level. However, the Wald test is anti-conservative and we suggest caution when it is used. The unstratified score test is not biased but it is less powerful than the stratified score test when base-line probabilities related to strata are not the same. This investigation shows that the stratified score test possesses optimum statistical properties in testing non-inferiority. A common difference between two proportions across strata is the basic assumption of the stratified tests, we present appropriate tests to validate the assumption and related remarks.     

Choice of delta: requirements and reality--results of a systematic review

An essential problem in planning clinical non-inferiority or equivalence studies is the specification of the 'irrelevant difference' (irrelevance margin; delta). This quantifies the amount of non-inferiority or difference, respectively, between a new test therapy and an established standard treatment which is to be considered as tolerable. In the past, most recommendations and guidelines for clinical non-inferiority and equivalence studies contained only general statements and formulations concerning the specification of delta. The current unsatisfactory situation was the reason for performing a systematic review of published clinical non-inferiority and equivalence studies. It was the aim to gain an overview on the irrelevance margins used in such studies, and on reasons for choosing the particular margins. For the sake of comparability, the irrelevance margins were converted into standardized differences and odds ratios. Overall, there were 332 non-inferiority or equivalence trials obtained by means of an extensive literature search. The results of the systematic review show that current requirements on the choice of delta and the reality of recent clinical non-inferiority and equivalence trials differ substantially. In about one half of the trials a difference of 0.5 standard deviations or more was regarded as 'irrelevant' explicitly or implicitly. Estimates of standard-placebo differences formed the basis of the irrelevance margin in less than every tenth trial. Reasons for this very low proportion might be (1) the possibly resulting very small irrelevance margins, and (2) unsolved problems of the requirements themselves. Overall, it seems that a more global definition of 'irrelevance' might be warranted.     

The large sample bounds on the principal strata effect with application to a prostate cancer prevention trial

Issues of post-randomization selection bias and truncation-by-death can arise in randomized clinical trials; for example, in a cancer prevention trial, an outcome such as cancer severity is undefined for individuals who do not develop cancer. Restricting analysis to a subpopulation selected after randomization can give rise to biased outcome comparisons. One approach to deal with such issues is to consider the principal strata effect (PSE, or equally, the survivor average causal effect). PSE is defined as the effect of treatment on the outcome among the subpopulation that would have been selected under either treatment arm. Unfortunately, the PSE cannot generally be estimated without the identifying assumptions; however, the bounds can be derived using a deterministic causal model. In this paper, we propose a number of assumptions for deriving the bounds with narrow width. The assumptions and bounds, which differ from those introduced by Zhang and Rubin (2003), are illustrated using data from a randomized prostate cancer prevention trial.     

Regression survival analysis with an assumed copula for dependent censoring: a sensitivity analysis approach

SUMMARY: In clinical studies, when censoring is caused by competing risks or patient withdrawal, there is always a concern about the validity of treatment effect estimates that are obtained under the assumption of independent censoring. Because dependent censoring is nonidentifiable without additional information, the best we can do is a sensitivity analysis to assess the changes of parameter estimates under different assumptions about the association between failure and censoring. This analysis is especially useful when knowledge about such association is available through literature review or expert opinions. In a regression analysis setting, the consequences of falsely assuming independent censoring on parameter estimates are not clear. Neither the direction nor the magnitude of the potential bias can be easily predicted. We provide an approach to do sensitivity analysis for the widely used Cox proportional hazards models. The joint distribution of the failure and censoring times is assumed to be a function of their marginal distributions. This function is called a copula. Under this assumption, we propose an iteration algorithm to estimate the regression parameters and marginal survival functions. Simulation studies show that this algorithm works well. We apply the proposed sensitivity analysis approach to the data from an AIDS clinical trial in which 27% of the patients withdrew due to toxicity or at the request of the patient or investigator.     

Equivalence of regression curves sharing common parameters

In clinical trials, the comparison of two different populations is a common problem. Nonlinear (parametric) regression models are commonly used to describe the relationship between covariates, such as concentration or dose, and a response variable in the two groups. In some situations, it is reasonable to assume some model parameters to be the same, for instance, the placebo effect or the maximum treatment effect. In this paper, we develop a (parametric) bootstrap test to establish the similarity of two regression curves sharing some common parameters. We show by theoretical arguments and by means of a simulation study that the new test controls its significance level and achieves a reasonable power. Moreover, it is demonstrated that under the assumption of common parameters, a considerably more powerful test can be constructed compared with the test that does not use this assumption. Finally, we illustrate the potential applications of the new methodology by a clinical trial example.     

Some remarks about the analysis of active control studies

In an active-controlled trial, the experimental treatment can be declared to be non-inferior to the control if the confidence interval for the difference excludes a fixed pre-specified margin. Recently, some articles have discussed an alternative method where the data from the current study and placebo-controlled studies for the active control are combined together into a single test statistic to test whether a fixed fraction of the effect of the active control is preserved. It has been shown that, conditional on nuisance parameters from the active-controlled study, a fixed margin can be defined that will be operationally equivalent to this latter method. In this article, we will discuss statistical properties associated with these approaches. Specifically, the interim monitoring boundaries and level of evidence will be considered.     

Statistical methods for the analysis of two-arm non-inferiority trials with binary outcomes

The aim of this contribution is to give an overview of approaches to testing for non-inferiority of one out of two binomial distributions as compared to the other in settings involving independent samples (the paired samples case is not considered here but the major conclusions and recommendations can be shown to hold for both sampling schemes). In principle, there is an infinite number of different ways of defining (one-sided) equivalence in any multiparameter setting. In the binomial two-sample problem, the following three choices of a measure of dissimilarity between the underlying distributions are of major importance for real applications: the odds ratio (OR), the relative risk (RR), and the difference (DEL) of both binomial parameters. It is shown that for all three possibilities of formulating the hypotheses of a non-inferiority problem concerning two binomial proportions, reasonable testing procedures providing exact control over the type-I error risk are available. As a particularly useful and versatile way of handling mathematically nonnatural parametrizations like RR and DELTA, the approach through Bayesian posterior probabilities of hypotheses with respect to some non-informative reference prior has much to recommend it. In order to ensure that the corresponding testing procedure be valid in the classical, i.e. frequentist sense, it suffices to use straightforward computational techniques yielding suitably corrected nominal significance levels. In view of the availability of testing procedures with satisfactory properties for all parametrizations of main practical interest, the discussion of the pros and cons of these methods has to focus on the question of which of the underlying measures of dissimilarity should be preferred on grounds of logic and intuition. It is argued that the OR clearly merits to be given preference also with regard to this latter kind of criteria since the non-inferiority hypotheses defined in terms of the other parametric functions are bounded by lines which cross the boundaries of the parameter space. From this fact, we conclude that the exact Fisher type test for one-sided equivalence provides the most reasonable approach to the confirmatory analysis of non-inferiority trials involving two independent samples of binary data. The marked conservatism of the nonrandomized version of this test can largely be removed by using a suitably increased nominal significance level (depending, in addition to the target level, on the sample sizes and the equivalence margin), or by replacing it with a Bayesian test for non-inferiority with respect to the odds ratio.     

A group sequential type design for three‐arm non‐inferiority trials with binary endpoints

The three‐arm design with a test treatment, an active control and a placebo group is the gold standard design for non‐inferiority trials if it is ethically justifiable to expose patients to placebo. In this paper, we first use the closed testing principle to establish the hierarchical testing procedure for the multiple comparisons involved in the three‐arm design. For the effect preservation test we derive the explicit formula for the optimal allocation ratios. We propose a group sequential type design, which naturally accommodates the hierarchical testing procedure. Under this proposed design, Monte Carlo simulations are conducted to evaluate the performance of the sequential effect preservation test when the variance of the test statistic is estimated based on the restricted maximum likelihood estimators of the response rates under the null hypothesis. When there are uncertainties for the placebo response rate, the proposed design demonstrates better operating characteristics than the fixed sample design.     

Re‐Formulating Non‐inferiority Trials as Superiority Trials: The Case of Binary Outcomes

Non‐inferiority trials are conducted for a variety of reasons including to show that a new treatment has a negligible reduction in efficacy or safety when compared to the current standard treatment, or a more complex setting of showing that a new treatment has a negligible reduction in efficacy when compared to the current standard yet is superior in terms of other treatment characteristics. The latter reason for conducting a non‐inferiority trial presents the challenge of deciding on a balance between a suitable reduction in efficacy, known as the non‐inferiority margin, in return for a gain in other important treatment characteristics/findings. It would be ideal to alleviate the dilemma on the choice of margin in this setting by reverting to a traditional superiority trial design where a single p ‐value for superiority of both the most important endpoint (efficacy) and the most important finding (treatment characteristic) is provided. We discuss how this can be done using the information‐preserving composite endpoint (IPCE) approach and consider binary outcome cases in which the combination of efficacy and treatment characteristics, but not one itself, paints a clear picture that the novel treatment is superior to the active control (© 2009 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Randomised Controlled Trials May Underestimate Drug Effects: Balanced Placebo Trial Design

Background

It is an inherent assumption in randomised controlled trials that the drug effect can be estimated by subtracting the response during placebo from the response during active drug treatment.

Objective

To test the assumption of additivity. The primary hypothesis was that the total treatment effect is smaller than the sum of the drug effect and the placebo effect. The secondary hypothesis was that non-additivity was most pronounced in participants with large placebo effects.

Methods

We used a within-subject randomised blinded balanced placebo design and included 48 healthy volunteers (50% males), mean (SD) age 23.4 (6.2) years. Experimental pain was induced by injections of hypertonic saline into the masseter muscle. Participants received four injections with hypertonic saline along with lidocaine or matching placebo in randomised order: A: received hypertonic saline/told hypertonic saline; B: received hypertonic saline+lidocaine/told hypertonic saline; C: received hypertonic saline+placebo/told hypertonic saline+pain killer; D: received hypertonic saline+lidocaine/told hypertonic saline+pain killer. The primary outcome measure was the area under the curve (AUC, mm²) of pain intensity during injections.

Results

There was a significant difference between the sum of the drug effect and the placebo effect (mean AUC 6279 mm² (95% CI, 4936–7622)) and the total treatment effect (mean AUC 5455 mm² (95% CI, 4585–6324)) (P = 0.049). This difference was larger for participants with large versus small placebo effects (P = 0.015), and the difference correlated significantly with the size of the placebo effect (r = 0.65, P = 0.006).

Conclusion

Although this study examined placebo effects and not the whole placebo response as in randomised controlled trials, it does suggest that the additivity assumption may be incorrect, and that the estimated drug effects in randomised controlled trials may be underestimated, particularly in studies reporting large placebo responses. The implications for randomised controlled trials and systematic reviews need to be discussed.     

Statin therapy and stroke prevention: what was known, what is new and what is next?

PURPOSE OF REVIEW: Randomized trials have shown that statins may reduce the risk of primary stroke. There is no evidence however that statins can reduce recurrent stroke incidence. RECENT FINDINGS: In the SPARCL trial, patients with a recent stroke or transient ischemic attack randomized to atorvastatin 80 mg/day had a significant 16% relative risk reduction of stroke, and a 35% reduction in major coronary events compared with placebo. This was obtained despite 25% of the placebo arm patients receiving a commercially-available statin outside of the trial. Post-hoc analysis used blinded LDL-cholesterol measurements as a marker of adherence to lipid-lowering therapy. Compared with the group with no change or an increase in LDL-cholesterol (the group adherent to placebo or not taking a statin), the group with over 50% reduction in LDL-cholesterol had a significant 31% reduction in stroke. The next step is to define whether achieving LDL-cholesterol below 70 mg/dl is better than a standard dose of statin (LDL around 100-110 mg/dl) in the secondary prevention of stroke. SUMMARY: Statins are effective in reducing both first-ever and recurrent stroke, and this effect seems driven by the extent of LDL-cholesterol lowering.