The consumer's risk in clinical trials

In any formal statistical test of the null hypothesis (the statement that a population parameter is equal to a specific value), there are two possible types of error. Type 1 or alpha error has occurred if the investigator rejects the null hypothesis when it is true. For example, an experimental treatment is declared an advance over standard treatment when it is not. Type 2 or beta error has occurred if the null hypothesis is not rejected when it is false. In this case, the investigator concludes that the experimental treatment is no different than the standard when it actually is. The two types of error can be conceptualized, respectively, as the consumer's risk and the producer's risk. In many reports of clinical trial methodology, it is the producer's risk that is emphasized. It is understandable why producer's risk would be of concern to authors of clinical studies. There are, however, numerous potential sources of consumer's risk. It is the latter type of risk that is the primary subject of this report.     

The performance of a two‐stage analysis of ABAB/BABA crossover trials

Freeman has considered the following two‐stage procedure for finding a confidence interval for the treatment difference theta, using data from an AB/BA crossover trial. In the first stage, a preliminary test of the null hypothesis that the differential carryover is zero is carried out. If this hypothesis is accepted then the confidence interval for theta is constructed assuming that the differential carryover is zero. If, on the other hand, this hypothesis is rejected then this confidence interval is constructed using only data from the first period. Freeman has shown that this confidence interval has minimum coverage probability far below nominal. He therefore concludes that this confidence interval should not be used. In the present paper, we analyze the performance of a similar two‐stage procedure for an ABAB/BABA crossover trial. This trial differs in very significant ways from an AB/BA crossover trial, including the fact that for an ABAB/BABA crossover trial there is an unbiased estimator of the differential carryover that is unaffected by between‐subject variation. Despite these great differences, we arrive at the same conclusion as Freeman. Namely, that the confidence interval resulting from the two‐stage procedure should not be used.     

Nonparametric Methods in Crossover Trials

The crossover design is often used in biomedical trials since it eliminates between subject variability. This paper is concerned with the statistical analysis of data arising from such trials when assumptions like normality do not necessarily apply. Nonparametric analysis of the two-period, two-treatment design was first described by Koch in a paper 1972. The purpose of this paper is to study nonparametric methods in crossover designs with three or more treatments and an equal number of periods. The proposed test for direct treatment effects is based on within subject comparisons after removing a possible period effect. With only two treatments this test reduces to the twosided Wilcoxon signed rank test. By simulation experiments the validity of the significance level of the test when using the asymptotic distribution of the test statistic are manifested and the power against different alternatives illustrated. A test for first order carryover effects can be constructed by a straightforward generalization of the test proposed by Koch in 1972. However, since this test is based on between subject comparisons its power will be low. Our recommendation is to consider the crossover design rather than the parallel group design if the carryover effects are assumed to be neglible or positive and smaller then the direct treatment effects.     

Clinical trials and rare diseases: a way out of a conundrum.

Currently, clinical trials tend to be individually funded and applicants must include a power calculation in their grant request. However, conventional levels of statistical precision are unlikely to be obtainable prospectively if the trial is required to evaluate treatment of a rare disease. This means that clinicians treating such diseases remain in ignorance and must form their judgments solely on the basis of (potentially biased) observational studies experience, and anecdote. Since some unbiased evidence is clearly better than none, this state of affairs should not continue. However, conventional (frequentist) confidence limits are unlikely to exclude a null result, even when treatments differ substantially. Bayesian methods utilise all available data to calculate probabilities that may be extrapolated directly to clinical practice. Funding bodies should therefore fund a repertoire of small trials, which need have no predetermined end, alongside standard larger studies.     

Bayesian design and analysis of active control clinical trials

We consider the design and analysis of active control clinical trials, i.e., clinical trials comparing an experimental treatment E to a control treatment C considered to be effective. Direct comparison of E to placebo P, or no treatment, is sometimes ethically unacceptable. Much discussion of the design and analysis of such clinical trials has focused on whether the comparison of E to C should be based on a test of the null hypothesis of equivalence, on a test of a nonnull hypothesis that the difference is of some minimally medically important size delta, or on one or two-sided confidence intervals. These approaches are essentially the same for study planning. They all suffer from arbitrariness in specifying the size of the difference delta that must be excluded. We propose an alternative Bayesian approach to the design and analysis of active control trials. We derive the posterior probability that E is superior to P or that E is at least k% as good as C and that C is more effective than P. We also derive approximations for use with logistic and proportional hazard models. Selection of prior distributions is discussed, and results are illustrated using data from an active control trial of a drug for the treatment of unstable angina.     

Biofeedback efficacy studies

Biofeedback, a field still in its infancy, has developed treatments that have been used with clinical success in the treatment of a number of disorders. Many have expressed their public concern that biofeedback has not lived up to its early promise and that it has not developed treatments that are, in fact, efficacious. A number of factors, which are inherent in biofeedback research, confound the results of clinical efficacy studies of biofeedback treatments. Researchers interested in the efficacy of biofeedback must address several issues: (1) Rejecting the null hypothesis is not equal to proving the null hypothesis (without the use of power analysis); (2) control for nonspecific effects is not equal to a double-blind experimental design; (3) ignorance of a mechanism of action is not equal to a lack of clinical efficacy; (4) the administration of training is not equal to the subject's learning to criterion; (5) untrained therapists are not equal to trained therapists; (6) statistical significance is not equal to clinical significance; and (7) the laboratory setting is not equal to the clinical setting.     

A Note on the Analysis of the Two-Period Crossover Design When the Period-Treatment Interaction is Significant

In the analysis of a two-period crossover study Grizzle (1965) suggests that, if a preliminary test for a period by treatment interaction (residual effect, carryover effect) is significant at the 10 % level, the direct effects of the treatments should be compared by performing a t-test on the data from the first period only. In this note it is shown that under Grizzle's model the comparison of the direct treatment effects is equivalent to a Behrens-Fisher problem. Depending on one's viewpoint–Bayesian, fiducial, or sampling theory–different solutions are possible. The solutions are illustrated using three well-known data sets.     

Comparing relative rates of pollen and seed gene flow in the island model using nuclear and organelle measures of population structure

We describe a method for comparing nuclear and organelle population differentiation (F(ST)) in seed plants to test the hypothesis that pollen and seed gene flow rates are equal. Wright's infinite island model is used, with arbitrary levels of self-fertilization and biparental organelle inheritance. The comparison can also be applied to gene flow in animals. Since effective population sizes are smaller for organelle genomes than for nuclear genomes and organelles are often uniparentally inherited, organelle F(ST) is expected to be higher at equilibrium than nuclear F(ST) even if pollen and seed gene flow rates are equal. To reject the null hypothesis of equal seed and pollen gene flow rates, nuclear and organelle F(ST)'s must differ significantly from their expected values under this hypothesis. Finite island model simulations indicate that infinite island model expectations are not greatly biased by finite numbers of populations (>/=100 subpopulations). The power to distinguish dissimilar rates of pollen and seed gene flow depends on confidence intervals for fixation index estimates, which shrink as more subpopulations and loci are sampled. Using data from the tropical tree Corythophora alta, we rejected the null hypothesis that seed and pollen gene flow rates are equal but cannot reject the alternative hypothesis that pollen gene flow is 200 times greater than seed gene flow.     

A nonparametric bootstrap method for testing close linkage vs. pleiotropy of coincident quantitative trait loci.

A novel method using the nonparametric bootstrap is proposed for testing whether a quantitative trait locus (QTL) at one chromosomal position could explain effects on two separate traits. If the single-QTL hypothesis is accepted, pleiotropy could explain the effect on two traits. If it is rejected, then the effects on two traits are due to linked QTLs. The method can be used in conjunction with several QTL mapping methods as long as they provide a straightforward estimate of the number of QTLs detectable from the data set. A selection step was introduced in the bootstrap procedure to reduce the conservativeness of the test of close linkage vs. pleiotropy, so that the erroneous rejection of the null hypothesis of pleiotropy only happens at a frequency equal to the nominal type I error risk specified by the user. The approach was assessed using computer simulations and proved to be relatively unbiased and robust over the range of genetic situations tested. An example of its application on a real data set from a saline stress experiment performed on a recombinant population of wheat (Triticum aestivum L. ) doubled haploid lines is also provided.     

Bayesian inferences on umbrella orderings

In regression applications with categorical predictors, interest often focuses on comparing the null hypothesis of homogeneity to an ordered alternative. This article proposes a Bayesian approach for addressing this problem in the setting of normal linear and probit regression models. The regression coefficients are assigned a conditionally conjugate prior density consisting of mixtures of point masses at 0 and truncated normal densities, with a (possibly unknown) changepoint parameter included to accommodate umbrella ordering. Two strategies of prior elicitation are considered: (1) a Bayesian Bonferroni approach in which the probability of the global null hypothesis is specified and local hypotheses are considered independent; and (2) an approach which treats these probabilities as random. A single Gibbs sampling chain can be used to obtain posterior probabilities for the different hypotheses and to estimate regression coefficients and predictive quantities either by model averaging or under the preferred hypothesis. The methods are applied to data from a carcinogenesis study.     

Group sequential methods for an ordinal logistic random-effects model under misspecification

Interim analyses in clinical trials are planned for ethical as well as economic reasons. General results have been published in the literature that allow the use of standard group sequential methodology if one uses an efficient test statistic, e.g., when Wald-type statistics are used in random-effects models for ordinal longitudinal data. These models often assume that the random effects are normally distributed. However, this is not always the case. We will show that, when the random-effects distribution is misspecified in ordinal regression models, the joint distribution of the test statistics over the different interim analyses is still a multivariate normal distribution, but a sandwich-type correction to the covariance matrix is needed in order to obtain the correct covariance matrix. The independent increment structure is also investigated. A bias in estimation will occur due to the misspecification. However, we will also show that the treatment effect estimate will be unbiased under the null hypothesis, thus maintaining the type I error. Extensive simulations based on a toenail dermatophyte onychomycosis trial are used to illustrate our results.     

Using phylogeographic analyses of gene trees to test species status and processes

A gene tree is an evolutionary reconstruction of the genealogical history of the genetic variation found in a sample of homologous genes or DNA regions that have experienced little or no recombination. Gene trees have the potential of straddling the interface between intra- and interspecific evolution. It is precisely at this interface that the process of speciation occurs, and gene trees can therefore be used as a powerful tool to probe this interface. One application is to infer species status. The cohesion species is defined as an evolutionary lineage or set of lineages with genetic exchangeability and/or ecological interchangeability. This species concept can be phrased in terms of null hypotheses that can be tested rigorously and objectively by using gene trees. First, an overlay of geography upon the gene tree is used to test the null hypothesis that the sample is from a single evolutionary lineage. This phase of testing can indicate that the sampled organisms are indeed from a single lineage and therefore a single cohesion species. In other cases, this null hypothesis is not rejected due to a lack of power or inadequate sampling. Alternatively, this null hypothesis can be rejected because two or more lineages are in the sample. The test can identify lineages even when hybridization and lineage sorting occur. Only when this null hypothesis is rejected is there the potential for more than one cohesion species. Although all cohesion species are evolutionary lineages, not all evolutionary lineages are cohesion species. Therefore, if the first null hypothesis is rejected, a second null hypothesis is tested that all lineages are genetically exchangeable and/or ecologically interchangeable. This second test is accomplished by direct contrasts of previously identified lineages or by overlaying reproductive and/or ecological data upon the gene tree and testing for significant transitions that are concordant with the previously identified lineages. Only when this second null hypothesis is rejected is a lineage elevated to the status of cohesion species. By using gene trees in this manner, species can be identified with objective, a priori criteria with an inference procedure that automatically yields much insight into the process of speciation. When one or more of the null hypotheses cannot be rejected, this procedure also provides specific guidance for future work that will be needed to judge species status.     

A test for constant fatality rate of an emerging epidemic: with applications to severe acute respiratory syndrome in Hong Kong and Beijing

Summary .   The etiology, pathogenesis, and prognosis for a newly emerging disease are generally unknown to clinicians. Effective interventions and treatments at the earliest possible times are warranted to suppress the fatality of the disease to a minimum, and inappropriate treatments should be abolished. In this situation, the ability to extract most information out of the data available is critical so that important decisions can be made. Ineffectiveness of the treatment can be reflected by a constant fatality over time while effective treatment normally leads to a decreasing fatality rate. A statistical test for constant fatality over time is proposed in this article. The proposed statistic is shown to converge to a Brownian motion asymptotically under the null hypothesis. With the special features of the Brownian motion, we are able to analyze the first passage time distribution based on a sequential tests approach. This allows the null hypothesis of constant fatality rate to be rejected at the earliest possible time when adequate statistical evidence accumulates. Simulation studies show that the performance of the proposed test is good and it is extremely sensitive in picking up decreasing fatality rate. The proposed test is applied to the severe acute respiratory syndrome data in Hong Kong and Beijing.     

One-Sided Tests in Clinical Trials with Multiple Endpoints

Treatment comparisons in clinical trials often involve several endpoints. For example, one might wish to demonstrate that a new treatment is superior to the current standard for some components of the multivariate response vector and is not inferior, modulo biologically unimportant difference to the standard treatment for all other components. We introduce a new approach to multiple-endpoint testing that incorporates the essential univariate and multivariate features of the treatment effects. This approach is compared with existing methods in a simulation study and applied to data on rheumatoid arthritis patients receiving one of two treatments.     

Bayesian Data Analysis: A Fresh Approach to Power Issues and Null Hypothesis Interpretation

One of the first things one learns in a basic psychology or statistics course is that you cannot prove the null hypothesis that there is no difference between two conditions such as a patient group and a normal control group. This remains true. However now, thanks to ongoing progress by a special group of devoted methodologists, even when the result of an inferential test is p?>?.05, it is now possible to rigorously and quantitatively conclude that (a) the null hypothesis is actually unlikely, and (b) that the alternative hypothesis of an actual difference between treatment and control is more probable than the null. Alternatively, it is also possible to conclude quantitatively that the null hypothesis is much more likely than the alternative. Without Bayesian statistics, we couldn’t say anything if a simple inferential analysis like a t-test yielded p?>?.05. The present, mostly non-quantitative article describes free resources and illustrative procedures for doing Bayesian analysis, with t-test and ANOVA examples.

     

Optimal crossover designs in the presence of carryover effects

Under either the random patient-effect model with sequence effects or the fixed patient-effect model, the usual two-period, two-treatment crossover design, AB,BA, cannot be used to estimate the contrast between direct treatment effects when unequal carryover effects are present. If baseline observations are available, the design AB,BA can validly be used to estimate a treatment contrast. However, the design AB,BA,AA,BB with baseline observations is more efficient. In fact, we show that this design is optimal whether or not baseline observations are available. For experiments with more than two periods, universally optimal designs are found for both models, with and without carryover effects. It is shown that uncertainty about the presence of carryover effects is of little or no consequence, and the addition of baseline observations is of little or no added value for designs with three or more periods; however, if the experiment is limited to only two periods the investigator pays a heavy penalty.     

The ARTS (Arterial Revascularization Therapies Study): Background,goals and methods

BACKGROUND: The rising costs of healthcare have forced policy makers to make choices, and new treatments are increasingly assessed in terms of the balance between additional costs and additional effects. The recent recognition that stenting has a major and long-lasting effect enhancing balloon PTCA procedure has made it imperative to compare in patients with multivessel disease the standard surgical procedure with multiple stenting in a large-scale multinational and multicenter approach (19 countries, 68 sites). METHODS: Selection and inclusion of patients is based on a consensus of the cardiac surgeon and interventional cardiologist on equal 'treatability' of patients by both techniques with analysis of clinical follow-up (event-free survival) on the short (30 days), medium (1 year), and long term (3 and 5 years) with analysis of cost-effectiveness and quality of life (EuroQol and SF-36). Of the entire trial, the primary null hypothesis which needs to be rejected is that there will be no difference in event-free survival or effectiveness (E), at 1 year and also that the direct and indirect costs (C) per event-free year are not different between surgery or stenting. For this to become significant with a power of 90% requires 1200 patients. Between April 97 and June 98, 1205 patients have been randomized with a monthly recruitment of 83 patients; the one year follow-up will thus be completed in June 1999. Expected costs, effects and cost-effectiveness ratio (CE ratio) for stents are: Stent: high-cost estimate, 2 vessel disease (C 3 $19,297, E 3 81%, CE ratio 3 $23,876); 3 vessel disease (C 3 $24,566, E 3 81%, CE ratio 3 $30,397) low-cost estimate, 2 vessel disease (C 3 $16,638, E 3 81%, CE ratio 3 $20,586); 3 vessel disease (C 3 $20,456, E 3 81%, CE ratio 3 $25,322) Compared with CABG (C 3 $21,350, E 3 88%, CE ratio 3 $24,348) CONCLUSION: Clinically, stenting is not expected to be more effective than CABG, but should be cost-effective in both the 2- and 3-vessel disease groups when using the lower-cost estimate and in the 2 vessel group when using the higher-cost assumptions. (Int J Cardiovasc     

P > .05: The incorrect interpretation of “not significant” results is a significant problem

Statistically nonsignificant (p > .05) results from a null hypothesis significance test (NHST) are often mistakenly interpreted as evidence that the null hypothesis is true—that there is “no effect” or “no difference.” However, many of these results occur because the study had low statistical power to detect an effect. Power below 50% is common, in which case a result of no statistical significance is more likely to be incorrect than correct. The inference of “no effect” is not valid even if power is high. NHST assumes that the null hypothesis is true; p is the probability of the data under the assumption that there is no effect. A statistical test cannot confirm what it assumes. These incorrect statistical inferences could be eliminated if decisions based on p values were replaced by a biological evaluation of effect sizes and their confidence intervals. For a single study, the observed effect size is the best estimate of the population effect size, regardless of the p value. Unlike p values, confidence intervals provide information about the precision of the observed effect. In the biomedical and pharmacology literature, methods have been developed to evaluate whether effects are “equivalent,” rather than zero, as tested with NHST. These methods could be used by biological anthropologists to evaluate the presence or absence of meaningful biological effects. Most of what appears to be known about no difference or no effect between sexes, between populations, between treatments, and other circumstances in the biological anthropology literature is based on invalid statistical inference.     

Wavelet thresholding with bayesian false discovery rate control

The false discovery rate (FDR) procedure has become a popular method for handling multiplicity in high-dimensional data. The definition of FDR has a natural Bayesian interpretation; it is the expected proportion of null hypotheses mistakenly rejected given a measure of evidence for their truth. In this article, we propose controlling the positive FDR using a Bayesian approach where the rejection rule is based on the posterior probabilities of the null hypotheses. Correspondence between Bayesian and frequentist measures of evidence in hypothesis testing has been studied in several contexts. Here we extend the comparison to multiple testing with control of the FDR and illustrate the procedure with an application to wavelet thresholding. The problem consists of recovering signal from noisy measurements. This involves extracting wavelet coefficients that result from true signal and can be formulated as a multiple hypotheses-testing problem. We use simulated examples to compare the performance of our approach to the Benjamini and Hochberg (1995, Journal of the Royal Statistical Society, Series B57, 289-300) procedure. We also illustrate the method with nuclear magnetic resonance spectral data from human brain.     

Statistical methods in recent HIV noninferiority trials: reanalysis of 11 trials

Background

In recent years the “noninferiority” trial has emerged as the new standard design for HIV drug development among antiretroviral patients often with a primary endpoint based on the difference in success rates between the two treatment groups. Different statistical methods have been introduced to provide confidence intervals for that difference. The main objective is to investigate whether the choice of the statistical method changes the conclusion of the trials.

Methods

We presented 11 trials published in 2010 using a difference in proportions as the primary endpoint. In these trials, 5 different statistical methods have been used to estimate such confidence intervals. The five methods are described and applied to data from the 11 trials. The noninferiority of the new treatment is not demonstrated if the prespecified noninferiority margin it includes in the confidence interval of the treatment difference.

Results

Results indicated that confidence intervals can be quite different according to the method used. In many situations, however, conclusions of the trials are not altered because point estimates of the treatment difference were too far from the prespecified noninferiority margins. Nevertheless, in few trials the use of different statistical methods led to different conclusions. In particular the use of “exact” methods can be very confusing.

Conclusion

Statistical methods used to estimate confidence intervals in noninferiority trials have a strong impact on the conclusion of such trials.