Simultaneous confidence regions corresponding to Holm's step-down procedure and other closed-testing procedures

Holm's (1979) step-down multiple-testing procedure (MTP) is appealing for its flexibility, transparency, and general validity, but the derivation of corresponding simultaneous confidence regions has remained an unsolved problem. This article provides such confidence regions. In fact, simultanenous confidence regions are provided for any MTP in the class of short-cut consonant closed-testing procedures based on marginal p -values and weighted Bonferroni tests for intersection hypotheses considered by Hommel, Bretz and Maurer (2007). In addition to Holm's MTP, this class includes the fixed-sequence MTP, recently proposed gatekeeping MTPs, and the fallback MTP. The simultaneous confidence regions are generally valid if underlying marginal p -values and corresponding marginal confidence regions (assumed to be available) are valid. The marginal confidence regions and estimated quantities are not assumed to be of any particular kinds/dimensions. Compared to the rejections made by the MTP for the family of null hypotheses H under consideration, the proposed confidence regions provide extra free information. In particular, with Holm's MTP, such extra information is provided: for all nonrejected H s, in case not all H s are rejected; or for certain (possibly all) H s, in case all H s are rejected. In case not all H s are rejected, no extra information is provided for rejected H s. This drawback seems however difficult to overcome. Illustrations concerning clinical studies are given.     

Alternative Confidence Regions for Bonferroni‐Based Closed‐Testing Procedures that are not Alpha‐Exhaustive

This article complements the results in Guilbaud (Biometrical Journal 2008; 50 :678–692). Simultaneous confidence regions were derived in that article that correspond to any given multiple testing procedure (MTP) in a fairly large class of consonant closed‐testing procedures based on marginal p‐values and weighted Bonferroni tests for intersection hypotheses. This class includes Holm's MTP, the fixed‐sequence MTP, gatekeeping MTPs, fallback MTPs, multi‐stage fallback MTPs, and recently proposed MTPs specified through a graphical representation and associated rejection algorithm. More general confidence regions are proposed in this article. These regions are such that for certain underlying MTPs which are not alpha‐exhaustive, they lead to confidence assertions that may be sharper than rejection assertions for some rejected null hypotheses H when not all Hs are rejected, which is not the case with the previously proposed regions. In fact, various alternative confidence regions may be available for such an underlying MTP. These results are shown through an extension of the previous direct arguments (without invoking the partitioning principle), and under the same general setup; so for instance, estimated quantities and marginal confidence regions are not restricted to be of any particular kinds/dimensions. The relation with corresponding confidence regions of Strassburger and Bretz (Statistics in Medicine 2008; 27 :4914–4927) is described. The results are illustrated with fallback and parallel‐gatekeeping MTPs.     

Bonferroni parallel gatekeeping - transparent generalizations, adjusted p -values, and short direct proofs

In the two-step version (Dmitrienko, Tamhane, Wang and Chen, 2006) of the Bonferroni parallel-gatekeeping multiple-testing procedure (MTP): (a) a family F1 of null hypotheses H is used as a gatekeeper for another family F2 in that no H in F2 can be rejected unless at least one H is rejected in F1; (b) a Bonferroni MTP is used for F1 at local multiple-level alpha in the first step; and (c) Holm's (1979) step-down MTP is used in the second step for F2 at a local multiple level that depends on the rejections made in the first step. It is shown in this article that this two-step procedure can be generalized in that any MTP with multiple-level control and available multiplicity-adjusted p -values can be used instead of Holm's MTP in the second step. A further generalization related to what Dmitrienko, Molenberghs, Chuang-Stein and Offen (2005) called modified Bonferroni parallel gatekeeping is also given where in case all H s in F2 are rejected, additional rejections in F1 can be made in a third step at local multiple-level alpha through any MTP that is more powerful than the initial Bonferroni MTP, e.g. Holm's MTP. The proofs that these two generalized Bonferroni parallel-gatekeeping MTPs have multiple-level alpha are short and direct, without closed-testing arguments. Multiplicity-adjusted p -values can easily be calculated for these MTPs. The extensions to several successive gatekeeper families are straightforward. An illustration is given.     

Stepwise gatekeeping procedures in clinical trial applications

This paper discusses multiple testing problems in which families of null hypotheses are tested in a sequential manner and each family serves as a gatekeeper for the subsequent families. Gatekeeping testing strategies of this type arise frequently in clinical trials with multiple objectives, e.g., multiple endpoints and/or multiple dose-control comparisons. It is demonstrated in this paper that the parallel gatekeeping procedure of Dmitrienko, Offen and Westfall (2003) admits a simple stepwise representation (n null hypotheses can be tested in n steps rather than 2n steps required in the closed procedure). The stepwise representation considerably simplifies the implementation of gatekeeping procedures in practice and provides an important insight into the nature of gatekeeping inferences. The derived stepwise gatekeeping procedure is illustrated using clinical trial examples.     

A Note on Plots of P-Values to Evaluate Many Tests Simultaneously

Schweder and Spjøtvoll (1982) proposed an informal graphical procedure for simultaneous evaluation of possibly related tests, based on a plot of cumulative p-values using the observed significance probabilities. We formalize this notion by application of Holm's (1979) sequentially rejective Bonferroni procedure: this maintains an overall experimentwise significance level, and yields an immediate estimate of the number of true hypotheses.     

Graphical approaches for multiple comparison procedures using weighted Bonferroni, Simes, or parametric tests

The confirmatory analysis of pre-specified multiple hypotheses has become common in pivotal clinical trials. In the recent past multiple test procedures have been developed that reflect the relative importance of different study objectives, such as fixed sequence, fallback, and gatekeeping procedures. In addition, graphical approaches have been proposed that facilitate the visualization and communication of Bonferroni-based closed test procedures for common multiple test problems, such as comparing several treatments with a control, assessing the benefit of a new drug for more than one endpoint, combined non-inferiority and superiority testing, or testing a treatment at different dose levels in an overall and a subpopulation. In this paper, we focus on extended graphical approaches by dissociating the underlying weighting strategy from the employed test procedure. This allows one to first derive suitable weighting strategies that reflect the given study objectives and subsequently apply appropriate test procedures, such as weighted Bonferroni tests, weighted parametric tests accounting for the correlation between the test statistics, or weighted Simes tests. We illustrate the extended graphical approaches with several examples. In addition, we describe briefly the gMCP package in R, which implements some of the methods described in this paper.     

Trimmed Weighted Simes' Test for Two One‐Sided Hypotheses With Arbitrarily Correlated Test Statistics

The two‐sided Simes test is known to control the type I error rate with bivariate normal test statistics. For one‐sided hypotheses, control of the type I error rate requires that the correlation between the bivariate normal test statistics is non‐negative. In this article, we introduce a trimmed version of the one‐sided weighted Simes test for two hypotheses which rejects if (i) the one‐sided weighted Simes test rejects and (ii) both p‐values are below one minus the respective weighted Bonferroni adjusted level. We show that the trimmed version controls the type I error rate at nominal significance level α if (i) the common distribution of test statistics is point symmetric and (ii) the two‐sided weighted Simes test at level 2α controls the level. These assumptions apply, for instance, to bivariate normal test statistics with arbitrary correlation. In a simulation study, we compare the power of the trimmed weighted Simes test with the power of the weighted Bonferroni test and the untrimmed weighted Simes test. An additional result of this article ensures type I error rate control of the usual weighted Simes test under a weak version of the positive regression dependence condition for the case of two hypotheses. This condition is shown to apply to the two‐sided p‐values of one‐ or two‐sample t‐tests for bivariate normal endpoints with arbitrary correlation and to the corresponding one‐sided p‐values if the correlation is non‐negative. The Simes test for such types of bivariate t‐tests has not been considered before. According to our main result, the trimmed version of the weighted Simes test then also applies to the one‐sided bivariate t‐test with arbitrary correlation.     

On the power of some binomial modifications of the Bonferroni multiple test

Widely used in testing statistical hypotheses, the Bonferroni multiple test has a rather low power that entails a high risk to accept falsely the overall null hypothesis and therefore to not detect really existing effects. We suggest that when the partial test statistics are statistically independent, it is possible to reduce this risk by using binomial modifications of the Bonferroni test. Instead of rejecting the null hypothesis when at least one of n partial null hypotheses is rejected at a very high level of significance (say, 0.005 in the case of n = 10), as it is prescribed by the Bonferroni test, the binomial tests recommend to reject the null hypothesis when at least k partial null hypotheses (say, k = [n/2]) are rejected at much lower level (up to 30-50%). We show that the power of such binomial tests is essentially higher as compared with the power of the original Bonferroni and some modified Bonferroni tests. In addition, such an approach allows us to combine tests for which the results are known only for a fixed significance level. The paper contains tables and a computer program which allow to determine (retrieve from a table or to compute) the necessary binomial test parameters, i.e. either the partial significance level (when k is fixed) or the value of k (when the partial significance level is fixed).     

A simple and flexible Holm gatekeeping procedure

Major objectives of a clinical trial are commonly stated in a hierarchical order as primary and secondary. The parallel gatekeeping testing strategy provides an opportunity to assess secondary objectives when all or partial primary objectives are achieved. The current available gatekeeping procedures have different pros and cons so users either need to justify the assumption associated with some procedures or tolerate suboptimal power performance of other procedures. By applying the Holm test with a flexible alpha splitting technique, we propose a procedure which (1) is powerful for assessing the primary objectives, (2) can be used when no assumption can be made on the dependency structure of test statistics, and (3) has the full flexibility to allocate user-preferred alpha to assess the secondary objectives based on the number of primary objectives achieved. A real clinical trial example is used for illustration of the proposed procedure.     

Nonparametric Evaluation of Learning Curves by Hierarchical Contingeney Analysis

Hierarchical contingency analysis (HCA) is derived from the Perli-Hommel-Lehmacher (1986) closed test procedure for nonparametrical evaluation of learning curves of a 2 x 2-factorial experiment. By HCA, univariate main effects are detected without Bonferroni alpha adjustment, as is shown by a numerical example from gold fish shock avoidance conditioning. Alternative approaches to nonparametrical evaluation of MANOVA designs with and without repeated measurements are discussed.     

A family of Bayes multiple testing procedures

Under the model of independent test statistics, we propose atwo-parameter family of Bayes multiple testing procedures. Thetwo parameters can be viewed as tuning parameters. Using theBenjamini–Hochberg step-up procedure for controlling falsediscovery rate as a baseline for conservativeness, we choosethe tuning parameters to compromise between the operating characteristicsof that procedure and a less conservative procedure that focuseson alternatives that a priori might be considered likely ormeaningful. The Bayes procedures do not have the theoreticaland practical shortcomings of the popular stepwise procedures.In terms of the number of mistakes, simulations for two examplesindicate that over a large segment of the parameter space, theBayes procedure is preferable to the step-up procedure. Anotherdesirable feature of the procedures is that they are computationallyfeasible for any number of hypotheses.     

To some reasonable test procedures in multiple contingency tables to investigate certain epidemiological or medicin-sociological relationships

One of the most important tasks of the application of mathematical-statistical methods consists in giving help in the search for possible relationships, and connected with this, the specification of new hypotheses. The progress of both the special diciplines of sciences and mathematical statistics itself leads to the application of more and more complex, that means multivariate, methods. In medical fields, especially in epidemiological and medicin-sociological studies, this fact means the necessity of analysing multidimensional contingency tables. The above formulated problem is equivalent to the problem of fitting an appropriate mathematical model (for contingency tables is this a log-linear model) to the data in a way which makes the structural relationships clear to us. In this paper it is shown that one is able to get to well-interpretable models of independence with relatively simple means. Two stepwise test procedures are described yielding essentially the same results: a so called reduction procedure which is particularly profitable in sparsely occupied tables and a procedure which uses a combination of hypotheses of conditional pairwise independence.     

Generalized Maximum Range Tests for Pairwise Comparisons of Several Populations

A multiple comparison procedure (MCP) is proposed for the comparison of all pairs of several independent samples. This MCP is essentially the closed procedure with union-intersection tests based on given single tests Q_ij for the minimal hypotheses H_ij. In such cases where the α-levels of the nominal tests associated with the MCP can be exhausted, this MCP has a uniformly higher all pair power than any refined Bonferroni test using the same Q_ij. Two different general algorithms are described in section 3. A probability inequality for ranges of i.i.d. random variables which is useful for some algorithms is proved in section 4. Section 5 contains the application to independent normally distributed estimates and section 6 the comparisons of polynomial distributions by multivariate ranges. Further applications are possible. Tables of the 0.05-bounds for the tests of section 5 and 6 are enclosed.     

Procedures for two-sample comparisons with multiple endpoints controlling the experimentwise error rate

Clinical trials are often concerned with the comparison of two treatment groups with multiple endpoints. As alternatives to the commonly used methods, the T2 test and the Bonferroni method, O'Brien (1984, Biometrics 40, 1079-1087) proposes tests based on statistics that are simple or weighted sums of the single endpoints. This approach turns out to be powerful if all treatment differences are in the same direction [compare Pocock, Geller, and Tsiatis (1987, Biometrics 43, 487-498)]. The disadvantage of these multivariate methods is that they are suitable only for demonstrating a global difference, whereas the clinician is further interested in which specific endpoints or sets of endpoints actually caused this difference. It is shown here that all tests are suitable for the construction of a closed multiple test procedure where, after the rejection of the global hypothesis, all lower-dimensional marginal hypotheses and finally the single hypotheses are tested step by step. This procedure controls the experimentwise error rate. It is just as powerful as the multivariate test and, in addition, it is possible to detect significant differences between the endpoints or sets of endpoints.     

Application of Multiple Test Procedures to the Combination of Multivariate and Univariate Tests with Varying Variable Sets

The positive ascertainment of location differences in a multivariate comparison of two or more groups gives rise to the question for the contribution of the single variables or of subsets of variables to the multivariate difference. In this paper two methods are proposed to accomplish the original multivariate test by tests in variable subsets or in single variables using a closed test procedure and Holm's procedure, respectively. Both control the multiple level of the whole procedure.     

Simultaneous confidence regions for closed tests,including Holm‐, Hochberg‐, and Hommel‐related procedures

The derivation of simultaneous confidence regions for some multiple‐testing procedures (MTPs) of practical interest has remained an unsolved problem. This is the case, for example, for Hochberg's step‐up MTP and Hommel's more powerful MTP that is neither a step‐up nor a step‐down procedure. It is shown in this article how the direct approach used previously by the author to construct confidence regions for certain closed‐testing procedures (CTPs) can be extended to a rather general setup. The general results are then applied to a situation with one‐sided inferences and CTPs belonging to a class studied by Wei Liu. This class consists of CTPs based on ordered marginal p‐values. It includes Holm's, Hochberg's, and Hommel's MTPs. A property of the confidence regions derived for these three MTPs is that no confidence assertions sharper than rejection assertions can be made unless all null hypotheses are rejected. Briefly, this is related to the fact that these MTPs are quite powerful. The class of CTPs considered includes, however, also MTPs related to Holm's, Hochberg's, and Hommel's MTPs that are less powerful but are such that confidence assertions sharper than rejection assertions are possible even if not all null hypotheses are rejected. One may thus choose and prespecify such an MTP, though this is at the cost of less rejection power.     

Test Procedures in Configural Frequency Analysis (CFA) Controlling the Local and Multiple Level

The test statistics used until now in the CFA have been developed under the assumption of the overall hypothesis of total independence. Therefore, the multiple test procedures based on these statistics are really only different tests of the overall hypothesis. If one likes to test a special cell hypothesis, one should only assume that this hypothesis is true and not the whole overall hypothesis. Such cell tests can then be used as elements of a multiple test procedure. In this paper it is shown that the usual test procedures can be very anticonservative (except of the two-dimensional, and, for some procedures, the three-dimensional case), and corrected test procedures are developed. Furthermore, for the construction of multiple tests controlling the multiple level, modifications of Holm's (1979) procedure are proposed which lead to sharper results than his general procedure and can also be performed very easily.     

A general method for controlling the genome-wide type I error rate in linkage and association mapping experiments in plants

Control of the genome-wide type I error rate (GWER) is an important issue in association mapping and linkage mapping experiments. For the latter, different approaches, such as permutation procedures or Bonferroni correction, were proposed. The permutation test, however, cannot account for population structure present in most association mapping populations. This can lead to false positive associations. The Bonferroni correction is applicable, but usually on the conservative side, because correlation of tests cannot be exploited. Therefore, a new approach is proposed, which controls the genome-wide error rate, while accounting for population structure. This approach is based on a simulation procedure that is equally applicable in a linkage and an association-mapping context. Using the parameter settings of three real data sets, it is shown that the procedure provides control of the GWER and the generalized genome-wide type I error rate (GWER(k)).     

Controlling False Discoveries in Multidimensional Directional Decisions,with Applications to Gene Expression Data on Ordered Categories

Summary Microarray gene expression studies over ordered categories are routinely conducted to gain insights into biological functions of genes and the underlying biological processes. Some common experiments are time‐course/dose‐response experiments where a tissue or cell line is exposed to different doses and/or durations of time to a chemical. A goal of such studies is to identify gene expression patterns/profiles over the ordered categories. This problem can be formulated as a multiple testing problem where for each gene the null hypothesis of no difference between the successive mean gene expressions is tested and further directional decisions are made if it is rejected. Much of the existing multiple testing procedures are devised for controlling the usual false discovery rate (FDR) rather than the mixed directional FDR (mdFDR), the expected proportion of Type I and directional errors among all rejections. Benjamini and Yekutieli (2005, Journal of the American Statistical Association 100, 71–93) proved that an augmentation of the usual Benjamini–Hochberg (BH) procedure can control the mdFDR while testing simple null hypotheses against two‐sided alternatives in terms of one‐dimensional parameters. In this article, we consider the problem of controlling the mdFDR involving multidimensional parameters. To deal with this problem, we develop a procedure extending that of Benjamini and Yekutieli based on the Bonferroni test for each gene. A proof is given for its mdFDR control when the underlying test statistics are independent across the genes. The results of a simulation study evaluating its performance under independence as well as under dependence of the underlying test statistics across the genes relative to other relevant procedures are reported. Finally, the proposed methodology is applied to a time‐course microarray data obtained by Lobenhofer et al. (2002, Molecular Endocrinology 16, 1215–1229). We identified several important cell‐cycle genes, such as DNA replication/repair gene MCM4 and replication factor subunit C2, which were not identified by the previous analyses of the same data by Lobenhofer et al. (2002) and Peddada et al. (2003, Bioinformatics 19, 834–841). Although some of our findings overlap with previous findings, we identify several other genes that complement the results of Lobenhofer et al. (2002) .     

More Specific Signal Detection in Functional Magnetic Resonance Imaging by False Discovery Rate Control for Hierarchically Structured Systems of Hypotheses

Signal detection in functional magnetic resonance imaging (fMRI) inherently involves the problem of testing a large number of hypotheses. A popular strategy to address this multiplicity is the control of the false discovery rate (FDR). In this work we consider the case where prior knowledge is available to partition the set of all hypotheses into disjoint subsets or families, e. g., by a-priori knowledge on the functionality of certain regions of interest. If the proportion of true null hypotheses differs between families, this structural information can be used to increase statistical power. We propose a two-stage multiple test procedure which first excludes those families from the analysis for which there is no strong evidence for containing true alternatives. We show control of the family-wise error rate at this first stage of testing. Then, at the second stage, we proceed to test the hypotheses within each non-excluded family and obtain asymptotic control of the FDR within each family at this second stage. Our main mathematical result is that this two-stage strategy implies asymptotic control of the FDR with respect to all hypotheses. In simulations we demonstrate the increased power of this new procedure in comparison with established procedures in situations with highly unbalanced families. Finally, we apply the proposed method to simulated and to real fMRI data.