Multiple comparisons in the randomization analysis of designed experiments with growth curve responses

A randomization approach to multiple comparisons is developed for comparing several growth curves in randomized experiments. The exact Type I probability error rate for these comparisons may be prespecified, and a Type I error probability for each component test can be evaluated. These procedures are free of many of the standard assumptions for analyzing growth curves and for making multiple comparisons. An application of the procedure gives all pairwise comparisons among the mean growth curves associated with four treatments in an animal experiment using a Youden square design, where growth curves are obtained on monitoring hormone levels over time.     

Simultaneous confidence intervals from randomization tests with application in testing bioequivalence with multiple endpoints

We propose a method to construct simultaneous confidence intervals for a parameter vector from inverting a series of randomization tests (RT). The randomization tests are facilitated by an efficient multivariate Robbins–Monro procedure that takes the correlation information of all components into account. The estimation method does not require any distributional assumption of the population other than the existence of the second moments. The resulting simultaneous confidence intervals are not necessarily symmetric about the point estimate of the parameter vector but possess the property of equal tails in all dimensions. In particular, we present the constructing the mean vector of one population and the difference between two mean vectors of two populations. Extensive simulation is conducted to show numerical comparison with four methods. We illustrate the application of the proposed method to test bioequivalence with multiple endpoints on some real data.     

Power and sample size estimation for the clustered wilcoxon test

The Wilcoxon rank sum test is widely used for two-group comparisons of nonnormal data. An assumption of this test is independence of sampling units both within and between groups, which will be violated in the clustered data setting such as in ophthalmological clinical trials, where the unit of randomization is the subject, but the unit of analysis is the individual eye. For this purpose, we have proposed the clustered Wilcoxon test to account for clustering among multiple subunits within the same cluster (Rosner, Glynn, and Lee, 2003, Biometrics 59, 1089-1098; 2006, Biometrics 62, 1251-1259). However, power estimation is needed to plan studies that use this analytic approach. We have recently published methods for estimating power and sample size for the ordinary Wilcoxon rank sum test (Rosner and Glynn, 2009, Biometrics 65, 188-197). In this article we present extensions of this approach to estimate power for the clustered Wilcoxon test. Simulation studies show a good agreement between estimated and empirical power. These methods are illustrated with examples from randomized trials in ophthalmology. Enhanced power is achieved with use of the subunit as the unit of analysis instead of the cluster using the ordinary Wilcoxon rank sum test.     

Testing for latitudinal and other body-size gradients

Testing for a relationship between the body size of animals and a gradient such as latitude is complicated by the fact that typically there is a single average size for each species, and each species occurs at several sample stations over the gradient. This results in standard tests for statistical significance being invalid. This problem can be overcome by using a randomization test. A more difficult problem, however, is determining whether the relationship between size and latitude is the same for two subfamilies of species. In this paper a general method for relating body size to latitude and subfamily differences is proposed, with the significance of effects determined by randomization. A simulation study suggests that this procedure has good properties. This approach to data analysis has promise both for the particular situation considered and for other related problems in biogeography.     

Random sampling of constrained phylogenies: conducting phylogenetic analyses when the phylogeny is partially known

Statistical randomization tests in evolutionary biology often require a set of random, computer-generated trees. For example, earlier studies have shown how large numbers of computer-generated trees can be used to conduct phylogenetic comparative analyses even when the phylogeny is uncertain or unknown. These methods were limited, however, in that (in the absence of molecular sequence or other data) they allowed users to assume that no phylogenetic information was available or that all possible trees were known. Intermediate situations where only a taxonomy or other limited phylogenetic information (e.g., polytomies) are available are technically more difficult. The current study describes a procedure for generating random samples of phylogenies while incorporating limited phylogenetic information (e.g., four taxa belong together in a subclade). The procedure can be used to conduct comparative analyses when the phylogeny is only partially resolved or can be used in other randomization tests in which large numbers of possible phylogenies are needed.     

Applying randomization tests to cluster analyses

Abstract. In applying randomization tests to hierarchical cluster analyses, we have noted a potentially misleading result: within a significant group, linkages are often identified as significant even when species are randomly distributed among the group's sites. We demonstrate this through a cluster analysis of a constructed matrix with two groups of 20 sites that share no species, and within each group species are randomly distributed among sites. A randomization test identified both of the groups and all linkages within them as significant, while the same test found all linkages non‐significant in the cluster analysis of a matrix containing just one of the two groups of 20 sites. In general, a non‐random distribution of species within a data set shortens linkages relative to distances in null distributions derived from randomized versions of the data. This confounds efforts to identify significant sub‐groups within a significant group. However, the significance of sub‐groups possibly could be tested by comparing linkage distances to a null distribution derived from the randomization and clustering of a sub‐matrix containing only the sites within the larger group. In essence, this comparison tests the null hypothesis that within the significant group, sites represent random assemblages of species. When applied to actual data sets, an approach involving sequential randomization tests could allow the evaluation of all nodes in a classification, increasing the utility of randomization tests and strengthening the interpretation of groups produced by cluster analysis.     

Split-test Bonferroni correction for QEEG statistical maps

With statistical testing, corrections for multiple comparisons, such as Bonferroni adjustments, have given rise to controversies in the scientific community, because of their negative impact on statistical power. This impact is especially problematic for high-multidimensional data, such as multi-electrode brain recordings. With brain imaging data, a reliable method is needed to assess statistical significance of the data without losing statistical power. Conjunction analysis allows the combination of significance and consistency of an effect. Through a balanced combination of information from retest experiments (multiple trials split testing), we present an intuitively appealing, novel approach for brain imaging conjunction. The method is then tested and validated on synthetic data followed by a real-world test on QEEG data from patients with Alzheimer’s disease. This latter application requires both reliable type-I error and type-II error rates, because of the poor signal-to-noise ratio inherent in EEG signals.     

On Mendelian randomization analysis of case-control study

Mendelian randomization (MR) analysis uses genotypes as instruments to estimate the causal effect of an exposure in the presence of unobserved confounders. The existing MR methods focus on the data generated from prospective cohort studies. We develop a procedure for studying binary outcomes under a case-control design. The proposed procedure is built upon two working models commonly used for MR analyses and adopts a quasi-empirical likelihood framework to address the ascertainment bias from case-control sampling. We derive various approaches for estimating the causal effect and hypothesis testing under the empirical likelihood framework. We conduct extensive simulation studies to evaluate the proposed methods. We find that the proposed empirical likelihood estimate is less biased than the existing estimates. Among all the approaches considered, the Lagrange multiplier (LM) test has the highest power, and the confidence intervals derived from the LM test have the most accurate coverage. We illustrate the use of our method in MR analysis of prostate cancer case-control data with vitamin D level as exposure and three single nucleotide polymorphisms as instruments.     

General multistage gatekeeping procedures

A general multistage (stepwise) procedure is proposed for dealing with arbitrary gatekeeping problems including parallel and serial gatekeeping. The procedure is very simple to implement since it does not require the application of the closed testing principle and the consequent need to test all nonempty intersections of hypotheses. It is based on the idea of carrying forward the Type I error rate for any rejected hypotheses to test hypotheses in the next ordered family. This requires the use of a so-called separable multiple test procedure (MTP) in the earlier family. The Bonferroni MTP is separable, but other standard MTPs such as Holm, Hochberg, Fallback and Dunnett are not. Their truncated versions are proposed which are separable and more powerful than the Bonferroni MTP. The proposed procedure is illustrated by a clinical trial example.     

Generalized Maximum Range Tests for Pairwise Comparisons of Several Populations

A multiple comparison procedure (MCP) is proposed for the comparison of all pairs of several independent samples. This MCP is essentially the closed procedure with union-intersection tests based on given single tests Q_ij for the minimal hypotheses H_ij. In such cases where the α-levels of the nominal tests associated with the MCP can be exhausted, this MCP has a uniformly higher all pair power than any refined Bonferroni test using the same Q_ij. Two different general algorithms are described in section 3. A probability inequality for ranges of i.i.d. random variables which is useful for some algorithms is proved in section 4. Section 5 contains the application to independent normally distributed estimates and section 6 the comparisons of polynomial distributions by multivariate ranges. Further applications are possible. Tables of the 0.05-bounds for the tests of section 5 and 6 are enclosed.     

On randomization testing in vegetation science: multifactor comparisons of relevé groups

Abstract. Hypothesis testing in phytocoenological applications is likely to be hindered when based on conventional statistical methods. The problem created by unrealistic assumptions can, however, be overcome by randomization. This paper discusses the general idea of randomization testing, describes a method and interprets its application in group comparisons. Two sets of variables are involved, the vegetation set on the basis of which the groups are compared and the environmental factors which delimit the groups under different analytical designs. Although simple partitioning of sum of squares is at the core of the test, the method has versatility of testing uni- or multifactor designs, which is novel in phytocoenological applications. The algorithm has been implemented in programs SYNCSA and MULTIV by V.P. Data from the Campos of southern Brazil are used for illustration.     

On the Numerical Availability of Multiple Comparison Procedures

In the past many multiple comparison procedure were difficult to perform. Usually, such procedures can be traced back to studentized multiple contrast tests. Numerical difficulties restricted the use of the exact procedures to simple, commonly balanced, designs. Conservative approximations or simulation based approaches have been used in the general cases. However, new efforts and results in the past few years have led to fast and efficient computations of the underlying multidimensional integrals. Inferences for any finite set of linear functions of normal means are now numerically feasible. These include all‐pairwise comparisons, comparisons with a control (including dose‐response contrasts), multiple comparison with the best, etc. The article applies the numerical progress on multiple comparisons procedures for common balanced and unbalanced designs within the general linear model.     

Simultaneous confidence intervals for pairwise multiple comparisons in a two-way unbalanced design

Turkey's (1953, The Problem of Multiple Comparisons, unpublished report, Princeton University) procedure is widely used for pairwise multiple comparisons in one-way ANOVA. It provides exact simultaneous pairwise confidence intervals (SPCI) for balanced designs and conservative SPCI for unbalanced designs. In this paper, we will extend Turkey's procedure to two-way unbalanced designs. Both the exact and the conservative methods will be introduced. The application of the new procedure is illustrated with sample data from two experiments.     

Nonparametric Two-Sample Tests for General Clinical Data

Starting from the discussion of a practical example a unifying concept for the derivation of meaningfully interpretable nonparametric tests for the two-sample case is developed which may well be adapted for other designs, too. This methodology covers other well-known procedures, e.g. Fisher's exact test, the Wilcoxon-Mann-Whitney and Gehan's tests, and may furthermore be extended to all situations sharing the same fundamental structural property of the sample space, namely its strict order induced by the substantial problem under study. The resulting test procedure is discussed for a randomization argument, exact and approximate, as well as for the general specific test problem. A numerical example is provided.     

Multiple comparisons: philosophies and illustrations

Statistical procedures underpin the process of scientific discovery. As researchers, one way we use these procedures is to test the validity of a null hypothesis. Often, we test the validity of more than one null hypothesis. If we fail to use an appropriate procedure to account for this multiplicity, then we are more likely to reach a wrong scientific conclusion-we are more likely to make a mistake. In physiology, experiments that involve multiple comparisons are common: of the original articles published in 1997 by the American Physiological Society, approximately 40% cite a multiple comparison procedure. In this review, I demonstrate the statistical issue embedded in multiple comparisons, and I summarize the philosophies of handling this issue. I also illustrate the three procedures-Newman-Keuls, Bonferroni, least significant difference-cited most often in my literature review; each of these procedures is of limited practical value. Last, I demonstrate the false discovery rate procedure, a promising development in multiple comparisons. The false discovery rate procedure may be the best practical solution to the problems of multiple comparisons that exist within physiology and other scientific disciplines.     

Pointwise testing with functional data using the Westfall-Young randomization method

We consider hypothesis testing with smooth functional data byperforming pointwise tests and applying a multiple comparisonsprocedure. Methods based on general inequalities, such as Bonferroni’smethod, do not perform well because of the high correlationbetween observations at nearby points. We consider the multiplecomparison procedure proposed by Westfall & Young (1993)and show that it approximates a multiple comparison correctionfor a continuum of comparisons as the grid for pointwise comparisonsbecomes finer. Simulations and an application verify that thisresult applies in practical settings.     

clonality V.0.4: a randomization-based program to test for heterozygosity-genet size relationships in clonal organisms

clonality V.0.4 is a program for testing heterozygosity-genet size relationships in clonal organisms using a randomization procedure. The software has been developed under the Borland Delphi developing environment and a Windows-executable version is freely downloadable from http://gemi.mpl.ird.fr/SiteSGASS/Prugnolle/ClonalityPage.html. The program compares the observed F(IS) of the population with the F(IS) expected if genets (multilocus genotypes present in multiple copies within the population) were chosen randomly from the set of different multilocus genotypes. The randomization procedure is performed with the same number of genets and the same number of repetitions per genet as what is observed in the original data set.     

On identification of the number of best treatments using the Newman-Keuls test

In this paper, we provide a stochastic ordering of the Studentized range statistics under a balanced one-way ANOVA model. Based on this result we show that, when restricted to the multiple comparisons with the best, the Newman-Keuls (NK) procedure strongly controls experimentwise error rate for a sequence of null hypotheses regarding the number of largest treatment means. In other words, the NK procedure provides an upper confidence bound for the number of best treatments.