SDMinP: a program to control the family wise error rate using step-down minP adjusted P-values

SDMinP is an easy-to-use program for fast calculation of empirical and adjusted P-values for correlated and uncorrelated hypotheses in multiple testing experiments. It is based on the Free Step-Down Resampling Method for controlling the family wise error rate, and implements a variation of an efficient algorithm, which reduces the originally required re-sampling effort considerably and makes the method computationally feasible. The program is independent of the underlying test statistic and works with provided observed and permutation test statistics.     

Approximating the distribution of maximally selected McNemar's statistics

It is common in epidemiologic analyses to summarize continuous outcomes as falling above or below a threshold. With paired data and with a threshold chosen without reference to the outcomes, McNemar's test of marginal homogeneity may be applied to the resulting dichotomous pairs when testing for equality of the marginal distributions of the underlying continuous outcomes. If the threshold is chosen to maximize the test statistic, however, referring the resulting test statistic to the nominal chi 2 distribution is incorrect; instead, the p-value must be adjusted for the multiple comparisons. Here the distribution of a maximally selected McNemar's statistic is derived, and it is shown that an approximation due to Durbin (1985, Journal of Applied Probability 22, 99-122) may be used to estimate approximate p-values. The methodology is illustrated by an application to measurements of insulin-like growth factor-I (IGF-I) in matched prostate cancer cases and controls from the Physicians' Health Study. The results of simulation experiments that assess the accuracy of the approximation in moderate sample sizes are reported.     

To permute or not to permute

Permutation test is a popular technique for testing a hypothesis of no effect, when the distribution of the test statistic is unknown. To test the equality of two means, a permutation test might use a test statistic which is the difference of the two sample means in the univariate case. In the multivariate case, it might use a test statistic which is the maximum of the univariate test statistics. A permutation test then estimates the null distribution of the test statistic by permuting the observations between the two samples. We will show that, for such tests, if the two distributions are not identical (as for example when they have unequal variances, correlations or skewness), then a permutation test for equality of means based on difference of sample means can have an inflated Type I error rate even when the means are equal. Our results illustrate permutation testing should be confined to testing for non-identical distributions. CONTACT: calian@raunvis.hi.is.     

Testing marginal homogeneity in square tables; with emphasis on matched data

A noniterative procedure based upon the minimum modified X2 approach is employed to test the model of homogeneity of one-dimensional margins in square tables. Such tables may arise from matched pairs with k outcomes. The special case of double dichotomy (i.e. matched pairs with two outcomes) reduces to the McNemar test statistic. The case of multiple matched controls is also dealt with. The Cochran's Q test is used to test the marginal homogeneity in cases comparing m distinct matched samples in addition to testing trends in proportions. Reference is made to the equivalence between these tests and the approach of hierarchical log-linear models for testing marginal homogeneity of square tables.     

Rank Analysis of Block Designs Having Different Cell Frequencies

A distribution–free test is considered for testing the treatment effects in block designs with different cell frequencies. A test statistic which is a function of treatment ranks has been proposed which is distributed as chi-square for large samples. The null distribution of the test statistic has been obtained. The entire procedure has been explained by a numerical example.     

Filtering,FDR and power

Background  

In high-dimensional data analysis such as differential gene expression analysis, people often use filtering methods like fold-change or variance filters in an attempt to reduce the multiple testing penalty and improve power. However, filtering may introduce a bias on the multiple testing correction. The precise amount of bias depends on many quantities, such as fraction of probes filtered out, filter statistic and test statistic used.     

Comparative application of the coefficient of variation and range-based statistics for assessing the taxonomic composition of fossil samples

Dental variation has been used commonly to assess taxonomic composition in morphologically homogeneous fossil samples. While the coefficient of variation (CV) has been used traditionally, range-based measures of variation, such as the range as a percentage of the mean (R%) and the maximum/minimum index (I_max/min) have recently become popular alternatives. The current study compares the performance of these statistics when applied to single- and pooled-species dental samples of extant Cercopithecus species. A common methodology for such problems of species discrimination has been to simply compare the maximum value of a variation statistic observed in extant samples with that observed in the fossil sample. However, regardless of what statistic is used, this approach has an unknowable Type I error rate, and usually has low power to detect multiple species. A more appropriate method involves a formal hypothesis test. The null hypothesis is that the level of variation in the fossil sample does not exceed what might be expected in a sample drawn randomly from a reference population, taking into account sampling error and the size of the fossil sample. Previous research using this method with the CV has indicated that it offers considerable power at an acceptable Type I error rate. In the current study, the data of primary interest were posterior dental dimensions for single- and pooled species samples from extant Cercopithecus species. In addition, the study also investigated the relative performance of variation statistics when applied to highly dimorphic canine dimensions, since much recent work has employed sexually dimorphic dental dimensions for assessing single-species hypotheses. The results indicate that the CV consistently out-performed the range-based statistics when using posterior dental dimensions to test a single-species hypothesis. Regardless of which statistic was used, tests on sexually dimorphic dimensions offered minimal power. In consideration of these results and the problem of studywise Type I error rates, we recommend against the use of multiple measures of variation to test for multiple species composition, and advocate the CV as the statistic of choice when using the method of Cope & Lacy (1992). For similar reasons, we argue for careful selection of dental variables for inclusion in such analyses, and in particular recommend against including sexually dimorphic dimensions when testing for multiple species composition.     

A multiple testing procedure for clinical trials.

A multiple testing procedure is proposed for comparing two treatments when response to treatment is both dichotomous (i.e., success or failure) and immediate. The proposed test statistic for each test is the usual (Pearson) chi-square statistic based on all data collected to that point. The maximum number (N) of tests and the number (m1 + m2) of observations collected between successive tests is fixed in advance. The overall size of the procedure is shown to be controlled with virtually the same accuracy as the single sample chi-square test based on N(m1 + m2) observations. The power is also found to be virtually the same. However, by affording the opportunity to terminate early when one treatment performs markedly better than the other, the multiple testing procedure may eliminate the ethical dilemmas that often accompany clinical trials.     

Statistical Testing for Seasonality in Data with Multiple Peaks and Troughs

EDWARDS'S test for seasonality is extended to multiple peaks and troughs. It is shown how the extended statistic may be adjusted for the population at risk and for unequal lengths of time intervals in the cycle of seasons. A simulation study shows that the extended test statistic is, for sample sizes N ≧ 100, very specific in detecting the number of peaks and troughs for which it is intended. The associated method of parameter estimation is also assessed; for N ≦ 100, the amplitude of a possibly adequate simple harmonic model is estimated well, but the initial value of the phase angle is not. Estimated percentage points for the extended test statistic are tabulated, and some recommendations are offered regarding usage of this method.     

A class of multiplicity adjusted tests for spatial clustering based on case-control point data

A class of tests with quadratic forms for detecting spatial clustering of health events based on case-control point data is proposed. It includes Cuzick and Edwards's test statistic (1990, Journal of the Royal Statistical Society, Series B 52, 73-104). Although they used the property of asymptotic normality of the test statistic, we show that such an approximation is generally poor for moderately large sample sizes. Instead, we suggest a central chi-square distribution as a better approximation to the asymptotic distribution of the test statistic. Furthermore, not only to estimate the optimal value of the unknown parameter on the scale of cluster but also to adjust for multiple testing due to repeating the procedure by changing the parameter value, we propose the minimum of the profile p-value of the test statistic for the parameter as an integrated test statistic. We also provide a statistic to estimate the areas or cases which make large contributions to significant clustering. The proposed methods are illustrated with a data set concerning the locations of cases of childhood leukemia and lymphoma and another on early medieval grave site locations consisting of affected and nonaffected grave sites.     

Methods for the statistical analysis of binary data in split-cluster designs

Split-cluster designs are frequently used in the health sciences when naturally occurring clusters such as multiple sites or organs in the same subject are assigned to different treatments. However, statistical methods for the analysis of binary data arising from such designs are not well developed. The purpose of this article is to propose and evaluate a new procedure for testing the equality of event rates in a design dividing each of k clusters into two segments having multiple sites (e.g., teeth, lesions). The test statistic proposed is a generalization of a previously published procedure based on adjusting the standard Pearson chi-square statistic, but can also be derived as a score test using the approach of generalized estimating equations.     

Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.     

Sample Size Determination for Designing a Strata-Matched Case-Control Study to Detect Multiple Risk Factors

Investigations of sample size for planning case-control studies have usually been limited to detecting a single factor. In this paper, we investigate sample size for multiple risk factors in strata-matched case-control studies. We construct an omnibus statistic for testing M different risk factors based on the jointly sufficient statistics of parameters associated with the risk factors. The statistic is non-iterative, and it reduces to the Cochran statistic when M = 1. The asymptotic power function of the test is a non-central chi-square with M degrees of freedom and the sample size required for a specific power can be obtained by the inverse relationship. We find that the equal sample allocation is optimum. A Monte Carlo experiment demonstrates that an approximate formula for calculating sample size is satisfactory in typical epidemiologic studies. An approximate sample size obtained using Bonferroni's method for multiple comparisons is much larger than that obtained using the omnibus test. Approximate sample size formulas investigated in this paper using the omnibus test, as well as the individual tests, can be useful in designing case-control studies for detecting multiple risk factors.     

Impact of missing genotype data on Monte-Carlo simulation based haplotype analysis

In the context of haplotype association analysis of unphased genotype data, methods based on Monte-Carlo simulations are often used to compensate for missing or inappropriate asymptotic theory. Moreover, such methods are an indispensable means to deal with multiple testing problems. We want to call attention to a potential trap in this usually useful approach: The simulation approach may lead to strongly inflated type I errors in the presence of different missing rates between cases and controls, depending on the chosen test statistic. Here, we consider four different testing strategies for haplotype analysis of case-control data. We recommend to interpret results for data sets with non-comparable distributions of missing genotypes with special caution, in case the test statistic is based on inferred haplotypes per individual. Moreover, our results are important for the conduction and interpretation of genome-wide association studies.     

A new expectation-maximization statistical test for case-control association studies considering rare variants obtained by high-throughput sequencing

Genome-wide association studies (GWAS) have been successful in identifying common genetic variation reproducibly associated with disease. However, most associated variants confer very small risk and after meta-analysis of large cohorts a large fraction of expected heritability still remains unexplained. A possible explanation is that rare variants currently undetected by GWAS with SNP arrays could contribute a large fraction of risk when present in cases. This concept has spurred great interest in exploring the role of rare variants in disease. As the cost of sequencing continue to plummet, it is becoming feasible to directly sequence case-control samples for testing disease association including rare variants. We have developed a test statistic that allows for association testing among cases and controls using data directly from sequencing reads. In addition, our method allows for random errors in reads. We determine the probability of a true genotype call based on the observed base pair reads using the expectation-maximization algorithm. We apply the SumStat procedure to obtain a single statistic for a group of multiple rare variant loci. We document the validity of our method through simulations. Our results suggest that our statistic maintains the correct type I error rate, even in the presence of differential misclassification for sequence reads, and that it has good power under a number of scenarios. Finally, our SumStat results show power at least as good as the maximum single locus results.     

Association Testing Strategy for Data from Dense Marker Panels

Genome wide association studies have been usually analyzed in a univariate manner. The commonly used univariate tests have one degree of freedom and assume an additive mode of inheritance. The experiment-wise significance of these univariate statistics is obtained by adjusting for multiple testing. Next generation sequencing studies, which assay 10-20 million variants, are beginning to come online. For these studies, the strategy of additive univariate testing and multiple testing adjustment is likely to result in a loss of power due to (1) the substantial multiple testing burden and (2) the possibility of a non-additive causal mode of inheritance. To reduce the power loss we propose: a new method (1) to summarize in a single statistic the strength of the association signals coming from all not-very-rare variants in a linkage disequilibrium block and (2) to incorporate, in any linkage disequilibrium block statistic, the strength of the association signals under multiple modes of inheritance. The proposed linkage disequilibrium block test consists of the sum of squares of nominally significant univariate statistics. We compare the performance of this method to the performance of existing linkage disequilibrium block/gene-based methods. Simulations show that (1) extending methods to combine testing for multiple modes of inheritance leads to substantial power gains, especially for a recessive mode of inheritance, and (2) the proposed method has a good overall performance. Based on simulation results, we provide practical advice on choosing suitable methods for applied analyses.     

Testing Jumps via False Discovery Rate Control

Many recently developed nonparametric jump tests can be viewed as multiple hypothesis testing problems. For such multiple hypothesis tests, it is well known that controlling type I error often makes a large proportion of erroneous rejections, and such situation becomes even worse when the jump occurrence is a rare event. To obtain more reliable results, we aim to control the false discovery rate (FDR), an efficient compound error measure for erroneous rejections in multiple testing problems. We perform the test via the Barndorff-Nielsen and Shephard (BNS) test statistic, and control the FDR with the Benjamini and Hochberg (BH) procedure. We provide asymptotic results for the FDR control. From simulations, we examine relevant theoretical results and demonstrate the advantages of controlling the FDR. The hybrid approach is then applied to empirical analysis on two benchmark stock indices with high frequency data.     

The Null Distributions of Test Statistics in Genomewide Association Studies

Genomewide association (GWA) studies assay hundreds of thousands of single nucleotide polymorphisms (SNPs) simultaneously across the entire genome and associate them with diseases, other biological or clinical traits. The association analysis usually tests each SNP as an independent entity and ignores the biological information such as linkage disequilibrium. Although the Bonferroni correction and other approaches have been proposed to address the issue of multiple comparisons as a result of testing many SNPs, there is a lack of understanding of the distribution of an association test statistic when an entire genome is considered together. In other words, there are extensive efforts in hypothesis testing, and almost no attempt in estimating the density under the null hypothesis. By estimating the true null distribution, we can apply the result directly to hypothesis testing; better assess the existing approaches of multiple comparisons; and evaluate the impact of linkage disequilibrium on the GWA studies. To this end, we estimate the empirical null distribution of an association test statistic in GWA studies using simulated population data. We further propose a convenient and accurate method based on adaptive spline to estimate the empirical value in GWA studies and validate our findings using a real data set. Our method enables us to fully characterize the null distribution of an association test that not only can be used to test the null hypothesis of no association, but also provides important information about the impact of density of the genetic markers on the significance of the tests. Our method does not require users to perform computationally intensive permutations, and hence provides a timely solution to an important and difficult problem in GWA studies.     

Resampling-based methods in single and multiple testing for equality of covariance/correlation matrices

Traditional resampling-based tests for homogeneity in covariance matrices across multiple groups resample residuals, that is, data centered by group means. These residuals do not share the same second moments when the null hypothesis is false, which makes them difficult to use in the setting of multiple testing. An alternative approach is to resample standardized residuals, data centered by group sample means and standardized by group sample covariance matrices. This approach, however, has been observed to inflate type I error when sample size is small or data are generated from heavy-tailed distributions. We propose to improve this approach by using robust estimation for the first and second moments. We discuss two statistics: the Bartlett statistic and a statistic based on eigen-decomposition of sample covariance matrices. Both statistics can be expressed in terms of standardized errors under the null hypothesis. These methods are extended to test homogeneity in correlation matrices. Using simulation studies, we demonstrate that the robust resampling approach provides comparable or superior performance, relative to traditional approaches, for single testing and reasonable performance for multiple testing. The proposed methods are applied to data collected in an HIV vaccine trial to investigate possible determinants, including vaccine status, vaccine-induced immune response level and viral genotype, of unusual correlation pattern between HIV viral load and CD4 count in newly infected patients.     

A group sequential type design for three‐arm non‐inferiority trials with binary endpoints

The three‐arm design with a test treatment, an active control and a placebo group is the gold standard design for non‐inferiority trials if it is ethically justifiable to expose patients to placebo. In this paper, we first use the closed testing principle to establish the hierarchical testing procedure for the multiple comparisons involved in the three‐arm design. For the effect preservation test we derive the explicit formula for the optimal allocation ratios. We propose a group sequential type design, which naturally accommodates the hierarchical testing procedure. Under this proposed design, Monte Carlo simulations are conducted to evaluate the performance of the sequential effect preservation test when the variance of the test statistic is estimated based on the restricted maximum likelihood estimators of the response rates under the null hypothesis. When there are uncertainties for the placebo response rate, the proposed design demonstrates better operating characteristics than the fixed sample design.