Exact tests for the analysis of case-control studies of genetic markers

OBJECTIVES: The association of a candidate gene with disease can be evaluated by a case-control study in which the genotype distribution is compared for diseased cases and unaffected controls. Usually, the data are analyzed with Armitage's test using the asymptotic null distribution of the test statistic. Since this test does not generally guarantee a type I error rate less than or equal to the significance level alpha, tests based on exact null distributions have been investigated. METHODS: An algorithm to generate the exact null distribution for both Armitage's test statistic and a recently proposed modification of the Baumgartner-Weiss-Schindler statistic is presented. I have compared the tests in a simulation study. RESULTS: The asymptotic Armitage test is slightly anticonservative whereas the exact tests control the type I error rate. The exact Armitage test is very conservative, but the exact test based on the modification of the Baumgartner-Weiss-Schindler statistic has a type I error rate close to alpha. The exact Armitage test is the least powerful test; the difference in power between the other two tests is often small and the comparison does not show a clear winner. CONCLUSION: Simulation results indicate that an exact test based on the modification of the Baumgartner-Weiss-Schindler statistic is preferable for the analysis of case-control studies of genetic markers.     

A new location-scale test based on a combination of the ideas of Levene and Lepage

Lepage's test combines the Wilcoxon rank-sum and the Ansari-Bradley statistics. We propose to replace the latter statistic by a Wilcoxon rank-sum calculated after Levene's transformation. We use the medians for this transformation, i.e. absolute deviations from sample medians are calculated. The new location-scale test can be carried out as a permutation test based on permutations of the original observations, the Levene transformation has to be applied for each permutation in an intermediate step to calculate the test statistic. Simulations indicate that the new test can be more powerful than an O'Brien-type test and Lepage's test, the latter is the standard nonparametric location-scale test. The new test is illustrated using real data about colony sizes of yellow-eyed penguins and an SAS program to perform the test is freely available.     

Using weighted permutation scores to detect differential gene expression with microarray data

A class of nonparametric statistical methods, including a nonparametric empirical Bayes (EB) method, the Significance Analysis of Microarrays (SAM) and the mixture model method (MMM) have been proposed to detect differential gene expression for replicated microarray experiments. They all depend on constructing a test statistic, for example, a t-statistic, and then using permutation to draw inferences. However, due to special features of microarray data, using standard permutation scores may not estimate the null distribution of the test statistic well, leading to possibly too conservative inferences. We propose a new method of constructing weighted permutation scores to overcome the problem: posterior probabilities of having no differential expression from the EB method are used as weights for genes to better estimate the null distribution of the test statistic. We also propose a weighted method to estimate the false discovery rate (FDR) using the posterior probabilities. Using simulated data and real data for time-course microarray experiments, we show the improved performance of the proposed methods when implemented in MMM, EB and SAM.     

A bootstrap test for the analysis of microarray experiments with a very small number of replications

In microarray studies it is common that the number of replications (i.e. the sample size) is small and that the distribution of expression values differs from normality. In this situation, permutation and bootstrap tests may be appropriate for the identification of differentially expressed genes. However, unlike bootstrap tests, permutation tests are not suitable for very small sample sizes, such as three per group. A variety of different bootstrap tests exists. For example, it is possible to adjust the data to have a common mean before the bootstrap samples are drawn. For small significance levels, which can occur when a large number of genes is investigated, the original bootstrap test, as well as a bootstrap test suggested for the Behrens-Fisher problem, have no power in cases of very small sample sizes. In contrast, the modified test based on adjusted data is powerful. Using a Monte Carlo simulation study, we demonstrate that the difference in power can be huge. In addition, the different tests are illustrated using microarray data.     

A class of permutation tests for stratified survival data

We propose a class of permutation tests for stratified survival data. The tests are derived using the framework of Fay and Shih (1998, Journal of the American Statistical Association 93, 387-396), which creates tests by permuting scores based on a functional of estimated distribution functions. Here the estimated distribution function for each possibly right-, left-, or interval-censored observation is based on a shrinkage estimator similar to the nonparametric empirical estimator of Ghosh, Lahiri, and Tiwari (1989, Communications in Statistics--Theory and Methods 18, 121-146), and permutation is carried out within strata. The proposed test with a weighted Mann-Whitney functional is similar to the permutation form of the stratified log-rank test when there is a large strata effect or the sample size in each stratum is large and is similar to the permutation form of the ordinary log-rank test when there is little strata effect. Thus, the proposed test unifies the advantages of both the stratified and ordinary log-rank tests. By changing the functional, we may obtain a stratified Prentice-Wilcoxon test or a difference in means test with similar unifying properties. We show through simulations the advantage of the proposed test over existing tests for uncensored and right-censored data.     

Exact log-rank tests for unequal follow-up

The asymptotic log-rank and generalized Wilcoxon tests are the standard procedures for comparing samples of possibly censored survival times. For comparison of samples of very different sizes, an exact test is available that is based on a complete permutation of log-rank or Wilcoxon scores. While the asymptotic tests do not keep their nominal sizes if sample sizes differ substantially, the exact complete permutation test requires equal follow-up of the samples. Therefore, we have developed and present two new exact tests also suitable for unequal follow-up. The first of these is an exact analogue of the asymptotic log-rank test and conditions on observed risk sets, whereas the second approach permutes survival times while conditioning on the realized follow-up in each group. In an empirical study, we compare the new procedures with the asymptotic log-rank test, the exact complete permutation test, and an earlier proposed approach that equalizes the follow-up distributions using artificial censoring. Results confirm highly satisfactory performance of the exact procedure conditioning on realized follow-up, particularly in case of unequal follow-up. The advantage of this test over other options of analysis is finally exemplified in the analysis of a breast cancer study.     

Nonparametric Tests for Trend: Jonckheere's Test,a Modification and a Maximum Test

Jonckheere's test is a frequently used nonparametric trend test for the evaluation of preclinical studies and clinical dose-finding trials. In this paper, a modification of Jonckheere's test is proposed. If the exact permutation distribution is used for inference, the modified test can fill out the level of the type I error in a much more complete way and is substantially more powerful than the common Jonckheere test. If the asymptotic normality is used for inference, the modified test is slightly more powerful. In addition, a maximum test is investigated which is more robust concerning an a priori unknown dose-response shape. The robustness is advantageous, especially in a closed testing procedure. The different tests are applied to two example data sets.     

On the use of permutation in and the performance of a class of nonparametric methods to detect differential gene expression

MOTIVATION: Recently a class of nonparametric statistical methods, including the empirical Bayes (EB) method, the significance analysis of microarray (SAM) method and the mixture model method (MMM), have been proposed to detect differential gene expression for replicated microarray experiments conducted under two conditions. All the methods depend on constructing a test statistic Z and a so-called null statistic z. The null statistic z is used to provide some reference distribution for Z such that statistical inference can be accomplished. A common way of constructing z is to apply Z to randomly permuted data. Here we point our that the distribution of z may not approximate the null distribution of Z well, leading to possibly too conservative inference. This observation may apply to other permutation-based nonparametric methods. We propose a new method of constructing a null statistic that aims to estimate the null distribution of a test statistic directly. RESULTS: Using simulated data and real data, we assess and compare the performance of the existing method and our new method when applied in EB, SAM and MMM. Some interesting findings on operating characteristics of EB, SAM and MMM are also reported. Finally, by combining the idea of SAM and MMM, we outline a simple nonparametric method based on the direct use of a test statistic and a null statistic.     

Testing for differentially expressed genes with microarray data

This paper compares the type I error and power of the one- and two-sample t-tests, and the one- and two-sample permutation tests for detecting differences in gene expression between two microarray samples with replicates using Monte Carlo simulations. When data are generated from a normal distribution, type I errors and powers of the one-sample parametric t-test and one-sample permutation test are very close, as are the two-sample t-test and two-sample permutation test, provided that the number of replicates is adequate. When data are generated from a t-distribution, the permutation tests outperform the corresponding parametric tests if the number of replicates is at least five. For data from a two-color dye swap experiment, the one-sample test appears to perform better than the two-sample test since expression measurements for control and treatment samples from the same spot are correlated. For data from independent samples, such as the one-channel array or two-channel array experiment using reference design, the two-sample t-tests appear more powerful than the one-sample t-tests.     

Permutation‐based inference for the AUC: A unified approach for continuous and discontinuous data

We investigate rank‐based studentized permutation methods for the nonparametric Behrens–Fisher problem, that is, inference methods for the area under the ROC curve. We hereby prove that the studentized permutation distribution of the Brunner‐Munzel rank statistic is asymptotically standard normal, even under the alternative. Thus, incidentally providing the hitherto missing theoretical foundation for the Neubert and Brunner studentized permutation test. In particular, we do not only show its consistency, but also that confidence intervals for the underlying treatment effects can be computed by inverting this permutation test. In addition, we derive permutation‐based range‐preserving confidence intervals. Extensive simulation studies show that the permutation‐based confidence intervals appear to maintain the preassigned coverage probability quite accurately (even for rather small sample sizes). For a convenient application of the proposed methods, a freely available software package for the statistical software R has been developed. A real data example illustrates the application.     

An omnibus consistent adaptive percentile modified Wilcoxon rank sum test with applications in gene expression studies

Summary We present an adaptive percentile modified Wilcoxon rank sum test for the two‐sample problem. The test is basically a Wilcoxon rank sum test applied on a fraction of the sample observations, and the fraction is adaptively determined by the sample observations. Most of the theory is developed under a location‐shift model, but we demonstrate that the test is also meaningful for testing against more general alternatives. The test may be particularly useful for the analysis of massive datasets in which quasi‐automatic hypothesis testing is required. We investigate the power characteristics of the new test in a simulation study, and we apply the test to a microarray experiment on colorectal cancer. These empirical studies demonstrate that the new test has good overall power and that it succeeds better in finding differentially expressed genes as compared to other popular tests. We conclude that the new nonparametric test is widely applicable and that its power is comparable to the power of the Baumgartner‐Weiß‐Schindler test.     

Estimating the power of the two-sample Wilcoxon test for location shift

Traditional methods for calculating the power of a statistical test for location shift require knowledge of the shape of the underlying probability distribution. The distribution shape, however, may be unknown. This paper describes a bootstrap method for using observed data (or pilot data) to approximate the power. No assumptions need be made about the shape of the underlying continuous probability distribution. Simulation evidence shows that, when applied to the Wilcoxon two-sample test for location shift, the suggested method is reliable. The evidence also shows that it is more accurate than a benchmark traditional approach. The bootstrap method is applied to a real-data example. The analysis demonstrates how the method can be used to determine sample sizes and how to choose the more powerful of two alternative tests for location shift.     

A comparison of parametric versus permutation methods with applications to general and temporal microarray gene expression data

MOTIVATION: In analyses of microarray data with a design of different biological conditions, ranking genes by their differential 'importance' is often desired so that biologists can focus research on a small subset of genes that are most likely related to the experiment conditions. Permutation methods are often recommended and used, in place of their parametric counterparts, due to the small sample sizes of microarray experiments and possible non-normality of the data. The recommendations, however, are based on classical knowledge in the hypothesis test setting. RESULTS: We explore the relationship between hypothesis testing and gene ranking. We indicate that the permutation method does not provide a metric for the distance between two underlying distributions. In our simulation studies permutation methods tend to be equally or less accurate than parametric methods in ranking genes. This is partially due to the discreteness of the permutation distributions, as well as the non-metric property. In data analysis the variability in ranking genes can be assessed by bootstrap. It turns out that the variability is much lower for permutation than parametric methods, which agrees with the known robustness of permutation methods to individual outliers in the data.     

Sample size calculation for multiple testing in microarray data analysis

Microarray technology is rapidly emerging for genome-wide screening of differentially expressed genes between clinical subtypes or different conditions of human diseases. Traditional statistical testing approaches, such as the two-sample t-test or Wilcoxon test, are frequently used for evaluating statistical significance of informative expressions but require adjustment for large-scale multiplicity. Due to its simplicity, Bonferroni adjustment has been widely used to circumvent this problem. It is well known, however, that the standard Bonferroni test is often very conservative. In the present paper, we compare three multiple testing procedures in the microarray context: the original Bonferroni method, a Bonferroni-type improved single-step method and a step-down method. The latter two methods are based on nonparametric resampling, by which the null distribution can be derived with the dependency structure among gene expressions preserved and the family-wise error rate accurately controlled at the desired level. We also present a sample size calculation method for designing microarray studies. Through simulations and data analyses, we find that the proposed methods for testing and sample size calculation are computationally fast and control error and power precisely.     

A Nonparametric Likelihood Ratio Test to Identify Differentially Expressed Genes from Microarray Data

Microarray experiments contribute significantly to the progress in disease treatment by enabling a precise and early diagnosis. One of the major objectives of microarray experiments is to identify differentially expressed genes under various conditions. The statistical methods currently used to analyse microarray data are inadequate, mainly due to the lack of understanding of the distribution of microarray data. We present a nonparametric likelihood ratio (NPLR) test to identify differentially expressed genes using microarray data. The NPLR test is highly robust against extreme values and does not assume the distribution of the parent population. Simulation studies show that the NPLR test is more powerful than some of the commonly used methods, such as the two-sample t-test, the Mann-Whitney U-test and significance analysis of microarrays (SAM). When applied to microarray data, we found that the NPLR test identifies more differentially expressed genes than its competitors. The asymptotic distribution of the NPLR test statistic and the p-value function is presented. The application of the NPLR method is shown, using both synthetic and real-life data. The biological significance of some of the genes detected only by the NPLR method is discussed.     

Nonparametric methods for identifying differentially expressed genes in microarray data

MOTIVATION: Gene expression experiments provide a fast and systematic way to identify disease markers relevant to clinical care. In this study, we address the problem of robust identification of differentially expressed genes from microarray data. Differentially expressed genes, or discriminator genes, are genes with significantly different expression in two user-defined groups of microarray experiments. We compare three model-free approaches: (1). nonparametric t-test, (2). Wilcoxon (or Mann-Whitney) rank sum test, and (3). a heuristic method based on high Pearson correlation to a perfectly differentiating gene ('ideal discriminator method'). We systematically assess the performance of each method based on simulated and biological data under varying noise levels and p-value cutoffs. RESULTS: All methods exhibit very low false positive rates and identify a large fraction of the differentially expressed genes in simulated data sets with noise level similar to that of actual data. Overall, the rank sum test appears most conservative, which may be advantageous when the computationally identified genes need to be tested biologically. However, if a more inclusive list of markers is desired, a higher p-value cutoff or the nonparametric t-test may be appropriate. When applied to data from lung tumor and lymphoma data sets, the methods identify biologically relevant differentially expressed genes that allow clear separation of groups in question. Thus the methods described and evaluated here provide a convenient and robust way to identify differentially expressed genes for further biological and clinical analysis.     

EDISON-WMW: Exact Dynamic Programing Solution of the Wilcoxon–Mann–Whitney Test

In many research disciplines, hypothesis tests are applied to evaluate whether findings are statistically significant or could be explained by chance. The Wilcoxon–Mann–Whitney(WMW) test is among the most popular hypothesis tests in medicine and life science to analyze if two groups of samples are equally distributed. This nonparametric statistical homogeneity test is commonly applied in molecular diagnosis. Generally, the solution of the WMW test takes a high combinatorial effort for large sample cohorts containing a significant number of ties. Hence, P value is frequently approximated by a normal distribution. We developed EDISON-WMW, a new approach to calculate the exact permutation of the two-tailed unpaired WMW test without any corrections required and allowing for ties. The method relies on dynamic programing to solve the combinatorial problem of the WMW test efficiently. Beyond a straightforward implementation of the algorithm, we presented different optimization strategies and developed a parallel solution. Using our program,the exact P value for large cohorts containing more than 1000 samples with ties can be calculated within minutes. We demonstrate the performance of this novel approach on randomly-generated data, benchmark it against 13 other commonly-applied approaches and moreover evaluate molecular biomarkers for lung carcinoma and chronic obstructive pulmonary disease(COPD). We foundthat approximated P values were generally higher than the exact solution provided by EDISONWMW. Importantly, the algorithm can also be applied to high-throughput omics datasets, where hundreds or thousands of features are included. To provide easy access to the multi-threaded version of EDISON-WMW, a web-based solution of our algorithm is freely available at http://www.ccb.uni-saarland.de/software/wtest/.     

Simultaneous Statistical Inference for Epigenetic Data

Epigenetic research leads to complex data structures. Since parametric model assumptions for the distribution of epigenetic data are hard to verify we introduce in the present work a nonparametric statistical framework for two-group comparisons. Furthermore, epigenetic analyses are often performed at various genetic loci simultaneously. Hence, in order to be able to draw valid conclusions for specific loci, an appropriate multiple testing correction is necessary. Finally, with technologies available for the simultaneous assessment of many interrelated biological parameters (such as gene arrays), statistical approaches also need to deal with a possibly unknown dependency structure in the data. Our statistical approach to the nonparametric comparison of two samples with independent multivariate observables is based on recently developed multivariate multiple permutation tests. We adapt their theory in order to cope with families of hypotheses regarding relative effects. Our results indicate that the multivariate multiple permutation test keeps the pre-assigned type I error level for the global null hypothesis. In combination with the closure principle, the family-wise error rate for the simultaneous test of the corresponding locus/parameter-specific null hypotheses can be controlled. In applications we demonstrate that group differences in epigenetic data can be detected reliably with our methodology.     

Construction of a confidence set of markers for the location of a disease gene using affected sib pair data

We have previously proposed a confidence set approach for finding tightly linked genomic regions under the setting of parametric linkage analysis. In this article, we extend the confidence set approach to nonparametric linkage analysis of affected sib pair (ASP) data based on their identity-by-descent (IBD) information. Two well-known statistics in nonparametric linkage analysis, the Two-IBD test (proportion of ASPs sharing two alleles IBD), and the Mean test (average number of alleles shared IBD in the ASPs), are used for constructing confidence sets. Some numerical analyses as well as a simulation study were carried out to demonstrate the utility of the methods. Our results show that the fundamental advantages of the confidence set approach in parametric linkage analysis are retained when the method is generalized to nonparametric analysis. Our study on the accuracy of confidence sets, in terms of choice of tests, underlying disease incidence data, and amount of data available, leads us to conclude, among other things, that the Mean test outperforms the Two-IBD test in most situations, with the reverse being true only for traits with small additive variance. Although we describe how to construct confidence sets based on only two familiar tests, one can construct confidence sets similarly using other allele sharing statistics.     

A New Powerful Nonparametric Rank Test for Ordered Alternative Problem

We propose a new nonparametric test for ordered alternative problem based on the rank difference between two observations from different groups. These groups are assumed to be independent from each other. The exact mean and variance of the test statistic under the null distribution are derived, and its asymptotic distribution is proven to be normal. Furthermore, an extensive power comparison between the new test and other commonly used tests shows that the new test is generally more powerful than others under various conditions, including the same type of distribution, and mixed distributions. A real example from an anti-hypertensive drug trial is provided to illustrate the application of the tests. The new test is therefore recommended for use in practice due to easy calculation and substantial power gain.