An efficient Monte Carlo approach to assessing statistical significance in genomic studies

MOTIVATION: Multiple hypothesis testing is a common problem in genome research, particularly in microarray experiments and genomewide association studies. Failure to account for the effects of multiple comparisons would result in an abundance of false positive results. The Bonferroni correction and Holm's step-down procedure are overly conservative, whereas the permutation test is time-consuming and is restricted to simple problems. RESULTS: We developed an efficient Monte Carlo approach to approximating the joint distribution of the test statistics along the genome. We then used the Monte Carlo distribution to evaluate the commonly used criteria for error control, such as familywise error rates and positive false discovery rates. This approach is applicable to any data structures and test statistics. Applications to simulated and real data demonstrate that the proposed approach provides accurate error control, and can be substantially more powerful than the Bonferroni and Holm methods, especially when the test statistics are highly correlated.     

A two-sample test sensitive to crossing hazards in uncensored and singly censored data

Savage score statistics are employed to develop a test for comparing survival distributions with right-hand singly censored data. The procedure is motivated by the interest in developing a powerful method for determining differences when true survival distributions cross. Examination of small-sample characteristics under the null hypothesis indicate that asymptotic critical values yield a slightly conservative test. Power of the test compares favorably with other criteria, including the modified Smirnov procedure, particularly if there is a single crossing of the survival curves.     

A new statistical test for linkage heterogeneity.

A new, statistical test for linkage heterogeneity is described. It is a likelihood-ratio test based on a beta distribution for the prior distribution of the recombination fraction among families (or individuals). The null distribution for this statistic (called the B-test) is derived under a broad range of circumstances. Two other heterogeneity test statistics--the admixture test or A-test first described by Smith and Morton's test (here referred to as the K-test)--are also examined. The probability distribution for the K-test statistic is very sensitive to family size, whereas the other two statistics are not. All three statistics are somewhat sensitive to the magnitude of the recombination fraction theta. Critical values for each of the test statistics are given. A conservative approximation for both the A-test and B-test is given by a chi 2 distribution when P/2 instead of P is used for the observed significance level. In terms of power, the B-test performs best among the three tests over a broad range of alternate heterogeneity hypotheses--except for the specific case of admixture with loose linkage, in which the A-test performs best. Overall, the difference in power among the three tests is not large. An application to some recently published data on the fragile-X syndrome and X-chromosome markers is given.     

Power and sample size calculation of comparative diagnostic accuracy studies with multiple correlated test results

We consider the power and sample size calculation of diagnostic studies with normally distributed multiple correlated test results. We derive test statistics and obtain power and sample size formulas. The methods are illustrated using an example of comparison of CT and PET scanner for detecting extra-hepatic disease for colorectal cancer.     

Significance levels for studies with correlated test statistics

When testing large numbers of null hypotheses, one needs to assess the evidence against the global null hypothesis that none of the hypotheses is false. Such evidence typically is based on the test statistic of the largest magnitude, whose statistical significance is evaluated by permuting the sample units to simulate its null distribution. Efron (2007) has noted that correlation among the test statistics can induce substantial interstudy variation in the shapes of their histograms, which may cause misleading tail counts. Here, we show that permutation-based estimates of the overall significance level also can be misleading when the test statistics are correlated. We propose that such estimates be conditioned on a simple measure of the spread of the observed histogram, and we provide a method for obtaining conditional significance levels. We justify this conditioning using the conditionality principle described by Cox and Hinkley (1974). Application of the method to gene expression data illustrates the circumstances when conditional significance levels are needed.     

Methods to evaluate populations for alleles to improve an elite hybrid

Elite hybrids can be improved by the introgression of favorable alleles not already present in the hybrid. Our first objective was to evaluate several estimators derived from quantitative genetic theory that attempt to quantify the relative number of useful alleles in potential donor populations. Secondly, we wanted to evaluate two proposed ways of determining relatedness of donor populations to the parents of the elite hybrid. Two experiments, each consisting of 21 maize populations of known pedigree, were grown at three and four environments in Minnesota in 1991. Yield and plant height means were used to provide estimates of each of the following statistics: (1) LPLU, a minimally biased statistic, (2) UBND, the minimum estimate of an upper bound, (3) NI, the net improvement, (4) PTC, the predicted three-way cross, and (5) TCSC, the testcross of the populations. These statistics are biased estimators of the relative number of unique favorable alleles contained within a population compared to a reference elite hybrid. Based on rank correlations, all statistics except NI ranked populations similarly. The percent novel germplasm relative to the single cross to be improved was positively correlated with the estimates of favorable alleles except when NI was used as the estimator. The relationship estimators agreed with the genetic constitution of the donor populations. Strong positive correlations existed between diversity, based on the relationship rankings, and all the estimator rankings, except NI. Potential donor populations were effectively identified by LPLU, UBND, PTC, and TCSC. NI was not a good estimator of unique favorable alleles.     

A test for a threshold in an ordered sequence of correlated proportions

A test procedure using chi-square statistics is proposed for determining a threshold in an ordered sequence of correlated proportions. The procedure is based on the multivariate Bernoulli model. It is applied to the problem of ascertaining when visual acuity has stabilized in a group of patients with regular follow-up after a vision-reducing acute abnormality.     

Statistical analysis of microarray data: a Bayesian approach

The potential of microarray data is enormous. It allows us to monitor the expression of thousands of genes simultaneously. A common task with microarray is to determine which genes are differentially expressed between two samples obtained under two different conditions. Recently, several statistical methods have been proposed to perform such a task when there are replicate samples under each condition. Two major problems arise with microarray data. The first one is that the number of replicates is very small (usually 2-10), leading to noisy point estimates. As a consequence, traditional statistics that are based on the means and standard deviations, e.g. t-statistic, are not suitable. The second problem is that the number of genes is usually very large (approximately 10,000), and one is faced with an extreme multiple testing problem. Most multiple testing adjustments are relatively conservative, especially when the number of replicates is small. In this paper we present an empirical Bayes analysis that handles both problems very well. Using different parametrizations, we develop four statistics that can be used to test hypotheses about the means and/or variances of the gene expression levels in both one- and two-sample problems. The methods are illustrated using experimental data with prior knowledge. In addition, we present the result of a simulation comparing our methods to well-known statistics and multiple testing adjustments.     

Controlling the false discovery rate with constraints: the Newman-Keuls test revisited

The Newman-Keuls (NK) procedure for testing all pairwise comparisons among a set of treatment means, introduced by Newman (1939) and in a slightly different form by Keuls (1952) was proposed as a reasonable way to alleviate the inflation of error rates when a large number of means are compared. It was proposed before the concepts of different types of multiple error rates were introduced by Tukey (1952a, b; 1953). Although it was popular in the 1950s and 1960s, once control of the familywise error rate (FWER) was accepted generally as an appropriate criterion in multiple testing, and it was realized that the NK procedure does not control the FWER at the nominal level at which it is performed, the procedure gradually fell out of favor. Recently, a more liberal criterion, control of the false discovery rate (FDR), has been proposed as more appropriate in some situations than FWER control. This paper notes that the NK procedure and a nonparametric extension controls the FWER within any set of homogeneous treatments. It proves that the extended procedure controls the FDR when there are well-separated clusters of homogeneous means and between-cluster test statistics are independent, and extensive simulation provides strong evidence that the original procedure controls the FDR under the same conditions and some dependent conditions when the clusters are not well-separated. Thus, the test has two desirable error-controlling properties, providing a compromise between FDR control with no subgroup FWER control and global FWER control. Yekutieli (2002) developed an FDR-controlling procedure for testing all pairwise differences among means, without any FWER-controlling criteria when there is more than one cluster. The empirica example in Yekutieli's paper was used to compare the Benjamini-Hochberg (1995) method with apparent FDR control in this context, Yekutieli's proposed method with proven FDR control, the Newman-Keuls method that controls FWER within equal clusters with apparent FDR control, and several methods that control FWER globally. The Newman-Keuls is shown to be intermediate in number of rejections to the FWER-controlling methods and the FDR-controlling methods in this example, although it is not always more conservative than the other FDR-controlling methods.     

A simulation test of Smith's “degrees of freedom” correction for comparative studies

Computer simulation was used to test Smith's (1994) correction for phylogenetic nonindependence in comparative studies. Smith's method finds effective N, which is computed using nested analysis of variance, and uses this value in place of observed N as the baseline degrees of freedom (df) for calculating statistical significance levels. If Smith's formula finds the correct df, distributions of computer-generated statistics from simulations with observed N nonindependent species should match theoretical distributions (from statistical tables) with the df based on effective N. The computer program developed to test Smith's method simulates character evolution down user-specified phylogenies. Parameters were systematically varied to discover their effects on Smith's method. In simulations in which the phylogeny and taxonomy were identical (tests of narrow-sense validity), Smith's method always gave conservative statistical results when the taxonomy had fewer than five levels. This conservative departure gave way to a liberal deviation in type I error rates in simulations using more than five taxonomic levels, except when species values were nearly independent. Reducing the number of taxonomic levels used in the analysis, and thereby eliminating available information regarding evolutionary relationships, also increased type I error rates (broad-sense validity), indicating that this may be inappropriate under conditions shown to have high type I error rates. However, the use of taxonomic categories over more accurate phylogenies did not create a liberal bias in all cases in the analysis performed here. The effect of correlated trait evolution was ambiguous but, relative to other parameters, negligible. © 1995 Wiley-Liss, Inc.     

On identification of the number of best treatments using the Newman-Keuls test

In this paper, we provide a stochastic ordering of the Studentized range statistics under a balanced one-way ANOVA model. Based on this result we show that, when restricted to the multiple comparisons with the best, the Newman-Keuls (NK) procedure strongly controls experimentwise error rate for a sequence of null hypotheses regarding the number of largest treatment means. In other words, the NK procedure provides an upper confidence bound for the number of best treatments.     

Homogeneity test of rate ratios in stratified matched-pair studies

This paper investigates homogeneity test of rate ratios in stratified matched-pair studies on the basis of asymptotic and bootstrap-resampling methods. Based on the efficient score approach, we develop a simple and computationally tractable score test statistic. Several other homogeneity test statistics are also proposed on the basis of the weighted least-squares estimate and logarithmic transformation. Sample size formulae are derived to guarantee a pre-specified power for the proposed tests at the pre-given significance level. Empirical results confirm that (i) the modified score statistic based on the bootstrap-resampling method performs better in the sense that its empirical type I error rate is much closer to the pre-specified nominal level than those of other tests and its power is greater than those of other tests, and is hence recommended, whilst the statistics based on the weighted least-squares estimate and logarithmic transformation are slightly conservative under some of the considered settings; (ii) the derived sample size formulae are rather accurate in the sense that their empirical powers obtained from the estimated sample sizes are very close to the pre-specified nominal powers. A real example is used to illustrate the proposed methodologies.     

Bootstrap method of interior-branch test for phylogenetic trees

Statistical properties of the bootstrap test of interior branch lengths of phylogenetic trees have been studied and compared with those of the standard interior-branch test in computer simulations. Examination of the properties of the tests under the null hypothesis showed that both tests for an interior branch of a predetermined topology are quite reliable when the distribution of the branch length estimate approaches a normal distribution. Unlike the standard interior-branch test, the bootstrap test appears to retain this property even when the substitution rate varies among sites. In this case, the distribution of the branch length estimate deviates from a normal distribution, and the standard interior-branch test gives conservative confidence probability values. A simple correction method was developed for both interior- branch tests to be applied for testing the reliability of tree topologies estimated from sequence data. This correction for the standard interior-branch test appears to be as effective as that obtained in our previous study, though it is much simpler. The bootstrap and standard interior-branch tests for estimated topologies become conservative as the number of sequence groups in a star-like tree increases.      

On the power of some binomial modifications of the Bonferroni multiple test

Widely used in testing statistical hypotheses, the Bonferroni multiple test has a rather low power that entails a high risk to accept falsely the overall null hypothesis and therefore to not detect really existing effects. We suggest that when the partial test statistics are statistically independent, it is possible to reduce this risk by using binomial modifications of the Bonferroni test. Instead of rejecting the null hypothesis when at least one of n partial null hypotheses is rejected at a very high level of significance (say, 0.005 in the case of n = 10), as it is prescribed by the Bonferroni test, the binomial tests recommend to reject the null hypothesis when at least k partial null hypotheses (say, k = [n/2]) are rejected at much lower level (up to 30-50%). We show that the power of such binomial tests is essentially higher as compared with the power of the original Bonferroni and some modified Bonferroni tests. In addition, such an approach allows us to combine tests for which the results are known only for a fixed significance level. The paper contains tables and a computer program which allow to determine (retrieve from a table or to compute) the necessary binomial test parameters, i.e. either the partial significance level (when k is fixed) or the value of k (when the partial significance level is fixed).     

In situ analysis of cross-hybridisation on microarrays and the inference of expression correlation

Background

When conducting multiple hypothesis tests, it is important to control the number of false positives, or the False Discovery Rate (FDR). However, there is a tradeoff between controlling FDR and maximizing power. Several methods have been proposed, such as the q-value method, to estimate the proportion of true null hypothesis among the tested hypotheses, and use this estimation in the control of FDR. These methods usually depend on the assumption that the test statistics are independent (or only weakly correlated). However, many types of data, for example microarray data, often contain large scale correlation structures. Our objective was to develop methods to control the FDR while maintaining a greater level of power in highly correlated datasets by improving the estimation of the proportion of null hypotheses.

Results

We showed that when strong correlation exists among the data, which is common in microarray datasets, the estimation of the proportion of null hypotheses could be highly variable resulting in a high level of variation in the FDR. Therefore, we developed a re-sampling strategy to reduce the variation by breaking the correlations between gene expression values, then using a conservative strategy of selecting the upper quartile of the re-sampling estimations to obtain a strong control of FDR.

Conclusion

With simulation studies and perturbations on actual microarray datasets, our method, compared to competing methods such as q-value, generated slightly biased estimates on the proportion of null hypotheses but with lower mean square errors. When selecting genes with controlling the same FDR level, our methods have on average a significantly lower false discovery rate in exchange for a minor reduction in the power.     

Hypothesis Testing Procedures for a Series of Independent Fourfold Tables under Inverse Sampling

This paper considers four summary test statistics, including the one recently proposed by Bennett (1986, Biometrical Journal 28, 859–862), for hypothesis testing of association in a series of independent fourfold tables under inverse sampling. This paper provides a systematic and quantitative evaluation of the small-sample performance for these summary test statistics on the basis of a Monte Carlo simulation. This paper notes that the test statistic developed by Bennett (1986) can be conservative and thereby possibly lose the power when the underlying disease is not rare. This paper also finds that for given a fixed total number of cases in each table, the conditional test statistic is the best in controlling type I error among all test statistics considered here.     

An exact trend test for correlated binary data

The problem of testing a dose-response relationship in the presence of exchangeably correlated binary data has been addressed using a variety of models. Most commonly used approaches are derived from likelihood or generalized estimating equations and rely on large-sample theory to justify their inferences. However, while earlier work has determined that these methods may perform poorly for small or sparse samples, there are few alternatives available to those faced with such data. We propose an exact trend test for exchangeably correlated binary data when groups of correlated observations are ordered. This exact approach is based on an exponential model derived by Molenberghs and Ryan (1999) and Ryan and Molenberghs (1999) and provides natural analogues to Fisher's exact test and the binomial trend test when the data are correlated. We use a graphical method with which one can efficiently compute the exact tail distribution and apply the test to two examples.     

Cluster-Based Statistics for Brain Connectivity in Correlation with Behavioral Measures

Graph theoretical approaches have successfully revealed abnormality in brain connectivity, in particular, for contrasting patients from healthy controls. Besides the group comparison analysis, a correlational study is also challenging. In studies with patients, for example, finding brain connections that indeed deepen specific symptoms is interesting. The correlational study is also beneficial since it does not require controls, which are often difficult to find, especially for old-age patients with cognitive impairment where controls could also have cognitive deficits due to normal ageing. However, one of the major difficulties in such correlational studies is too conservative multiple comparison correction. In this paper, we propose a novel method for identifying brain connections that are correlated with a specific cognitive behavior by employing cluster-based statistics, which is less conservative than other methods, such as Bonferroni correction, false discovery rate procedure, and extreme statistics. Our method is based on the insight that multiple brain connections, rather than a single connection, are responsible for abnormal behaviors. Given brain connectivity data, we first compute a partial correlation coefficient between every edge and the behavioral measure. Then we group together neighboring connections with strong correlation into clusters and calculate their maximum sizes. This procedure is repeated for randomly permuted assignments of behavioral measures. Significance levels of the identified sub-networks are estimated from the null distribution of the cluster sizes. This method is independent of network construction methods: either structural or functional network can be used in association with any behavioral measures. We further demonstrated the efficacy of our method using patients with subcortical vascular cognitive impairment. We identified sub-networks that are correlated with the disease severity by exploiting diffusion tensor imaging techniques. The identified sub-networks were consistent with the previous clinical findings having valid significance level, while other methods did not assert any significant findings.     

A powerful and robust test statistic for randomization inference in group-randomized trials with matched pairs of groups

For group-randomized trials, randomization inference based on rank statistics provides robust, exact inference against nonnormal distributions. However, in a matched-pair design, the currently available rank-based statistics lose significant power compared to normal linear mixed model (LMM) test statistics when the LMM is true. In this article, we investigate and develop an optimal test statistic over all statistics in the form of the weighted sum of signed Mann-Whitney-Wilcoxon statistics under certain assumptions. This test is almost as powerful as the LMM even when the LMM is true, but it is much more powerful for heavy tailed distributions. A simulation study is conducted to examine the power.     

The HKA test revisited: a maximum-likelihood-ratio test of the standard neutral model

We present a maximum-likelihood-ratio test of the standard neutral model, using multilocus data on polymorphism within species and divergence between species. The model is based on the Hudson-Kreitman-Aguade (HKA) test, but allows for an explicit test of selection at individual loci in a multilocus framework. We use coalescent simulations to show that the likelihood-ratio test statistic is conservative, particularly when the assumption of no recombination is violated. Application of the method to polymorphism data from 18 loci from a population of Arabidopsis lyrata provides significant evidence for a balanced polymorphism at a candidate locus thought to be linked to the centromere. The method is also applied to polymorphism data in maize, providing support for the hypothesis of directional selection on genes in the starch pathway.