An efficient Monte Carlo approach to assessing statistical significance in genomic studies

MOTIVATION: Multiple hypothesis testing is a common problem in genome research, particularly in microarray experiments and genomewide association studies. Failure to account for the effects of multiple comparisons would result in an abundance of false positive results. The Bonferroni correction and Holm's step-down procedure are overly conservative, whereas the permutation test is time-consuming and is restricted to simple problems. RESULTS: We developed an efficient Monte Carlo approach to approximating the joint distribution of the test statistics along the genome. We then used the Monte Carlo distribution to evaluate the commonly used criteria for error control, such as familywise error rates and positive false discovery rates. This approach is applicable to any data structures and test statistics. Applications to simulated and real data demonstrate that the proposed approach provides accurate error control, and can be substantially more powerful than the Bonferroni and Holm methods, especially when the test statistics are highly correlated.     

The Benjamini-Hochberg method with infinitely many contrasts in linear models

Benjamini and Hochberg's method for controlling the false discoveryrate is applied to the problem of testing infinitely many contrastsin linear models. Exact, easily calculated critical values arederived, defining a new multiple comparisons method for testingcontrasts in linear models. The method is adaptive, dependingon the data through the F-statistic, like the Waller–DuncanBayesian multiple comparisons method. Comparisons with Scheffé'smethod are given, and the method is extended to the simultaneousconfidence intervals of Benjamini and Yekutieli.     

Localization of Thermoreceptor Systems that Induce Step-Up and Step-Down Thermophobic Responses and Switching in the Dominance of These Systems in Blepharisma

The distribution of thermoreceptor systems that initiate step-up and step-down thermophobic responses in bisected cells of Blepharisma was examined. Anterior cell fragments responded by ciliary reversal to a step-down in temperature and by repression of spontaneous ciliary reversal to a step-up. Posterior fragments responded by ciliary reversal to a step-up in thermal stimulation and by repression of spontaneous ciliary reversal in response to step-down stimulation. Results indicate that two kinds of thermoreceptor systems exist in the anterior half of each cell; one is responsible for ciliary reversal induced by step-down stimulation, and the other is responsible for repression of the ciliary reversal caused by step-up thermal stimulation. Likewise, there are also two kinds of thermoreceptor systems in the posterior half of the cell; one is responsible for ciliary reversal in response to a step-up in temperature, and the other is responsible for the repression of ciliary reversal on a step-down in thermal stimulation. Below about 27°C, intact cells showed ciliary reversal only when a step-down in thermal stimulation occurred, while above about 27°C cells only responded to a step-up in thermal stimulation. At about 27°C there was a switch in the dominant response from the anterior to the posterior half of an individual cell.     

Adjusted regression trend test for a multicenter clinical trial

Studies using a series of increasing doses of a compound, including a zero dose control, are often conducted to study the effect of the compound on the response of interest. For a one-way design, Tukey et al. (1985, Biometrics 41, 295-301) suggested assessing trend by examining the slopes of regression lines under arithmetic, ordinal, and arithmetic-logarithmic dose scalings. They reported the smallest p-value for the three significance tests on the three slopes for safety assessments. Capizzi et al. (1992, Biometrical Journal 34, 275-289) suggested an adjusted trend test, which adjusts the p-value using a trivariate t-distribution, the joint distribution of the three slope estimators. In this paper, we propose an adjusted regression trend test suitable for two-way designs, particularly for multicenter clinical trials. In a step-down fashion, the proposed trend test can be applied to a multicenter clinical trial to compare each dose with the control. This sequential procedure is a closed testing procedure for a trend alternative. Therefore, it adjusts p-values and maintains experimentwise error rate. Simulation results show that the step-down trend test is overall more powerful than a step-down least significant difference test.     

Simultaneous confidence regions corresponding to Holm's step-down procedure and other closed-testing procedures

Holm's (1979) step-down multiple-testing procedure (MTP) is appealing for its flexibility, transparency, and general validity, but the derivation of corresponding simultaneous confidence regions has remained an unsolved problem. This article provides such confidence regions. In fact, simultanenous confidence regions are provided for any MTP in the class of short-cut consonant closed-testing procedures based on marginal p -values and weighted Bonferroni tests for intersection hypotheses considered by Hommel, Bretz and Maurer (2007). In addition to Holm's MTP, this class includes the fixed-sequence MTP, recently proposed gatekeeping MTPs, and the fallback MTP. The simultaneous confidence regions are generally valid if underlying marginal p -values and corresponding marginal confidence regions (assumed to be available) are valid. The marginal confidence regions and estimated quantities are not assumed to be of any particular kinds/dimensions. Compared to the rejections made by the MTP for the family of null hypotheses H under consideration, the proposed confidence regions provide extra free information. In particular, with Holm's MTP, such extra information is provided: for all nonrejected H s, in case not all H s are rejected; or for certain (possibly all) H s, in case all H s are rejected. In case not all H s are rejected, no extra information is provided for rejected H s. This drawback seems however difficult to overcome. Illustrations concerning clinical studies are given.     

The Application of Hunter's Inequality in Simultaneous Testing

Using the inequality given by HUNTER (1976) critical limits are derived on the basis of bivariate distributions of the respective statistics for various situations of simultaneous hypothesis testing. The gain in power as compared to the widely used Bonferroni-inequality can be considerable if at least some of the simultaneously investigated test-statistics are highly correlated. In such situations, the loss in power as compared to the use of the exact critical limits might be negligible for practical purposes, the exact limits often being hardly accessible.     

Blinded sample size recalculation in multiple composite population designs with normal data and baseline adjustments

The increasing interest in subpopulation analysis has led to the development of various new trial designs and analysis methods in the fields of personalized medicine and targeted therapies. In this paper, subpopulations are defined in terms of an accumulation of disjoint population subsets and will therefore be called composite populations. The proposed trial design is applicable to any set of composite populations, considering normally distributed endpoints and random baseline covariates. Treatment effects for composite populations are tested by combining p-values, calculated on the subset levels, using the inverse normal combination function to generate test statistics for those composite populations while the closed testing procedure accounts for multiple testing. Critical boundaries for intersection hypothesis tests are derived using multivariate normal distributions, reflecting the joint distribution of composite population test statistics given no treatment effect exists. For sample size calculation and sample size, recalculation multivariate normal distributions are derived which describe the joint distribution of composite population test statistics under an assumed alternative hypothesis. Simulations demonstrate the absence of any practical relevant inflation of the type I error rate. The target power after sample size recalculation is typically met or close to being met.     

Multiple testing on the directed acyclic graph of gene ontology

MOTIVATION: Current methods for multiplicity adjustment do not make use of the graph structure of Gene Ontology (GO) when testing for association of expression profiles of GO terms with a response variable. RESULTS: We propose a multiple testing method, called the focus level procedure, that preserves the graph structure of Gene Ontology (GO). The procedure is constructed as a combination of a Closed Testing procedure with Holm's method. It requires a user to choose a 'focus level' in the GO graph, which reflects the level of specificity of terms in which the user is most interested. This choice also determines the level in the GO graph at which the procedure has most power. We prove that the procedure strongly controls the family-wise error rate without any additional assumptions on the joint distribution of the test statistics used. We also present an algorithm to calculate multiplicity-adjusted P-values. Because the focus level procedure preserves the structure of the GO graph, it does not generally preserve the ordering of the raw P-values in the adjusted P-values. AVAILABILITY: The focus level procedure has been implemented in the globaltest and GlobalAncova packages, both of which are available on www.bioconductor.org.     

Applying the generalized partitioning principle to control the generalized familywise error rate

In multiple testing, strong control of the familywise error rate (FWER) may be unnecessarily stringent in some situations such as bioinformatic studies. An alternative approach, discussed by Hommel and Hoffmann (1988) and Lehmann and Romano (2005), is to control the generalized familywise error rate (gFWER), the probability of incorrectly rejecting more than m hypotheses. This article presents the generalized Partitioning Principle as a systematic technique of constructing gFWER-controlling tests that can take the joint distribution of test statistics into account. The paper is structured as follows. We first review classical partitioning principle, indicating its conditioning nature. Then the generalized partitioning principle is introduced, with a set of sufficient conditions that allows it to be executed as a computationally more feasible step-down test. Finally, we show the importance of having some knowledge of the distribution of the observations in multiple testing. In particular, we show that step-down permutation tests require an assumption on the joint distribution of the observations in order to control the familywise error rate.     

The Exact Joint Distribution of the Sum of Heads and Apparent Size Statistics of a “Tandem Repeats Finder” Algorithm

Tandem repeats play many important roles in biological research. However, accurate characterization of their properties is limited by the inability to easily detect them. For this reason, much work has been devoted to developing detection algorithms. A widely used algorithm for detecting tandem repeats is the ‘`tandem repeats finder’' (Benson, G., Nucleic Acids Res. 27, 573--580, 1999). In that algorithm, tandem repeats are modeled by percent matches and frequency of indels between adjacent pattern copies, and statistical criteria are used to recognize them. We give a method for computing the exact joint distribution of a pair of statistics that are used in the testing procedures of the ‘`tandem repeats finder’': the total number of matches in matching tuples of length k or longer, and the total number of observations from the beginning of the first such matching tuple to the end of the last one. This allows the computation of the conditional distribution of the latter statistic given the former, a conditional distribution that is used to test for tandem repeats as opposed to non-tandem direct repeats. The setting is a Markovian sequence of a general order. Current approaches to this distributional problem deal only with independent trials and are based on approximations via simulation.     

Power for detecting genetic divergence: differences between statistical methods and marker loci

Information on statistical power is critical when planning investigations and evaluating empirical data, but actual power estimates are rarely presented in population genetic studies. We used computer simulations to assess and evaluate power when testing for genetic differentiation at multiple loci through combining test statistics or P values obtained by four different statistical approaches, viz. Pearson's chi-square, the log-likelihood ratio G-test, Fisher's exact test, and an F(ST)-based permutation test. Factors considered in the comparisons include the number of samples, their size, and the number and type of genetic marker loci. It is shown that power for detecting divergence may be substantial for frequently used sample sizes and sets of markers, also at quite low levels of differentiation. The choice of statistical method may be critical, though. For multi-allelic loci such as microsatellites, combining exact P values using Fisher's method is robust and generally provides a high resolving power. In contrast, for few-allele loci (e.g. allozymes and single nucleotide polymorphisms) and when making pairwise sample comparisons, this approach may yield a remarkably low power. In such situations chi-square typically represents a better alternative. The G-test without Williams's correction frequently tends to provide an unduly high proportion of false significances, and results from this test should be interpreted with great care. Our results are not confined to population genetic analyses but applicable to contingency testing in general.     

Statistical power when testing for genetic differentiation

A variety of statistical procedures are commonly employed when testing for genetic differentiation. In a typical situation two or more samples of individuals have been genotyped at several gene loci by molecular or biochemical means, and in a first step a statistical test for allele frequency homogeneity is performed at each locus separately, using, e.g. the contingency chi-square test, Fisher's exact test, or some modification thereof. In a second step the results from the separate tests are combined for evaluation of the joint null hypothesis that there is no allele frequency difference at any locus, corresponding to the important case where the samples would be regarded as drawn from the same statistical and, hence, biological population. Presently, there are two conceptually different strategies in use for testing the joint null hypothesis of no difference at any locus. One approach is based on the summation of chi-square statistics over loci. Another method is employed by investigators applying the Bonferroni technique (adjusting the P-value required for rejection to account for the elevated alpha errors when performing multiple tests simultaneously) to test if the heterogeneity observed at any particular locus can be regarded significant when considered separately. Under this approach the joint null hypothesis is rejected if one or more of the component single locus tests is considered significant under the Bonferroni criterion. We used computer simulations to evaluate the statistical power and realized alpha errors of these strategies when evaluating the joint hypothesis after scoring multiple loci. We find that the 'extended' Bonferroni approach generally is associated with low statistical power and should not be applied in the current setting. Further, and contrary to what might be expected, we find that 'exact' tests typically behave poorly when combined in existing procedures for joint hypothesis testing. Thus, while exact tests are generally to be preferred over approximate ones when testing each particular locus, approximate tests such as the traditional chi-square seem preferable when addressing the joint hypothesis.     

On the finite sample distribution of the likelihood ratio statistic for testing heterogeneity in meta-analysis

In meta-analysis, hypothesis testing is one of the commonly used approaches for assessing whether heterogeneity exists in effects between studies. The literature concluded that the Q-statistic is clearly the best choice and criticized the performance of the likelihood ratio test in terms of the type I error control and power. However, all the criticism for the likelihood ratio test is based on the use of a mixture of two chi-square distributions with 0 and 1 degrees of freedom, which is justified only asymptotically. In this study, we develop a novel method to derive the finite sample distribution of the likelihood ratio test and restricted likelihood ratio test statistics for testing the zero variance component in the random effects model for meta-analysis. We also extend this result to the heterogeneity test when metaregression is applied. A numerical study shows that the proposed statistics have superior performance to the Q-statistic, especially when the number of studies collected for meta-analysis is small to moderate.     

Weighted Logrank Testing Procedures for the Simple Tree Alternatives

Weighted logrank testing procedures for comparing r treatments with a control when some of the data are randomly censored are discussed. Four kinds of test statistics for the simple tree alternatives are considered. The weighted logrank statistics based on pairwise ranking scheme is proposed and the covariances of the test statistics are explicitly obtained. This class of test statistics can be viewed as the general statistics of constructing the test procedures for various order restricted alternatives by modifying weights. Four kinds of weighted logrank tests are illustrated with an example. Simulation studies are performed to compare the sizes and the powers of the considered tests with the other.     

A family of Bayes multiple testing procedures

Under the model of independent test statistics, we propose atwo-parameter family of Bayes multiple testing procedures. Thetwo parameters can be viewed as tuning parameters. Using theBenjamini–Hochberg step-up procedure for controlling falsediscovery rate as a baseline for conservativeness, we choosethe tuning parameters to compromise between the operating characteristicsof that procedure and a less conservative procedure that focuseson alternatives that a priori might be considered likely ormeaningful. The Bayes procedures do not have the theoreticaland practical shortcomings of the popular stepwise procedures.In terms of the number of mistakes, simulations for two examplesindicate that over a large segment of the parameter space, theBayes procedure is preferable to the step-up procedure. Anotherdesirable feature of the procedures is that they are computationallyfeasible for any number of hypotheses.     

FDR control by the BH procedure for two-sided correlated tests with implications to gene expression data analysis

The multiple testing problem attributed to gene expression analysis is challenging not only by its size, but also by possible dependence between the expression levels of different genes resulting from coregulations of the genes. Furthermore, the measurement errors of these expression levels may be dependent as well since they are subjected to several technical factors. Multiple testing of such data faces the challenge of correlated test statistics. In such a case, the control of the False Discovery Rate (FDR) is not straightforward, and thus demands new approaches and solutions that will address multiplicity while accounting for this dependency. This paper investigates the effects of dependency between bormal test statistics on FDR control in two-sided testing, using the linear step-up procedure (BH) of Benjamini and Hochberg (1995). The case of two multiple hypotheses is examined first. A simulation study offers primary insight into the behavior of the FDR subjected to different levels of correlation and distance between null and alternative means. A theoretical analysis follows in order to obtain explicit upper bounds to the FDR. These results are then extended to more than two multiple tests, thereby offering a better perspective on the effect of the proportion of false null hypotheses, as well as the structure of the test statistics correlation matrix. An example from gene expression data analysis is presented.     

Tests for Genetic Differentiation

When testing for genetic differentiation the joint null hypothesis that there is no allele frequency difference at any locus is of interest. Common approaches to test this hypothesis are based on the summation of χ² statistics over loci and on the Bonferroni correction, respectively. Here, we also consider the Simes adjustment and a recently proposed truncated product method (TPM) to combine P‐values. The summation and the TPM (using a relatively large truncation point) are powerful when there are differences in many or all loci. The Simes adjustment, however, is powerful when there are differences regarding one or a few loci only. As a compromise between the different approaches we introduce a combination between the Simes adjustment and the TPM, i.e. the joint null hypothesis is rejected if at least one of the two methods, Simes and TPM, is significant at the α/2‐level. Simulation results indicate that this combination is a robust procedure with high power over the different types of alternatives.     

On generalized Simes critical constants

We consider the problem treated by Simes of testing the overall null hypothesis formed by the intersection of a set of elementary null hypotheses based on ordered p‐values of the associated test statistics. The Simes test uses critical constants that do not need tabulation. Cai and Sarkar gave a method to compute generalized Simes critical constants which improve upon the power of the Simes test when more than a few hypotheses are false. The Simes constants can be viewed as the first order (requiring solution of a linear equation) and the Cai‐Sarkar constants as the second order (requiring solution of a quadratic equation) constants. We extend the method to third order (requiring solution of a cubic equation) constants, and also offer an extension to an arbitrary kth order. We show by simulation that the third order constants are more powerful than the second order constants for testing the overall null hypothesis in most cases. However, there are some drawbacks associated with these higher order constants especially for , which limits their practical usefulness.     

Estimating the false discovery rate using the stochastic approximation algorithm

Testing of multiple hypotheses involves statistics that arestrongly dependent in some applications, but most work on thissubject is based on the assumption of independence. We proposea new method for estimating the false discovery rate of multiplehypothesis tests, in which the density of test scores is estimatedparametrically by minimizing the Kullback–Leibler distancebetween the unknown density and its estimator using the stochasticapproximation algorithm, and the false discovery rate is estimatedusing the ensemble averaging method. Our method is applicableunder general dependence between test statistics. Numericalcomparisons between our method and several competitors, conductedon simulated and real data examples, show that our method achievesmore accurate control of the false discovery rate in almostall scenarios.     

Discovery of signaling effect of UV-B/C light in the extended UV-A/blue-type action spectra for step-down and step-up photophobic responses in the unicellular flagellate algaEuglena gracilis

Summary Cultures of unicellular algal flagellateEuglena gracilis grown in different conditions were subjected to action spectroscopy for step-down and step-up photophobic responses, respectively. The spectral region was extended into the UV-B/C as well as in the UV-A and visible regions with the Okazaki Large Spectrograph as the monochromatic light source. The photophobic responses of the cells were measured with an individual-cell assay method with the aid of a computerized video motion analyzer. In the UV-A and visible regions, the shapes of the action spectra were the so-called UV-A/blue type. In the newly studied UV-B/C region, new action peaks were found at 270 nm for the step-down response and at 280 nm for the step-up one. The absorption spectrum of flavin adenine dinucleotide (FAD) appeared to fit the action spectrum for the step-up response, whereas the shape of the step-down action spectrum, which has a UV-A peak (at 370 nm) higher than the blue peak (at 450 nm), appeared to be mimicked by the absorption spectrum of a mixed solution of 6-biopterin and FAD. These observations might also account for the fact that the UV-B/C peak wavelength at 270 nm of the action spectrum for the step-down response is shorter by 10 nm than the action spectrum for the step-up response at 280 nm.Abbreviations FAD flavin adenine dinucleotide - FWHM spectral full width at half maximum - NIBB National Institute for Basic Biology - OLS Okazaki Large Spectrograph - PFB paraflagellar body - UV-A ultraviolet light of spectral region between 320 and 400 nm - UV-B/C ultraviolet light of spectral region between 190 and 320 nm