Testing the Independence of Two Diagnostic Tests

Consider two diagnostic procedures having binary outcomes. If one of the tests results in a positive finding, a more definitive diagnostic procedure will be administered to establish the presence or absence of a disease. The use of both tests will improve the overall screening sensitivity when the two tests are independent, compared with employing two tests that are positively correlated. We estimate the correlation coefficient of the two tests and derive statistical methods for testing the independence of the two diagnostic procedures conditional on disease status. The statistical tests are used to investigate the independence of mammography and clinical breast exams aimed at establishing the benefit of early detection of breast cancer. The data used in the analysis are obtained from periodic screening examinations of three randomized clinical trials of breast cancer screening. Analysis of each of these trials confirms the independence of the clinical breast and mammography examinations. Based on these three large clinical trials, we conclude that a clinical breast exam considerably increases the overall sensitivity relative to screening with mammography alone and should be routinely included in early breast cancer detection programs.     

Simultaneous Randomization Tests

Randomization analyses have been developed for testing main effects and interactions in standard experimental designs. However, exact multiple comparisons procedures for these randomization analyses have received little attention. This article proposes a general procedure for constructing simultaneous randomization tests that have prescribed type I error rates. An application of the procedure does provide for multiple comparisons in the randomization analyses of designed experiments. This application is made to data collected in a biopharmaceutical experiment.     

On weighted Hochberg procedures

We consider different ways of constructing weighted Hochberg-typestep-up multiple test procedures including closed proceduresbased on weighted Simes tests and their conservative step-upshort-cuts, and step-up counterparts of two weighted Holm procedures.It is shown that the step-up counterparts have some seriouspitfalls such as lack of familywise error rate control and lackof monotonicity in rejection decisions in terms of p-values.Therefore an exact closed procedure appears to be the best alternative,its only drawback being lack of simple stepwise structure. Aconservative step-up short-cut to the closed procedure may beused instead, but with accompanying loss of power. Simulationsare used to study the familywise error rate and power propertiesof the competing procedures for independent and correlated p-values.Although many of the results of this paper are negative, theyare useful in highlighting the need for caution when procedureswith similar pitfalls may be used.     

A multiple comparison procedure for dose-finding trials with subpopulations

Identifying subgroups of patients with an enhanced response to a new treatment has become an area of increased interest in the last few years. When there is knowledge about possible subpopulations with an enhanced treatment effect before the start of a trial it might be beneficial to set up a testing strategy, which tests for a significant treatment effect not only in the full population, but also in these prespecified subpopulations. In this paper, we present a parametric multiple testing approach for tests in multiple populations for dose-finding trials. Our approach is based on the MCP-Mod methodology, which uses multiple comparison procedures (MCPs) to test for a dose–response signal, while considering multiple possible candidate dose–response shapes. Our proposed methods allow for heteroscedastic error variances between populations and control the family-wise error rate over tests in multiple populations and for multiple candidate models. We show in simulations that the proposed multipopulation testing approaches can increase the power to detect a significant dose–response signal over the standard single-population MCP-Mod, when the specified subpopulation has an enhanced treatment effect.     

Simultaneous Non‐inferiority Test of Sensitivity and Specificity for Two Diagnostic Procedures in the Presence of a Gold Standard

Sensitivity and specificity have traditionally been used to assess the performance of a diagnostic procedure. Diagnostic procedures with both high sensitivity and high specificity are desirable, but these procedures are frequently too expensive, hazardous, and/or difficult to operate. A less sophisticated procedure may be preferred, if the loss of the sensitivity or specificity is determined to be clinically acceptable. This paper addresses the problem of simultaneous testing of sensitivity and specificity for an alternative test procedure with a reference test procedure when a gold standard is present. The hypothesis is formulated as a compound hypothesis of two non‐inferiority (one‐sided equivalence) tests. We present an asymptotic test statistic based on the restricted maximum likelihood estimate in the framework of comparing two correlated proportions under the prospective and retrospective sampling designs. The sample size and power of an asymptotic test statistic are derived. The actual type I error and power are calculated by enumerating the exact probabilities in the rejection region. For applications that require high sensitivity as well as high specificity, a large number of positive subjects and a large number of negative subjects are needed. We also propose a weighted sum statistic as an alternative test by comparing a combined measure of sensitivity and specificity of the two procedures. The sample size determination is independent of the sampling plan for the two tests.     

False discovery rate control in two-stage designs

ABSTRACT: BACKGROUND: For gene expression or gene association studies with a large number of hypotheses the number of measurements per marker in a conventional single-stage design is often low due to limited resources. Two-stage designs have been proposed where in a first stage promising hypotheses are identified and further investigated in the second stage with larger sample sizes. For two types of two-stage designs proposed in the literature we derive multiple testing procedures controlling the False Discovery Rate (FDR) demonstrating FDR control by simulations: designs where a fixed number of top-ranked hypotheses are selected and designs where the selection in the interim analysis is based on an FDR threshold. In contrast to earlier approaches which use only the second-stage data in the hypothesis tests (pilot approach), the proposed testing procedures are based on the pooled data from both stages (integrated approach). Results: For both selection rules the multiple testing procedures control the FDR in the considered simulation scenarios. This holds for the case of independent observations across hypotheses as well as for certain correlation structures. Additionally, we show that in scenarios with small effect sizes the testing procedures based on the pooled data from both stages can give a considerable improvement in power compared to tests based on the second-stage data only. Conclusion: The proposed hypothesis tests provide a tool for FDR control for the considered two-stage designs. Comparing the integrated approaches for both selection rules with the corresponding pilot approaches showed an advantage of the integrated approach in many simulation scenarios.     

Implementing false discovery rate control: increasing your power

Popular procedures to control the chance of making type I errors when multiple statistical tests are performed come at a high cost: a reduction in power. As the number of tests increases, power for an individual test may become unacceptably low. This is a consequence of minimizing the chance of making even a single type I error, which is the aim of, for instance, the Bonferroni and sequential Bonferroni procedures. An alternative approach, control of the false discovery rate (FDR), has recently been advocated for ecological studies. This approach aims at controlling the proportion of significant results that are in fact type I errors. Keeping the proportion of type I errors low among all significant results is a sensible, powerful, and easy-to-interpret way of addressing the multiple testing issue. To encourage practical use of the approach, in this note we illustrate how the proposed procedure works, we compare it to more traditional methods that control the familywise error rate, and we discuss some recent useful developments in FDR control.     

Cluster formation as a measure of interpretability in multiple testing

Multiple test procedures are usually compared on various aspects of error control and power. Power is measured as some function of the number of false hypotheses correctly identified as false. However, given equal numbers of rejected false hypotheses, the pattern of rejections, i.e. the particular set of false hypotheses identified, may be crucial in interpreting the results for potential application.In an important area of application, comparisons among a set of treatments based on random samples from populations, two different approaches, cluster analysis and model selection, deal implicitly with such patterns, while traditional multiple testing procedures generally focus on the outcomes of subset and pairwise equality hypothesis tests, without considering the overall pattern of results in comparing methods. An important feature involving the pattern of rejections is their relevance for dividing the treatments into distinct subsets based on some parameter of interest, for example their means. This paper introduces some new measures relating to the potential of methods for achieving such divisions. Following Hartley (1955), sets of treatments with equal parameter values will be called clusters. Because it is necessary to distinguish between clusters in the populations and clustering in sample outcomes, the population clusters will be referred to as P -clusters; any related concepts defined in terms of the sample outcome will be referred to with the prefix outcome. Outcomes of multiple comparison procedures will be studied in terms of their probabilities of leading to separation of treatments into outcome clusters, with various measures relating to the number of such outcome clusters and the proportion of true vs. false outcome clusters. The definitions of true and false outcome clusters and related concepts, and the approach taken here, is in the tradition of hypothesis testing with attention to overall error control and power, but with added consideration of cluster separation potential.The pattern approach will be illustrated by comparing two methods with apparent FDR control but with different ways of ordering outcomes for potential significance: The original Benjamini-Hochberg (1995) procedure (BH), and the Newman-Keuls (Newman, 1939; Keuls, 1952) procedure (NK).     

Genetic spatial autocorrelation can readily detect sex-biased dispersal

Sex-biased dispersal is expected to generate differences in the fine-scale genetic structure of males and females. Therefore, spatial analyses of multilocus genotypes may offer a powerful approach for detecting sex-biased dispersal in natural populations. However, the effects of sex-biased dispersal on fine-scale genetic structure have not been explored. We used simulations and multilocus spatial autocorrelation analysis to investigate how sex-biased dispersal influences fine-scale genetic structure. We evaluated three statistical tests for detecting sex-biased dispersal: bootstrap confidence intervals about autocorrelation r values and recently developed heterogeneity tests at the distance class and whole correlogram levels. Even modest sex bias in dispersal resulted in significantly different fine-scale spatial autocorrelation patterns between the sexes. This was particularly evident when dispersal was strongly restricted in the less-dispersing sex (mean distance <200 m), when differences between the sexes were readily detected over short distances. All tests had high power to detect sex-biased dispersal with large sample sizes (n ≥ 250). However, there was variation in type I error rates among the tests, for which we offer specific recommendations. We found congruence between simulation predictions and empirical data from the agile antechinus, a species that exhibits male-biased dispersal, confirming the power of individual-based genetic analysis to provide insights into asymmetries in male and female dispersal. Our key recommendations for using multilocus spatial autocorrelation analyses to test for sex-biased dispersal are: (i) maximize sample size, not locus number; (ii) concentrate sampling within the scale of positive structure; (iii) evaluate several distance class sizes; (iv) use appropriate methods when combining data from multiple populations; (v) compare the appropriate groups of individuals.     

Testing Jumps via False Discovery Rate Control

Many recently developed nonparametric jump tests can be viewed as multiple hypothesis testing problems. For such multiple hypothesis tests, it is well known that controlling type I error often makes a large proportion of erroneous rejections, and such situation becomes even worse when the jump occurrence is a rare event. To obtain more reliable results, we aim to control the false discovery rate (FDR), an efficient compound error measure for erroneous rejections in multiple testing problems. We perform the test via the Barndorff-Nielsen and Shephard (BNS) test statistic, and control the FDR with the Benjamini and Hochberg (BH) procedure. We provide asymptotic results for the FDR control. From simulations, we examine relevant theoretical results and demonstrate the advantages of controlling the FDR. The hybrid approach is then applied to empirical analysis on two benchmark stock indices with high frequency data.     

Estimating species trees using multiple-allele DNA sequence data

Several techniques, such as concatenation and consensus methods, are available for combining data from multiple loci to produce a single statement of phylogenetic relationships. However, when multiple alleles are sampled from individual species, it becomes more challenging to estimate relationships at the level of species, either because concatenation becomes inappropriate due to conflicts among individual gene trees, or because the species from which multiple alleles have been sampled may not form monophyletic groups in the estimated tree. We propose a Bayesian hierarchical model to reconstruct species trees from multiple-allele, multilocus sequence data, building on a recently proposed method for estimating species trees from single allele multilocus data. A two-step Markov Chain Monte Carlo (MCMC) algorithm is adopted to estimate the posterior distribution of the species tree. The model is applied to estimate the posterior distribution of species trees for two multiple-allele datasets--yeast (Saccharomyces) and birds (Manacus-manakins). The estimates of the species trees using our method are consistent with those inferred from other methods and genetic markers, but in contrast to other species tree methods, it provides credible regions for the species tree. The Bayesian approach described here provides a powerful framework for statistical testing and integration of population genetics and phylogenetics.     

Applying the generalized partitioning principle to control the generalized familywise error rate

In multiple testing, strong control of the familywise error rate (FWER) may be unnecessarily stringent in some situations such as bioinformatic studies. An alternative approach, discussed by Hommel and Hoffmann (1988) and Lehmann and Romano (2005), is to control the generalized familywise error rate (gFWER), the probability of incorrectly rejecting more than m hypotheses. This article presents the generalized Partitioning Principle as a systematic technique of constructing gFWER-controlling tests that can take the joint distribution of test statistics into account. The paper is structured as follows. We first review classical partitioning principle, indicating its conditioning nature. Then the generalized partitioning principle is introduced, with a set of sufficient conditions that allows it to be executed as a computationally more feasible step-down test. Finally, we show the importance of having some knowledge of the distribution of the observations in multiple testing. In particular, we show that step-down permutation tests require an assumption on the joint distribution of the observations in order to control the familywise error rate.     

The strain-dependent osmotic pressure and stiffness of the bovine nucleus pulposus apportioned into ionic and non-ionic contributors

Elucidation of the load-bearing mechanism of the nucleus pulposus (NP) facilitates understanding of the mechanical and metabolic functioning of the intervertebral disc and provides key data for mathematical models. Negatively charged proteoglycans in the NP generate an ionic osmotic pressure, pi(i), which contributes to the tissue's resistance to load and, moreover, is the main mechanism by which the unloaded disc rehydrates. Functionally important, pi(i) has seldom been investigated in situ and, crucially, its variation with strain has not been reported. In a confined compression apparatus, we aimed to apportion the strain-dependent load-bearing mechanism of the NP at equilibrium to the tissue matrix and ionic osmotic pressure; and to determine whether any proteoglycan loss occurs during confined compression testing. Forty-eight confined compression experiments were conducted in isotonic (0.15M NaCl) and hypertonic (3.0 and 6.1M NaCl) external solutions in single and multiple step-strain protocols. The 6.1M NaCl external solution was needed to eliminate as much of the ionic effects as possible. The ionic osmotic pressure was well described by pi(i)=19.1lambda(-1.58) (R(2)=0.992), and was approximately 70% of the applied load at equilibrium, independent of lambda. The effective aggregate modulus, H(A)(eff), also increased with strain: H(A)(eff)=59.0lambda(-2.18). Concentrations of sulphated glycosaminoglycans were obtained for the samples tested in isotonic NaCl with no proteoglycan loss detected from the confined compression tests. These results highlight the non-linearity of the stress-strain response of NP tissue and the necessity to include a non-linear function for osmotic pressure in mathematical models of this tissue.     

Estimating and controlling for spatial structure in the study of ecological communities

Aim Variation partitioning based on canonical analysis is the most commonly used analysis to investigate community patterns according to environmental and spatial predictors. Ecologists use this method in order to understand the pure contribution of the environment independent of space, and vice versa, as well as to control for inflated type I error in assessing the environmental component under spatial autocorrelation. Our goal is to use numerical simulations to compare how different spatial predictors and model selection procedures perform in assessing the importance of the spatial component and in controlling for type I error while testing environmental predictors. Innovation We determine for the first time how the ability of commonly used (polynomial regressors) and novel methods based on eigenvector maps compare in the realm of spatial variation partitioning. We introduce a novel forward selection procedure to select spatial regressors for community analysis. Finally, we point out a number of issues that have not been previously considered about the joint explained variation between environment and space, which should be taken into account when reporting and testing the unique contributions of environment and space in patterning ecological communities. Main conclusions In tests of species‐environment relationships, spatial autocorrelation is known to inflate the level of type I error and make the tests of significance invalid. First, one must determine if the spatial component is significant using all spatial predictors (Moran's eigenvector maps). If it is, consider a model selection for the set of spatial predictors (an individual‐species forward selection procedure is to be preferred) and use the environmental and selected spatial predictors in a partial regression or partial canonical analysis scheme. This is an effective way of controlling for type I error in such tests. Polynomial regressors do not provide tests with a correct level of type I error.     

A tutorial on statistical methods for population association studies

Although genetic association studies have been with us for many years, even for the simplest analyses there is little consensus on the most appropriate statistical procedures. Here I give an overview of statistical approaches to population association studies, including preliminary analyses (Hardy-Weinberg equilibrium testing, inference of phase and missing data, and SNP tagging), and single-SNP and multipoint tests for association. My goal is to outline the key methods with a brief discussion of problems (population structure and multiple testing), avenues for solutions and some ongoing developments.     

Consequences of genotyping errors for estimation of clonality: a case study on <Emphasis Type="Italic">Populus euphratica</Emphasis> Oliv. (Salicaceae)

A study including eight microsatellite loci for 1,014 trees from seven mapped stands of the partially clonal Populus euphratica was used to demonstrate how genotyping errors influence estimates of clonality. With a threshold of 0 (identical multilocus genotypes constitute one clone) we identified 602 genotypes. A threshold of 1 (compensating for an error in one allele) lowered this number to 563. Genotyping errors can seemingly merge (type 1 error), split really existing clones (type 2), or convert a unique genotype into another unique genotype (type 3). We used context information (sex and spatial position) to estimate the type 1 error. For thresholds of 0 and 1 the estimate was below 0.021, suggesting a high resolution for the marker system. The rate of genotyping errors was estimated by repeated genotyping for a cohort of 41 trees drawn at random (0.158), and a second cohort of 40 trees deviating in one allele from another tree (0.368). For the latter cohort, most of these deviations turned out to be errors, but 8 out of 602 obtained multilocus genotypes may represent somatic mutations, corresponding to a mutation rate of 0.013. A simulation of genotyping errors for populations with varying clonality and evenness showed the number of genotypes always to be overestimated for a system with high resolution, and this mistake increases with increasing clonality and evenness. Allowing a threshold of 1 compensates for most genotyping errors and leads to much more precise estimates of clonality compared with a threshold of 0. This lowers the resolution of the marker system, but comparison with context information can help to check if the resolution is sufficient to apply a higher threshold. We recommend simulation procedures to investigate the behavior of a marker system for different thresholds and error rates to obtain the best estimate of clonality.     

Testing for group effect in a 2 x k heteroscedastic ANOVA model

It is natural to want to relax the assumption of homoscedasticity and Gaussian error in ANOVA models. For a two-way ANOVA model with 2 x k cells, one can derive tests of main effect for the factor with two levels (referred to as group) without assuming homoscedasticity or Gaussian error. Empirical likelihood can be used to derive testing procedures. An approximate empirical likelihood ratio test (AELRT) is derived for the test of group main effect. To approximate the distributions of the test statistics under the null hypothesis, simulation from the approximate empirical maximum likelihood estimate (AEMLE) restricted by the null hypothesis is used. The homoscedastic ANOVA F -test and a Box-type approximation to the distribution of the heteroscedastic ANOVA F -test are compared to the AELRT in level and power. The AELRT procedure is shown by simulation to have appropriate type I error control (although possibly conservative) when the distribution of the test statistics are approximated by simulation from the constrained AEMLE. The methodology is motivated and illustrated by an analysis of folate levels in the blood among two alcohol intake groups while accounting for gender.     

Empirical Bayes screening of many p-values with applications to microarray studies

MOTIVATION: Statistical tests for the detection of differentially expressed genes lead to a large collection of p-values one for each gene comparison. Without any further adjustment, these p-values may lead to a large number of false positives, simply because the number of genes to be tested is huge, which might mean wastage of laboratory resources. To account for multiple hypotheses, these p-values are typically adjusted using a single step method or a step-down method in order to achieve an overall control of the error rate (the so-called familywise error rate). In many applications, this may lead to an overly conservative strategy leading to too few genes being flagged. RESULTS: In this paper we introduce a novel empirical Bayes screening (EBS) technique to inspect a large number of p-values in an effort to detect additional positive cases. In effect, each case borrows strength from an overall picture of the alternative hypotheses computed from all the p-values, while the entire procedure is calibrated by a step-down method so that the familywise error rate at the complete null hypothesis is still controlled. It is shown that the EBS has substantially higher sensitivity than the standard step-down approach for multiple comparison at the cost of a modest increase in the false discovery rate (FDR). The EBS procedure also compares favorably when compared with existing FDR control procedures for multiple testing. The EBS procedure is particularly useful in situations where it is important to identify all possible potentially positive cases which can be subjected to further confirmatory testing in order to eliminate the false positives. We illustrated this screening procedure using a data set on human colorectal cancer where we show that the EBS method detected additional genes related to colon cancer that were missed by other methods.This novel empirical Bayes procedure is advantageous over our earlier proposed empirical Bayes adjustments due to the following reasons: (i) it offers an automatic screening of the p-values the user may obtain from a univariate (i.e., gene by gene) analysis package making it extremely easy to use for a non-statistician, (ii) since it applies to the p-values, the tests do not have to be t-tests; in particular they could be F-tests which might arise in certain ANOVA formulations with expression data or even nonparametric tests, (iii) the empirical Bayes adjustment uses nonparametric function estimation techniques to estimate the marginal density of the transformed p-values rather than using a parametric model for the prior distribution and is therefore robust against model mis-specification. AVAILABILITY: R code for EBS is available from the authors upon request. SUPPLEMENTARY INFORMATION: http://www.stat.uga.edu/~datta/EBS/supp.htm     

Exact unconditional inference for risk ratio in a correlated 2 x 2 table with structural zero

In this article, we consider small-sample statistical inference for rate ratio (RR) in a correlated 2 x 2 table with a structural zero in one of the off-diagonal cells. Existing Wald's test statistic and logarithmic transformation test statistic will be adopted for this purpose. Hypothesis testing and confidence interval construction based on large-sample theory will be reviewed first. We then propose reliable small-sample exact unconditional procedures for hypothesis testing and confidence interval construction. We present empirical results to evince the better confidence interval performance of our proposed exact unconditional procedures over the traditional large-sample procedures in small-sample designs. Unlike the findings given in Lui (1998, Biometrics 54, 706-711), our empirical studies show that the existing asymptotic procedures may not attain a prespecified confidence level even in moderate sample-size designs (e.g., n = 50). Our exact unconditional procedures on the other hand do not suffer from this problem. Hence, the asymptotic procedures should be applied with caution. We propose two approximate unconditional confidence interval construction methods that outperform the existing asymptotic ones in terms of coverage probability and expected interval width. Also, we empirically demonstrate that the approximate unconditional tests are more powerful than their associated exact unconditional tests. A real data set from a two-step tuberculosis testing study is used to illustrate the methodologies.     

Controlling the proportion of false positives in multiple dependent tests

Genome scan mapping experiments involve multiple tests of significance. Thus, controlling the error rate in such experiments is important. Simple extension of classical concepts results in attempts to control the genomewise error rate (GWER), i.e., the probability of even a single false positive among all tests. This results in very stringent comparisonwise error rates (CWER) and, consequently, low experimental power. We here present an approach based on controlling the proportion of false positives (PFP) among all positive test results. The CWER needed to attain a desired PFP level does not depend on the correlation among the tests or on the number of tests as in other approaches. To estimate the PFP it is necessary to estimate the proportion of true null hypotheses. Here we show how this can be estimated directly from experimental results. The PFP approach is similar to the false discovery rate (FDR) and positive false discovery rate (pFDR) approaches. For a fixed CWER, we have estimated PFP, FDR, pFDR, and GWER through simulation under a variety of models to illustrate practical and philosophical similarities and differences among the methods.