Cluster formation as a measure of interpretability in multiple testing

Multiple test procedures are usually compared on various aspects of error control and power. Power is measured as some function of the number of false hypotheses correctly identified as false. However, given equal numbers of rejected false hypotheses, the pattern of rejections, i.e. the particular set of false hypotheses identified, may be crucial in interpreting the results for potential application.In an important area of application, comparisons among a set of treatments based on random samples from populations, two different approaches, cluster analysis and model selection, deal implicitly with such patterns, while traditional multiple testing procedures generally focus on the outcomes of subset and pairwise equality hypothesis tests, without considering the overall pattern of results in comparing methods. An important feature involving the pattern of rejections is their relevance for dividing the treatments into distinct subsets based on some parameter of interest, for example their means. This paper introduces some new measures relating to the potential of methods for achieving such divisions. Following Hartley (1955), sets of treatments with equal parameter values will be called clusters. Because it is necessary to distinguish between clusters in the populations and clustering in sample outcomes, the population clusters will be referred to as P -clusters; any related concepts defined in terms of the sample outcome will be referred to with the prefix outcome. Outcomes of multiple comparison procedures will be studied in terms of their probabilities of leading to separation of treatments into outcome clusters, with various measures relating to the number of such outcome clusters and the proportion of true vs. false outcome clusters. The definitions of true and false outcome clusters and related concepts, and the approach taken here, is in the tradition of hypothesis testing with attention to overall error control and power, but with added consideration of cluster separation potential.The pattern approach will be illustrated by comparing two methods with apparent FDR control but with different ways of ordering outcomes for potential significance: The original Benjamini-Hochberg (1995) procedure (BH), and the Newman-Keuls (Newman, 1939; Keuls, 1952) procedure (NK).     

Correcting for measurement error in individual ancestry estimates in structured association tests

We present theoretical explanations and show through simulation that the individual admixture proportion estimates obtained by using ancestry informative markers should be seen as an error-contaminated measurement of the underlying individual ancestry proportion. These estimates can be used in structured association tests as a control variable to limit type I error inflation or reduce loss of power due to population stratification observed in studies of admixed populations. However, the inclusion of such error-containing variables as covariates in regression models can bias parameter estimates and reduce ability to control for the confounding effect of admixture in genetic association tests. Measurement error correction methods offer a way to overcome this problem but require an a priori estimate of the measurement error variance. We show how an upper bound of this variance can be obtained, present four measurement error correction methods that are applicable to this problem, and conduct a simulation study to compare their utility in the case where the admixed population results from the intermating between two ancestral populations. Our results show that the quadratic measurement error correction (QMEC) method performs better than the other methods and maintains the type I error to its nominal level.     

Efficient measurement error correction with spatially misaligned data

Association studies in environmental statistics often involve exposure and outcome data that are misaligned in space. A common strategy is to employ a spatial model such as universal kriging to predict exposures at locations with outcome data and then estimate a regression parameter of interest using the predicted exposures. This results in measurement error because the predicted exposures do not correspond exactly to the true values. We characterize the measurement error by decomposing it into Berkson-like and classical-like components. One correction approach is the parametric bootstrap, which is effective but computationally intensive since it requires solving a nonlinear optimization problem for the exposure model parameters in each bootstrap sample. We propose a less computationally intensive alternative termed the "parameter bootstrap" that only requires solving one nonlinear optimization problem, and we also compare bootstrap methods to other recently proposed methods. We illustrate our methodology in simulations and with publicly available data from the Environmental Protection Agency.     

Comparative analysis of alternative statistical models for differentiation of herring stocks based on meristic characters

A basic stock assessment problem is the mixing and separation of herring populations in their specific areas. Within the transition zone between the North and the Baltic seas (Skagerrak, Kattegat, The Sound) a mixing of two herring populations ( Clupea harengus L.) temporarily takes place. One major component stems from the Baltic Sea (spring spawner), and the other from the North Sea (autumn spawner); each stock exhibits different meristic characteristics. In order to separate the two herring populations, 'pure' learning samples of meristic characters were created in 1995 as differential variables. Mainly gained during two research surveys at two geographically remote areas where a mixing of herring components was relatively unlikely, these learning samples were considered as representative for one or the other herring population. This paper compares the current (ICES) separation approach for herring stock assessment with two alternative methods wherein vertebra counts (vc) are used as meristic characters. The two proposed methods are (a) an inverted variance weighted linear model, and (b) a separation rule based on a quadratic discriminant analysis. The paper summarizes and discusses the results of predicting fractions of the two herring components derived from all three separation models. The underlying example data set stems from a 1991–97 routine Swedish survey in the transition zone. In comparison, the quadratic discriminant analysis separation model was identified as superior to the two other methods. Furthermore, the model suggests a higher degree of mixing of the two herring stocks in the transition area than was previously thought; inter-annual changes in the geographical distribution of the two populations are suggested as being less variable than previously assumed.     

Inferences on the common mean of several normal populations based on the generalized variable method

This article presents procedures for hypothesis testing and interval estimation of the common mean of several normal populations. The methods are based on the concepts of generalized p-value and generalized confidence limit. The merits of the proposed methods are evaluated numerically and compared with those of the existing methods. Numerical studies show that the new procedures are accurate and perform better than the existing methods when the sample sizes are moderate and the number of populations is four or less. If the number of populations is five or more, then the generalized variable method performs much better than the existing methods regardless of the sample sizes. The generalized variable method and other existing methods are illustrated using two examples.     

Comparative methods based on species mean values

Comparative methods that use simple linear regression based on species mean values introduce three difficulties with respect to the standard regression model. First, species values may not be independent because they form part of a hierarchically structured phylogeny. Second, variation about the regression line includes two sources of error: 'biological error' due to deviations of the true species mean values from the regression line and sampling error associated with the estimation of these mean values [B. Riska, Am. Natural. 138 (1991) 283]. Third, sampling error in the independent variable results in an attenuated estimate of the regression slope. We consider estimation and hypothesis testing using two statistical models which explicitly justify the use of the species mean values, without the need to account for phylogenetic relationships. The first (random-effects) is based on an evolutionary model whereby species evolve to fill a bivariate normal niche space, and the second (fixed-effects) is concerned with describing a relationship among the particular species included in a study, where the only source of error is in the estimation of species mean values. We use a modification of the maximum-likelihood method to obtain an unbiased estimate of the regression slope. For three real datasets we find a close correspondence between this slope and that obtained by simply regressing the species mean values on each other. In the random effects model, the P-value also approximates that based on the regression of species mean values. In the fixed effects model, the P-value is typically much lower. Simulated examples illustrate that the maximum-likelihood approach is useful when the accuracy in estimating the species mean values is low, but the traditional method based on a regression of the species mean values may often be justified provided that the evolutionary model can be justified.     

Stepwise Variable Selection in Classification Problems

Two variable selection procedures are evaluated for classification problems: a forward stepwise discrimination procedure, and a stepwise procedure preceded by a preliminary screening of variables on the basis of individual t statistics. Expected probability of correct classification is used as the measure of performance. A comparison is made of the procedures using samples from multi-variate normal populations and from several nonnormal populations. The study demonstrated some situations where the use of all variables is preferable to the use of a stepwise discriminant procedure stopping after a few steps, though usually the latter procedure was superior in performance. However where the stepwise procedure performed better than using all variables, the modified stepwise procedure performed still better. The use of modified stepwise procedures in which not all the covariances of the problem need be estimated seems promising.     

Semiparametric regression analysis of longitudinal data with informative drop-outs

Informative drop-out arises in longitudinal studies when the subject's follow-up time depends on the unobserved values of the response variable. We specify a semiparametric linear regression model for the repeatedly measured response variable and an accelerated failure time model for the time to informative drop-out. The error terms from the two models are assumed to have a common, but completely arbitrary joint distribution. Using a rank-based estimator for the accelerated failure time model and an artificial censoring device, we construct an asymptotically unbiased estimating function for the linear regression model. The resultant estimator is shown to be consistent and asymptotically normal. A resampling scheme is developed to estimate the limiting covariance matrix. Extensive simulation studies demonstrate that the proposed methods are suitable for practical use. Illustrations with data taken from two AIDS clinical trials are provided.     

A new method for dealing with measurement error in explanatory variables of regression models

We introduce a new method, moment reconstruction, of correcting for measurement error in covariates in regression models. The central idea is similar to regression calibration in that the values of the covariates that are measured with error are replaced by "adjusted" values. In regression calibration the adjusted value is the expectation of the true value conditional on the measured value. In moment reconstruction the adjusted value is the variance-preserving empirical Bayes estimate of the true value conditional on the outcome variable. The adjusted values thereby have the same first two moments and the same covariance with the outcome variable as the unobserved "true" covariate values. We show that moment reconstruction is equivalent to regression calibration in the case of linear regression, but leads to different results for logistic regression. For case-control studies with logistic regression and covariates that are normally distributed within cases and controls, we show that the resulting estimates of the regression coefficients are consistent. In simulations we demonstrate that for logistic regression, moment reconstruction carries less bias than regression calibration, and for case-control studies is superior in mean-square error to the standard regression calibration approach. Finally, we give an example of the use of moment reconstruction in linear discriminant analysis and a nonstandard problem where we wish to adjust a classification tree for measurement error in the explanatory variables.     

An evaluation of fitting methods for the sum of two hyperbolas: application to uptake studies

Analysis of data in terms of the sum of two rectangular hyperbolas is frequently required in solute uptake studies. Four methods for such analysis have been compared. Three are based on least-squares fitting whereas the fourth (partition method I) is an extension of a single hyperbola fitting procedure based on non-parametric statistics. The four methods were tested using data sets which had been generated with two primary types of random, normal error in the dependent variable: one of constant error variance and the other of constant coefficient of variation. The methods were tested on further data sets which were obtained by incorporating single 10% bias errors at different positions in the original two sets. Partition method I consistently gave good estimates for the four parameters defining the double hyperbola and was highly insensitive to the bias errors. The least-squares procedures performed well under conditions satisfying the least-squares assumptions regarding error distribution, but frequently gave poor estimates when these assumptions did not hold. Our conclusion is that in view of the errors inherent in many solute uptake experiments it would usually be preferable to analyse data by a method such as partition method I rather than to rely on a least-squares procedure.     

Approximation of bias and mean-squared error in two-sample Mendelian randomization analyses

Mendelian randomization (MR) is a type of instrumental variable (IV) analysis that uses genetic variants as IVs for a risk factor to study its causal effect on an outcome. Extensive investigations on the performance of IV analysis procedures, such as the one based on the two-stage least squares (2SLS) procedure, have been conducted under the one-sample scenario, where measures on IVs, the risk factor, and the outcome are assumed to be available for each study participant. Recent MR analysis usually is performed with data from two independent or partially overlapping genetic association studies (two-sample setting), with one providing information on the association between the IVs and the outcome, and the other on the association between the IVs and the risk factor. We investigate the performance of 2SLS in the two-sample–based MR when the IVs are weakly associated with the risk factor. We derive closed form formulas for the bias and mean squared error of the 2SLS estimate and verify them with numeric simulations under realistic circumstances. Using these analytic formulas, we can study the pros and cons of conducting MR analysis under one-sample and two-sample settings and assess the impact of having overlapping samples. We also propose and validate a bias-corrected estimator for the causal effect.     

A method for determining the position and size of optimal sequence regions for phylogenetic analysis

The availability of fast and accurate sequencing procedures along with the use of PCR has led to a proliferation of studies of variability at the molecular level in populations. Nevertheless, it is often impractical to examine long genomic stretches and a large number of individuals at the same time. In order to optimize this kind of study, we suggest a heuristic procedure for detection of the shortest region whose informational content can be considered sufficient for significant phylogenetic reconstruction. The method is based on the comparison of the pairwise genetic distances obtained from a set of sequences of reference to those obtained for different windows of variable size and position by means of a simple index. We also present an approach for testing whether the informative content in the stretches selected in this way is significantly different from the corresponding content shown by the larger genomic regions used as reference. Application of this test to the analysis of the VP1 protein gene of foot-and-mouth-disease type C virus allowed us to define optimal stretches whose informative content is not significantly different from that displayed by the complete VP1 sequence. We showed that the predictions made for type C sequences are valid for type O sequences, indicating that the results of the procedure are consistent. Correspondence to: J. Dopazo     

Notes on Testing Equality in Dichotomous Data with Matched Pairs

In attempting to improve the efficiency of McNemar's test statistic, we develop two test procedures that account for the information on both the discordant and concordant pairs for testing equality between two comparison groups in dichotomous data with matched pairs. Furthermore, we derive a test procedure derived from one of the most commonly‐used interval estimators for odds ratio. We compare these procedures with those using McNemar's test, McNemar's test with the continuity correction, and the exact test with respect to type I error and power in a variety of situations. We note that the test procedures using McNemar's test with the continuity correction and the exact test can be quite conservative and hence lose much efficiency, while the test procedure using McNemar's test can actually perform well even when the expected number of discordant pairs is small. We also find that the two test procedures, which incorporate the information on all matched pairs into hypothesis testing, may slightly improve the power of using McNemar's test without essentially losing the precision of type I error. On the other hand, the test procedure derived from an interval estimator of adds ratio with use of the logarithmic transformation may have type I error much larger than the nominal α‐level when the expected number of discordant pairs is not large and therefore, is not recommended for general use.     

A general method for controlling the genome-wide type I error rate in linkage and association mapping experiments in plants

Control of the genome-wide type I error rate (GWER) is an important issue in association mapping and linkage mapping experiments. For the latter, different approaches, such as permutation procedures or Bonferroni correction, were proposed. The permutation test, however, cannot account for population structure present in most association mapping populations. This can lead to false positive associations. The Bonferroni correction is applicable, but usually on the conservative side, because correlation of tests cannot be exploited. Therefore, a new approach is proposed, which controls the genome-wide error rate, while accounting for population structure. This approach is based on a simulation procedure that is equally applicable in a linkage and an association-mapping context. Using the parameter settings of three real data sets, it is shown that the procedure provides control of the GWER and the generalized genome-wide type I error rate (GWER(k)).     

Consistency of linkage disequilibrium between Chinese and Nordic Holsteins and genomic prediction for Chinese Holsteins using a joint reference population

Background

In China, the reference population of genotyped Holstein cattle is relatively small with to date, 80 bulls and 2091 cows genotyped with the Illumina 54 K chip. Including genotyped Holstein cattle from other countries in the reference population could improve the accuracy of genomic prediction of the Chinese Holstein population. This study investigated the consistency of linkage disequilibrium between adjacent markers between the Chinese and Nordic Holstein populations, and compared the reliability of genomic predictions based on the Chinese reference population only or the combined Chinese and Nordic reference populations.

Methods

Genomic estimated breeding values of Chinese Holstein cattle were predicted using a single-trait GBLUP model based on the Chinese reference dataset, and using a two-trait GBLUP model based on a joint reference dataset that included both the Chinese and Nordic Holstein data.

Results

The extent of linkage disequilibrium was similar in the Chinese and Nordic Holstein populations and the consistency of linkage disequilibrium between the two populations was very high, with a correlation of 0.97. Genomic prediction using the joint versus the Chinese reference dataset increased reliabilities of genomic predictions of Chinese Holstein bulls in the test data from 0.22, 0.15 and 0.11 to 0.51, 0.47 and 0.36 for milk yield, fat yield and protein yield, respectively. Using five-fold cross-validation, reliabilities of genomic predictions of Chinese cows increased from 0.15, 0.12 and 0.15 to 0.26, 0.17 and 0.20 for milk yield, fat yield and protein yield, respectively.

Conclusions

The linkage disequilibrium between the two populations was very consistent and using the combined Nordic and Chinese reference dataset substantially increased reliabilities of genomic predictions for Chinese Holstein cattle.     

Simultaneous Non‐inferiority Test of Sensitivity and Specificity for Two Diagnostic Procedures in the Presence of a Gold Standard

Sensitivity and specificity have traditionally been used to assess the performance of a diagnostic procedure. Diagnostic procedures with both high sensitivity and high specificity are desirable, but these procedures are frequently too expensive, hazardous, and/or difficult to operate. A less sophisticated procedure may be preferred, if the loss of the sensitivity or specificity is determined to be clinically acceptable. This paper addresses the problem of simultaneous testing of sensitivity and specificity for an alternative test procedure with a reference test procedure when a gold standard is present. The hypothesis is formulated as a compound hypothesis of two non‐inferiority (one‐sided equivalence) tests. We present an asymptotic test statistic based on the restricted maximum likelihood estimate in the framework of comparing two correlated proportions under the prospective and retrospective sampling designs. The sample size and power of an asymptotic test statistic are derived. The actual type I error and power are calculated by enumerating the exact probabilities in the rejection region. For applications that require high sensitivity as well as high specificity, a large number of positive subjects and a large number of negative subjects are needed. We also propose a weighted sum statistic as an alternative test by comparing a combined measure of sensitivity and specificity of the two procedures. The sample size determination is independent of the sampling plan for the two tests.     

Genetic variation in original and colonizing Drosophila buzzatii populations analysed by microsatellite loci isolated with a new PCR screening method

A new polymerase chain reaction-based screening method for microsatellites is presented. Using this method, we isolated 12 microsatellite loci from Drosophila buzzatii, two of which were X-linked. We applied the other 10 microsatellite loci to the analysis of genetic variation in five natural populations of D. buzzatii. Two populations were from the species' original distribution in Argentina, whereas the other three were from Europe (two) and Australia that were colonized 200 and 65 years ago, respectively. Allelic variation was much larger in the original populations than in the colonizing ones and there was a tendency to decreased heterozygosity in the colonizing populations. We used three different statistical procedures for detecting population bottlenecks. All procedures suggested that the low variability in the populations in the Old World was not the result of the recent population decline, but was due to a founder effect followed by a population expansion. In fact, one procedure which detects population expansions and declines based on the genealogical history of microsatellite data suggested that an expansion had taken place in all the colonized populations.     

On the False Discovery Rate and Expected Type I Errors

The paper is concerned with expected type I errors of some stepwise multiple test procedures based on independent p‐values controlling the so‐called false discovery rate (FDR). We derive an asymptotic result for the supremum of the expected type I error rate(EER) when the number of hypotheses tends to infinity. Among others, it will be shown that when the original Benjamini‐Hochberg step‐up procedure controls the FDR at level α, its EER may approach a value being slightly larger than α/4 when the number of hypotheses increases. Moreover, we derive some least favourable parameter configuration results, some bounds for the FDR and the EER as well as easily computable formulae for the familywise error rate (FWER) of two FDR‐controlling procedures. Finally, we discuss some undesirable properties of the FDR concept, especially the problem of cheating.     

The important choice of reference environment in microevolutionary climate response predictions

It is well documented that individuals of wild populations can adjust to climate change by means of phenotypic plasticity, but few reports on adaptation by means of genetically based microevolution caused by selection. Disentanglement of these separate effects requires that the reference environment (the environmental zero point) is defined, and this should not be done arbitrarily. The problem is that an error in the reference environment may lead to large errors in predicted microevolution. Together with parameter values and initial mean trait values, the reference environment can be estimated from environmental, phenotypic and fitness data. A prediction error method for this purpose is described, with the feasibility shown by simulations. As shown in a toy example, an estimated reference environment may have large errors, especially for small populations. This may still be a better choice than use of an initial environmental value in a recorded time series, or the mean value, which is often used. Another alternative may be to use the mean value of a past and stationary stochastic environment, which the population is judged to have been fully adapted to, in the sense that the expected geometric mean fitness was at a global maximum. Exceptions are cases with constant phenotypic plasticity, where the microevolutionary changes per generation follow directly from phenotypic and environmental data, independent of the chosen reference environment.     

Improved two‐stage group sequential procedures for testing a secondary endpoint after the primary endpoint achieves significance

In two‐stage group sequential trials with a primary and a secondary endpoint, the overall type I error rate for the primary endpoint is often controlled by an α‐level boundary, such as an O'Brien‐Fleming or Pocock boundary. Following a hierarchical testing sequence, the secondary endpoint is tested only if the primary endpoint achieves statistical significance either at an interim analysis or at the final analysis. To control the type I error rate for the secondary endpoint, this is tested using a Bonferroni procedure or any α‐level group sequential method. In comparison with marginal testing, there is an overall power loss for the test of the secondary endpoint since a claim of a positive result depends on the significance of the primary endpoint in the hierarchical testing sequence. We propose two group sequential testing procedures with improved secondary power: the improved Bonferroni procedure and the improved Pocock procedure. The proposed procedures use the correlation between the interim and final statistics for the secondary endpoint while applying graphical approaches to transfer the significance level from the primary endpoint to the secondary endpoint. The procedures control the familywise error rate (FWER) strongly by construction and this is confirmed via simulation. We also compare the proposed procedures with other commonly used group sequential procedures in terms of control of the FWER and the power of rejecting the secondary hypothesis. An example is provided to illustrate the procedures.