A Generalization of the Intraclass Tau Correlation for Tied and Censored Data

Intraclass tau correlations and a corresponding asymptotic permutation test have already been derived by WHITFIELD (1949), though for continuous data only. WHITFIELD'S concepts are now generalized for use with tied and censored data and a suggestion is made for the application of the methods to arbitrary and varying group sizes. Special attention is given to their application to dichotomous data. Extensive simulations confirming the safe use of the asymptotic test even for samples of moderate size are reported and calculated examples with data from twin-births and kidney transplantations presented.     

A permutation test to compare receiver operating characteristic curves

We developed a permutation test in our earlier paper (Venkatraman and Begg, 1996, Biometrika 83, 835-848) to test the equality of receiver operating characteristic curves based on continuous paired data. Here we extend the underlying concepts to develop a permutation test for continuous unpaired data, and we study its properties through simulations.     

Precision Intervals for Estimates of the Difference in Success Rates for Binary Random Variables Based on the Permutation Principle

Fisher's exact test is a very commonly applied test in clinical trials with a binary outcome variable (e.g. success/failure). However confidence statements about the difference of success rates are usually based on the normal approximation. This may sometimes lead to the confusing statement that the test is statistically significant at a prespecified level while the corresponding confidence interval includes the zero difference and vice versa. Here, we construct precision intervals for the difference of success rates from two independent samples based on the permutation principle which are in perfect agreement with the discrete (permutation) test and compare it to examples from the literature. APL programs are provided.     

Comparing Two Samples of Response Curves by Nonparamètric Tests of Predicted Order

Starting from MOSTELLER'S (1955) one-sample test of predicted order, some two-sample versions are proposed for comparing nonparametrically a treatment group (m response curves) with a control group (n response curves) as to a predicted shape represented by a specified rank order. The tests are applied to two groups of learning curves.     

Exact,Distribution Free Confidence Intervals for Late Effects in Censored Matched Pairs

When comparing censored survival times for matched treated and control subjects, a late effect on survival is one that does not begin to appear until some time has passed. In a study of provider specialty in the treatment of ovarian cancer, a late divergence in the Kaplan–Meier survival curves hinted at superior survival among patients of gynecological oncologists, who employ chemotherapy less intensively, when compared to patients of medical oncologists, who employ chemotherapy more intensively; we ask whether this late divergence should be taken seriously. Specifically, we develop exact, permutation tests, and exact confidence intervals formed by inverting the tests, for late effects in matched pairs subject to random but heterogeneous censoring. Unlike other exact confidence intervals with censored data, the proposed intervals do not require knowledge of censoring times for patients who die. Exact distributions are consequences of two results about signs, signed ranks, and their conditional independence properties. One test, the late effects sign test, has the binomial distribution; the other, the late effects signed rank test, uses nonstandard ranks but nonetheless has the same exact distribution as Wilcoxon's signed rank test. A simulation shows that the late effects signed rank test has substantially more power to detect late effects than do conventional tests. The confidence statement provides information about both the timing and magnitude of late effects (© 2009 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Independent contrasts and regression through the origin

Following the pioneering work of Felsenstein and Garland, phylogeneticists have been using regression through the origin to analyze comparative data using independent contrasts. The reason why regression through the origin must be used with such data was revisited. The demonstration led to the formulation of a permutation test for the coefficient of determination and the regression coefficient estimates in regression through the origin. Simulations were carried out to measure type I error and power of the parametric and permutation tests under two models of data generation: regression models I and II (correlation model). Although regression through the origin assumes model I data, in independent contrast data error is present in the explanatory as well as the response variables. Two forms of permutations were investigated to test the regression coefficients: permutation of the values of the response variable y, and permutation of the residuals of the regression model. The simulations showed that the parametric tests or any of the permutation tests can be used when the error is normal, which is the usual assumption in independent contrast studies; only the test by permutation of y should be used when the error is highly asymmetric; and the parametric tests should be used when extreme values are present in covariables. Two examples are presented. The first one concerns non-specificity in fish parasites of the genus Lamellodiscus, the second the richness in parasites in 78 species of mammals.     

Distribution-free R sample tests for the hypothesis of parallelism of response profiles

In this paper two nonparametric tests are given for testing the nullhypothesis of parallelity of response curves in r samples. The first procedure is done by a permutation test, whose practical applicability is ensured by a FORTRAN-subroutine available from the author. As the computational work and time grows rapidly with sample sizes, a Monte-Carlo-solution is optionally given by the same procedure. Alternatively an asymptotically distribution-free test based on quadratic forms is constructed for large sample situations.     

Simulation‐based evaluation of the performance of the F test in a linear multilevel model setting with sparseness at the level of the primary unit

In a linear multilevel model, significance of all fixed effects can be determined using F tests under maximum likelihood (ML) or restricted maximum likelihood (REML). In this paper, we demonstrate that in the presence of primary unit sparseness, the performance of the F test under both REML and ML is rather poor. Using simulations based on the structure of a data example on ceftriaxone consumption in hospitalized children, we studied variability, type I error rate and power in scenarios with a varying number of secondary units within the primary units. In general, the variability in the estimates for the effect of the primary unit decreased as the number of secondary units increased. In the presence of singletons (i.e., only one secondary unit within a primary unit), REML consistently outperformed ML, although even under REML the performance of the F test was found inadequate. When modeling the primary unit as a random effect, the power was lower while the type I error rate was unstable. The options of dropping, regrouping, or splitting the singletons could solve either the problem of a high type I error rate or a low power, while worsening the other. The permutation test appeared to be a valid alternative as it outperformed the F test, especially under REML. We conclude that in the presence of singletons, one should be careful in using the F test to determine the significance of the fixed effects, and propose the permutation test (under REML) as an alternative.     

A modified sign test for comparing paired ROC curves

We develop a permutation test for assessing a difference in the areas under the curve (AUCs) in a paired setting where both modalities are given to each diseased and nondiseased subject. We propose that permutations be made between subjects specifically by shuffling the diseased/nondiseased labels of the subjects within each modality. As these permutations are made within modality, the permutation test is valid even if both modalities are measured on different scales. We show that our permutation test is a sign test for the symmetry of an underlying discrete distribution whose size remains valid under the assumption of equal AUCs. We demonstrate the operating characteristics of our test via simulation and show that our test is equal in power to a permutation test recently proposed by Bandos and others (2005).     

A New Nonparametric Approach to the Comparison of K Independent Samples of Response Curves II: A K Sample Generalization of the FRIEDMAN Test

A K sample generalization of the FRIEDMAN test (1937) is introduced which can be used as a nonparametric procedure for testing the homogeneity of the profiles of K independent samples of response curves measured at T identical points of time. While a similar procedure in LEHMACHER & WALL (1978), section 3, is based on T combined tests, each of them at level a/T, here a finite and asymptotic test is presented which is based on a single test statistic. The application of the new multivariate test is illustrated by the same numerical example as in LEHMACHER & WALL (1978). The properties of this test are discussed and compared with the combined test mentioned above.     

A note on controlling the number of false positives

Summary :   Lehmann and Romano (2005, Annals of Statistics 33, 1138–1154) discuss a Bonferroni-type procedure that bounds the probability that the number of false positives is larger than a specified number. We note that this procedure will have poor power as compared to a multivariate permutation test type procedure when the experimental design accommodates a permutation test. An example is given involving gene expression microarray data of breast cancer tumors.     

The Statistical Significance of Mutation Frequencies

TRAUT has proposed a method for determining the statistical significance between a mutation frequency observed after treatment with a potential mutagen and a control frequency. The method is developed further by us for an easier practical use. A table for the minimal sample size in the region of interest for the sex-linked recessive lethal test in Drosophila melanogaster has been calculated. TRACT'S test is compared with FISHER'S exact test and the KASTENBAUM -BOWMAN test. TRAUT'S test turns out to be more sensitive than the other tests. This observation strongly supports TRAUT'S conclusion that his test should only be used if very accurate determinations of the spontaneous frequencies are available.     

Nonparametric Tests for Trend: Jonckheere's Test,a Modification and a Maximum Test

Jonckheere's test is a frequently used nonparametric trend test for the evaluation of preclinical studies and clinical dose-finding trials. In this paper, a modification of Jonckheere's test is proposed. If the exact permutation distribution is used for inference, the modified test can fill out the level of the type I error in a much more complete way and is substantially more powerful than the common Jonckheere test. If the asymptotic normality is used for inference, the modified test is slightly more powerful. In addition, a maximum test is investigated which is more robust concerning an a priori unknown dose-response shape. The robustness is advantageous, especially in a closed testing procedure. The different tests are applied to two example data sets.     

A class of permutation tests for stratified survival data

We propose a class of permutation tests for stratified survival data. The tests are derived using the framework of Fay and Shih (1998, Journal of the American Statistical Association 93, 387-396), which creates tests by permuting scores based on a functional of estimated distribution functions. Here the estimated distribution function for each possibly right-, left-, or interval-censored observation is based on a shrinkage estimator similar to the nonparametric empirical estimator of Ghosh, Lahiri, and Tiwari (1989, Communications in Statistics--Theory and Methods 18, 121-146), and permutation is carried out within strata. The proposed test with a weighted Mann-Whitney functional is similar to the permutation form of the stratified log-rank test when there is a large strata effect or the sample size in each stratum is large and is similar to the permutation form of the ordinary log-rank test when there is little strata effect. Thus, the proposed test unifies the advantages of both the stratified and ordinary log-rank tests. By changing the functional, we may obtain a stratified Prentice-Wilcoxon test or a difference in means test with similar unifying properties. We show through simulations the advantage of the proposed test over existing tests for uncensored and right-censored data.     

Application of a Combination of a Knowledge-Based Algorithm and 2-Stage Screening to Hypothesis-Free Genomic Data on Irinotecan-Treated Patients for Identification of a Candidate Single Nucleotide Polymorphism Related to an Adverse Effect

Interindividual variation in a drug response among patients is known to cause serious problems in medicine. Genomic information has been proposed as the basis for “personalized” health care. The genome-wide association study (GWAS) is a powerful technique for examining single nucleotide polymorphisms (SNPs) and their relationship with drug response variation; however, when using only GWAS, it often happens that no useful SNPs are identified due to multiple testing problems. Therefore, in a previous study, we proposed a combined method consisting of a knowledge-based algorithm, 2 stages of screening, and a permutation test for identifying SNPs. In the present study, we applied this method to a pharmacogenomics study where 109,365 SNPs were genotyped using Illumina Human-1 BeadChip in 168 cancer patients treated with irinotecan chemotherapy. We identified the SNP rs9351963 in potassium voltage-gated channel subfamily KQT member 5 (KCNQ5) as a candidate factor related to incidence of irinotecan-induced diarrhea. The p value for rs9351963 was 3.31×10⁻⁵ in Fisher''s exact test and 0.0289 in the permutation test (when multiple testing problems were corrected). Additionally, rs9351963 was clearly superior to the clinical parameters and the model involving rs9351963 showed sensitivity of 77.8% and specificity of 57.6% in the evaluation by means of logistic regression. Recent studies showed that KCNQ4 and KCNQ5 genes encode members of the M channel expressed in gastrointestinal smooth muscle and suggested that these genes are associated with irritable bowel syndrome and similar peristalsis diseases. These results suggest that rs9351963 in KCNQ5 is a possible predictive factor of incidence of diarrhea in cancer patients treated with irinotecan chemotherapy and for selecting chemotherapy regimens, such as irinotecan alone or a combination of irinotecan with a KCNQ5 opener. Nonetheless, clinical importance of rs9351963 should be further elucidated.     

Minimize the use of minimization with unequal allocation

Summary Minimization as an alternative to randomization is gaining popularity for small clinical trials. In response to critics’ questions about the proper analysis of such a trial, proponents have argued that a rerandomization approach, akin to a permutation test with conventional randomization, can be used. However, they add that this computationally intensive approach is not necessary because its results are very similar to those of a t ‐test or test of proportions unless the sample size is very small. We show that minimization applied with unequal allocation causes problems that challenge this conventional wisdom.     

Rotation gene set testing for longitudinal expression data

Gene set analysis methods are popular tools for identifying differentially expressed gene sets in microarray data. Most existing methods use a permutation test to assess significance for each gene set. The permutation test's assumption of exchangeable samples is often not satisfied for time‐series data and complex experimental designs, and in addition it requires a certain number of samples to compute p‐values accurately. The method presented here uses a rotation test rather than a permutation test to assess significance. The rotation test can compute accurate p‐values also for very small sample sizes. The method can handle complex designs and is particularly suited for longitudinal microarray data where the samples may have complex correlation structures. Dependencies between genes, modeled with the use of gene networks, are incorporated in the estimation of correlations between samples. In addition, the method can test for both gene sets that are differentially expressed and gene sets that show strong time trends. We show on simulated longitudinal data that the ability to identify important gene sets may be improved by taking the correlation structure between samples into account. Applied to real data, the method identifies both gene sets with constant expression and gene sets with strong time trends.     

A remark on a paper by misra

MISRA (1978) sets confidence intervals for a double linear compound of multivariate normal regression coefficients by using ROY'S maximum root test criterion. The exact test statistic to be used is STUDENT'S t. The t statistic gives narrower confidence bounds than those given by ROY's maximum root statistic. A result given by MORRISON (1975, p. 18, equation 10) for profile analysis is also obtained by using the STUDENT'S t test.     

Accounting for variability in the use of permutation testing to detect quantitative trait loci

Locating quantitative trait loci (QTL), or genomic regions associated with known molecular markers, is of increasing interest in a wide variety of applications ranging from human genetics to agricultural genetics. The hope of locating QTL (or genes) affecting a quantitative trait is that it will lead to characterization and possible manipulations of these genes. However, the complexity of both statistical and genetic issues surrounding the location of these regions calls into question the asymptotic statistical results supplying the distribution of the test statistics employed. Coupled with the power of current-day computing, permutation theory was reintroduced for the purpose of estimating the distribution of any test statistic used to test for the location of QTL. Permutation techniques have offered an attractive alternative to significance measures based on asymptotic theory. The ideas of permutation testing are extended in this application to include confidence intervals for the thresholds and p-values estimated in permutation testing procedures. The confidence intervals developed account for the Monte Carlo error associated with practical applications of permutation testing and lead to an effective method of determining an efficient permutation sample size.     

identix,a software to test for relatedness in a population using permutation methods

The computer program identix estimates relatedness in natural populations using multilocus genotypic data. Queller & Goodnight's (1989) and Lynch & Ritland's (1999) estimators of pairwise relatedness are implemented, as well as the identity index of Mathieu et al. (1990). Estimates of the confidence intervals around these pairwise values are also provided. The null hypothesis of no relatedness (multilocus genotypes are independent draws from a panmictic population) is tested using a permutation method that compares the observed distribution of the moments of pairwise relatedness coefficients to that expected in unstructured population.