A Two Sample CRAMÉR-VON MISES Test for Randomly Censored Data

A CRAMÉR-VON MISES type statistic is introduced for testing the equality of the underlying survival distributions of two populations when observations are subject to arbitrary right censorship. The statistic is appropriate in testing problems where a two-sided alternative is of interest. The asymptotic distribution of the statistic is found; under certain circumstances, the limiting distribution coincides with that of a one sample CRAMÉR-VON MISES type statistic for randomly censored data investigated previously. Approximations to the asymptotic distribution are discussed; an example is given.     

On Testing Equality of Means of Correlated Variates with Missing Data on One Response

A statistic is proposed for testing the hypothesis of equality of the means of a bivariate normal distribution with unknown common variance and correlation coefficient when observations are missing on one of the variates. The distribution of the statistic is approximated by a normal distribution under the null hypothesis. The empirical powers of the statistic are computed and compared with those of the conventional paired t and the other known statistics. The power comparisons support the use of the proposed test.     

Testing Equality of Means in the Presence of Correlation and Missing Data

A statistic, derived from the combination of two dependent tests, is proposed for testing the hypothesis of equality of the means of a bivariate normal distribution with unknown common variance and correlation coefficient when observations are missing on one or both variates. The null distribution of the statistic is approximated by a well-known distribution. The empirical powers of the statistic are computed and compared with some of the known statistics. The comparisons support the use of the proposed test.     

Valid inference in random effects meta-analysis

The standard approach to inference for random effects meta-analysis relies on approximating the null distribution of a test statistic by a standard normal distribution. This approximation is asymptotic on k, the number of studies, and can be substantially in error in medical meta-analyses, which often have only a few studies. This paper proposes permutation and ad hoc methods for testing with the random effects model. Under the group permutation method, we randomly switch the treatment and control group labels in each trial. This idea is similar to using a permutation distribution for a community intervention trial where communities are randomized in pairs. The permutation method theoretically controls the type I error rate for typical meta-analyses scenarios. We also suggest two ad hoc procedures. Our first suggestion is to use a t-reference distribution with k-1 degrees of freedom rather than a standard normal distribution for the usual random effects test statistic. We also investigate the use of a simple t-statistic on the reported treatment effects.     

Improving data interpretation of fragmentary data-sets on invertebrate dispersal with permutation tests

Biological data often tend to have heterogeneous, discontinuous non-normal distributions. Statistical non-parametric tests, like the Mann–Whitney U-test or the extension for more than two samples, the Kruskal–Wallis test, are often used in these cases, although they assume certain preconditions which are often ignored. We developed a permutation test procedure that uses the ratio of the interquartile distances and the median differences of the original non-classified data to assess the properties of the real distribution more appropriately than the classical methods. We used this test on a heterogeneous, skewed biological data set on invertebrate dispersal and showed how different the reactions of the Kruskal–Wallis test and the permutation approach are. We then evaluated the new testing procedure with reproducible data that were generated from the normal distribution. Here, we tested the influence of four different experimental trials on the new testing procedure in comparison to the Kruskal–Wallis test. These trials showed the impact of data that were varying in terms of (a) negative correlation between variances and means of the samples, (b) changing variances that were not correlated with the means of the samples, (c) constant variances and means, but different sample sizes and in trials (d) we evaluated the testing power of the new procedure. Due to the different test statistics, the permutation test reacted more sensibly to the data presented in trials (a) and c) and non-uniformly in trial (b). In the evaluation of the testing power, no significant differences between the Kruskal–Wallis test and the new permutation testing procedure could be detected. We consider this test to be an alternative for working on heterogeneous data where the preconditions of the classical non-parametric tests are not met.     

A multivariate two-sample mean test for small sample size and missing data

We develop a new statistic for testing the equality of two multivariate mean vectors. A scaled chi-squared distribution is proposed as an approximating null distribution. Because the test statistic is based on componentwise statistics, it has the advantage over Hotelling's T2 test of being applicable to the case where the dimension of an observation exceeds the number of observations. An appealing feature of the new test is its ability to handle missing data by relying on only componentwise sample moments. Monte Carlo studies indicate good power compared to Hotelling's T2 and a recently proposed test by Srivastava (2004, Technical Report, University of Toronto). The test is applied to drug discovery data.     

A Test of Equality of Survival Distributions for Two Sample Censored Grouped Survival Data

A common testing problem for a life table or survival data is to test the equality of two survival distributions when the data is both grouped and censored. Several tests have been proposed in the literature which require various assumptions about the censoring distributions. It is shown that if these conditions are relaxed then the tests may no longer have the stated properties. The maximum likelihood test of equality when no assumptions are made about the censoring marginal distributions is derived. The properties of the test are found and it is compared to the existing tests. The fact that no assumptions are required about the censoring distributions make the test a useful initial testing procedure.     

Probability binning comparison: a metric for quantitating multivariate distribution differences

BACKGROUND: While several algorithms for the comparison of univariate distributions arising from flow cytometric analyses have been developed and studied for many years, algorithms for comparing multivariate distributions remain elusive. Such algorithms could be useful for comparing differences between samples based on several independent measurements, rather than differences based on any single measurement. It is conceivable that distributions could be completely distinct in multivariate space, but unresolvable in any combination of univariate histograms. Multivariate comparisons could also be useful for providing feedback about instrument stability, when only subtle changes in measurements are occurring. METHODS: We apply a variant of Probability Binning, described in the accompanying article, to multidimensional data. In this approach, hyper-rectangles of n dimensions (where n is the number of measurements being compared) comprise the bins used for the chi-squared statistic. These hyper-dimensional bins are constructed such that the control sample has the same number of events in each bin; the bins are then applied to the test samples for chi-squared calculations. RESULTS: Using a Monte-Carlo simulation, we determined the distribution of chi-squared values obtained by comparing sets of events from the same distribution; this distribution of chi-squared values was identical as for the univariate algorithm. Hence, the same formulae can be used to construct a metric, analogous to a t-score, that estimates the probability with which distributions are distinct. As for univariate comparisons, this metric scales with the difference between two distributions, and can be used to rank samples according to similarity to a control. We apply the algorithm to multivariate immunophenotyping data, and demonstrate that it can be used to discriminate distinct samples and to rank samples according to a biologically-meaningful difference. CONCLUSION: Probability binning, as shown here, provides a useful metric for determining the probability with which two or more multivariate distributions represent distinct sets of data. The metric can be used to identify the similarity or dissimilarity of samples. Finally, as demonstrated in the accompanying paper, the algorithm can be used to gate on events in one sample that are different from a control sample, even if those events cannot be distinguished on the basis of any combination of univariate or bivariate displays. Published 2001 Wiley-Liss, Inc.     

Probability binning and testing agreement between multivariate immunofluorescence histograms: extending the chi-squared test

BACKGROUND: A key problem in immunohistochemistry is assessing when two sample histograms are significantly different. One test that is commonly used for this purpose in the univariate case is the chi-squared test. Comparing multivariate distributions is qualitatively harder, as the "curse of dimensionality" means that the number of bins can grow exponentially. For the chi-squared test to be useful, data-dependent binning methods must be employed. An example of how this can be done is provided by the "probability binning" method of Roederer et al. (1,2,3). METHODS: We derive the theoretical distribution of the probability binning statistic, giving it a more rigorous foundation. We show that the null distribution is a scaled chi-square, and show how it can be related to the standard chi-squared statistic. RESULTS: A small simulation shows how the theoretical results can be used to (a) modify the probability binning statistic to make it more sensitive and (b) suggest variant statistics which, while still exploiting the data-dependent strengths of the probability binning procedure, may be easier to work with. CONCLUSIONS: The probability binning procedure effectively uses adaptive binning to locate structure in high-dimensional data. The derivation of a theoretical basis provides a more detailed interpretation of its behavior and renders the probability binning method more flexible.     

Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.     

Partitioning to uncover conditions for permutation tests to control multiple testing error rates

This article discusses specific assumptions necessary for permutation multiple tests to control the Familywise Error Rate (FWER). At issue is that, in comparing parameters of the marginal distributions of two sets of multivariate observations, validity of permutation testing is affected by all the parameters in the joint distributions of the observations. We show the surprising fact that, in the case of a linear model with i.i.d. errors such as in the analysis of Quantitative Trait Loci (QTL), this issue has no impact on control of FWER, if the test statistic is of a particular form. On the other hand, in the analysis of gene expression levels or multiple safety endpoints, unless some assumption connecting the marginal distributions of the observations to their joint distributions is made, permutation multiple tests may not control FWER.     

Testing Equality of Correlated Means in the Presence of Unequal Variances and Missing Values

Several statistics are proposed for testing the hypothesis of equality of the means of bivariate normal distribution with unknown variances and correlation coefficient when observations are missing on both variatea. The null distributions of the statistics are approximated by well-known distributions. The empirical sizes and powers of the statistics are computed and compared with paired t test and some of the known statistics based on available data. The comparisons support the use of two of the statistics proposed in this paper.     

Accounting for variability in the use of permutation testing to detect quantitative trait loci

Locating quantitative trait loci (QTL), or genomic regions associated with known molecular markers, is of increasing interest in a wide variety of applications ranging from human genetics to agricultural genetics. The hope of locating QTL (or genes) affecting a quantitative trait is that it will lead to characterization and possible manipulations of these genes. However, the complexity of both statistical and genetic issues surrounding the location of these regions calls into question the asymptotic statistical results supplying the distribution of the test statistics employed. Coupled with the power of current-day computing, permutation theory was reintroduced for the purpose of estimating the distribution of any test statistic used to test for the location of QTL. Permutation techniques have offered an attractive alternative to significance measures based on asymptotic theory. The ideas of permutation testing are extended in this application to include confidence intervals for the thresholds and p-values estimated in permutation testing procedures. The confidence intervals developed account for the Monte Carlo error associated with practical applications of permutation testing and lead to an effective method of determining an efficient permutation sample size.     

Tests for Homogeneity of Variances Using Robust Weighted Likelihood Estimates

The classical normal-theory tests for testing the null hypothesis of common variance and the classical estimates of scale have long been known to be quite nonrobust to even mild deviations from normality assumptions for moderate sample sizes. Levene (1960) suggested a one-way ANOVA type statistic as a robust test. Brown and Forsythe (1974) considered a modified version of Levene's test by replacing the sample means with sample medians as estimates of population locations, and their test is computationally the simplest among the three tests recommended by Conover , Johnson , and Johnson (1981) in terms of robustness and power. In this paper a new robust and powerful test for homogeneity of variances is proposed based on a modification of Levene's test using the weighted likelihood estimates (Markatou , Basu , and Lindsay , 1996) of the population means. For two and three populations the proposed test using the Hellinger distance based weighted likelihood estimates is observed to achieve better empirical level and power than Brown-Forsythe's test in symmetric distributions having a thicker tail than the normal, and higher empirical power in skew distributions under the use of F distribution critical values.     

A Modified Chi-Square Test for Testing Equality of Two Multinomial Populations Against an Ordering Restricted Alternative

A modified chi-square test for testing the equality of two multinomial populations against an ordering restricted alternative in one sample and two sample cases is constructed. The relation between a concept of dependence called dependence by chi-square and stochastic ordering is established. A tabulation of the asymptotic distribution of the test statistic under the null hypothesis is given. Simulations are used to compare the power of this test with the power of the likelihood ratio test of stochastic ordering of the two multinomial populations.     

On Testing Equality of Means of Correlated Variates with Incomplete Data

A statistic is proposed for testing the hypothesis of equality of the means of a bivariate normal distribution with unknown common variance and correlation coefficient when observations are missing on one of the variates. Expressions for the second and fourth central moments of the statistic are obtained. These moments are used to approximate the distribution of the statistic by a Student's t distribution under the null hypothesis. The powers of the test are computed and compared with those of the conventional paired t and the other known statistics.     

Approximating the distribution of maximally selected McNemar's statistics

It is common in epidemiologic analyses to summarize continuous outcomes as falling above or below a threshold. With paired data and with a threshold chosen without reference to the outcomes, McNemar's test of marginal homogeneity may be applied to the resulting dichotomous pairs when testing for equality of the marginal distributions of the underlying continuous outcomes. If the threshold is chosen to maximize the test statistic, however, referring the resulting test statistic to the nominal chi 2 distribution is incorrect; instead, the p-value must be adjusted for the multiple comparisons. Here the distribution of a maximally selected McNemar's statistic is derived, and it is shown that an approximation due to Durbin (1985, Journal of Applied Probability 22, 99-122) may be used to estimate approximate p-values. The methodology is illustrated by an application to measurements of insulin-like growth factor-I (IGF-I) in matched prostate cancer cases and controls from the Physicians' Health Study. The results of simulation experiments that assess the accuracy of the approximation in moderate sample sizes are reported.     

Probability binning comparison: a metric for quantitating univariate distribution differences

BACKGROUND: Comparing distributions of data is an important goal in many applications. For example, determining whether two samples (e.g., a control and test sample) are statistically significantly different is useful to detect a response, or to provide feedback regarding instrument stability by detecting when collected data varies significantly over time. METHODS: We apply a variant of the chi-squared statistic to comparing univariate distributions. In this variant, a control distribution is divided such that an equal number of events fall into each of the divisions, or bins. This approach is thereby a mini-max algorithm, in that it minimizes the maximum expected variance for the control distribution. The control-derived bins are then applied to test sample distributions, and a normalized chi-squared value is computed. We term this algorithm Probability Binning. RESULTS: Using a Monte-Carlo simulation, we determined the distribution of chi-squared values obtained by comparing sets of events derived from the same distribution. Based on this distribution, we derive a conversion of any given chi-squared value into a metric that is analogous to a t-score, i.e., it can be used to estimate the probability that a test distribution is different from a control distribution. We demonstrate that this metric scales with the difference between two distributions, and can be used to rank samples according to similarity to a control. Finally, we demonstrate the applicability of this metric to ranking immunophenotyping distributions to suggest that it indeed can be used to objectively determine the relative distance of distributions compared to a single control. CONCLUSION: Probability Binning, as shown here, provides a useful metric for determining the probability that two or more flow cytometric data distributions are different. This metric can also be used to rank distributions to identify which are most similar or dissimilar. In addition, the algorithm can be used to quantitate contamination of even highly-overlapping populations. Finally, as demonstrated in an accompanying paper, Probability Binning can be used to gate on events that represent significantly different subsets from a control sample. Published 2001 Wiley-Liss, Inc.     

A Robust Test for Multi-Sample Location Problems with Unequal Group Variances and Nonnormality

A robust test (to be referred to as M^* test) is proposed for testing equality of several group means without assuming normality and equality of variances. This test statistic is obtained by combining Tiku's MML robust procedure with the James statistic. Monte Carlo simulation studies indicate that the M^* test is more powerful than the Welch test, the James test, and the tests based on Huber's M-estimators over a wide range of nonnormal universes. It is also more powerful than the Brown and Forsythe test under most of nonnormal distributions and has substantially the same power as the Brown and Forsythe test under normal distribution. Comparing with Tan-Tabatabai test, M^* is almost as powerful as Tan-Tabatabai test.     

Likelihood analysis for the ratio of means of two independent log-normal distributions

Existing methods for comparing the means of two independent skewed log-normal distributions do not perform well in a range of small-sample settings such as a small-sample bioavailability study. In this article, we propose two likelihood-based approaches-the signed log-likelihood ratio statistic and modified signed log-likelihood ratio statistic-for inference about the ratio of means of two independent log-normal distributions. More specifically, we focus on obtaining p-values for testing the equality of means and also constructing confidence intervals for the ratio of means. The performance of the proposed methods is assessed through simulation studies that show that the modified signed log-likelihood ratio statistic is nearly an exact approach even for very small samples. The methods are also applied to two real-life examples.