Is the Exact Test Better than the Asymptotic Test for Testing Marginal Homogeneity in 2 × 2 Tables?

McNemar test is commonly used to test for the marginal homogeneity in 2 × 2 contingency tables. McNemar test is an asymptotic test based either on standard normal distribution or on the chi‐square distribution. When the total sample size is small, an exact version of McNemar test is available based on the binomial probabilities. The example in the paper came from a clinical study to investigate the effect of epidermal growth factor for children who had microvillus inclusion diseases. There were only six observations available. The test results differ between the exact test and the asymptotic test. It is a common belief that with this small sample size the exact test be used. However, we claim that McNemar test performs better than the exact test even when the sample size is small. In order to investigate the performances of McNemar test and the exact test, we identify the parameters that affect the test results and then perform sensitivity analysis. In addition, through Monte Carlo simulation studies we compare the empirical sizes and powers of these tests as well as other asymptotic tests such as Wald test and the likelihood ratio test.     

Exact analysis of dose response for multiple correlated binary outcomes

The neurotoxicity of a substance is often tested using animal bioassays. In the functional observational battery, animals are exposed to a test agent and multiple outcomes are recorded to assess toxicity, using approximately 40 animals measured on up to 30 different items. This design gives rise to a challenging statistical problem: a large number of outcomes for a small sample of subjects. We propose an exact test for multiple binary outcomes, under the assumption that the correlation among these items is equal. This test is based upon an exponential model described by Molenberghs and Ryan (1999, Environmetrics 10, 279-300) and extends the methods developed by Corcoran et al. (2001, Biometrics 57, 941-948) who developed an exact test for exchangeably correlated binary data for groups (clusters) of correlated observations. We present a method that computes an exact p-value testing for a joint dose-response relationship. An estimate of the parameter for dose response is also determined along with its 95% confidence bound. The method is illustrated using data from a neurotoxicity bioassay for the chemical perchlorethylene.     

Significance Levels of Randomization Trend Tests in the Event of Rare Occurrences

In the statistical evaluation of data from a dose-response experiment, it is frequently of interest to test for dose-related trend: an increasing trend in response with increasing dose. The randomization trend test, a generalization of Fisher's exact test, has been recommended for animal tumorigenicity testing when the numbers of tumor occurrences are small. This paper examines the type I error of the randomization trend test, and the Cochran-Armitage and Mantel-Haenszel tests. Simulation results show that when the tumor incidence rates are less than 10%, the randomization test is conservative; the test becomes very conservative when the incidence rate is less than 5%. The Cochran-Armitage and Mantel-Haenszel tests are slightly anti-conservative (liberal) when the incidence rates are larger than 3%. Further, we propose a less conservatived method of calculating the p-value of the randomization trend test by excluding some permutations whose probabilities of occurrence are greater than the probability of the the observed outcome.     

An exact trend test for correlated binary data

The problem of testing a dose-response relationship in the presence of exchangeably correlated binary data has been addressed using a variety of models. Most commonly used approaches are derived from likelihood or generalized estimating equations and rely on large-sample theory to justify their inferences. However, while earlier work has determined that these methods may perform poorly for small or sparse samples, there are few alternatives available to those faced with such data. We propose an exact trend test for exchangeably correlated binary data when groups of correlated observations are ordered. This exact approach is based on an exponential model derived by Molenberghs and Ryan (1999) and Ryan and Molenberghs (1999) and provides natural analogues to Fisher's exact test and the binomial trend test when the data are correlated. We use a graphical method with which one can efficiently compute the exact tail distribution and apply the test to two examples.     

A Maximum Rank-Sum Test for One-Pulse Variation in Monthly Data

A nonparametric test to detect a pulse in monthly data is presented. This test is a maximum rank-sum test. The test statistic can be computed from frequencies or rates. The exact null distribution of the test statistic is tabulated for pulses that last 3, 4, 5, or 6 months. Estimates from a simulation study of the test's type I error rate and power are presented. The statistical modeling of the data is discussed. Several examples are given to illustrate the application of the test and the modeling procedures. Practical matters such as the treatment of tied observations, the effect of unequal lengths in the months, sample-size calculation, and post-test power analysis are discussed and illustrated with examples.     

Unconditional Exact Tests for Equivalence or Noninferiority for Paired Binary Endpoints

Problems of establishing equivalence or noninferiority between two medical diagnostic procedures involve comparisons of the response rates between correlated proportions. When the sample size is small, the asymptotic tests may not be reliable. This article proposes an unconditional exact test procedure to assess equivalence or noninferiority. Two statistics, a sample-based test statistic and a restricted maximum likelihood estimation (RMLE)-based test statistic, to define the rejection region of the exact test are considered. We show the p-value of the proposed unconditional exact tests can be attained at the boundary point of the null hypothesis. Assessment of equivalence is often based on a comparison of the confidence limits with the equivalence limits. We also derive the unconditional exact confidence intervals on the difference of the two proportion means for the two test statistics. A typical data set of comparing two diagnostic procedures is analyzed using the proposed unconditional exact and asymptotic methods. The p-value from the unconditional exact tests is generally larger than the p-value from the asymptotic tests. In other words, an exact confidence interval is generally wider than the confidence interval obtained from an asymptotic test.     

Test-based exact confidence intervals for the difference of two binomial proportions

Confidence intervals are often provided to estimate a treatment difference. When the sample size is small, as is typical in early phases of clinical trials, confidence intervals based on large sample approximations may not be reliable. In this report, we propose test-based methods of constructing exact confidence intervals for the difference in two binomial proportions. These exact confidence intervals are obtained from the unconditional distribution of two binomial responses, and they guarantee the level of coverage. We compare the performance of these confidence intervals to ones based on the observed difference alone. We show that a large improvement can be achieved by using the standardized Z test with a constrained maximum likelihood estimate of the variance.     

On the exact null distribution of the generalised likelihood ratio test for analysing unreplicated factorial designs

Al-Shiha and Yang (1999) proposed a multistage procedure for analysing unreplicated factorial experiments, which is based on the statistic that is derived from the generalised likelihood ratio test statistic under the assumption of normality. It was shown by their simulation study that the method is quite competitive with Lenth's (1989) method. In their paper, because of the difficulty of determining the null distribution analytically, the quantiles of the null distribution were empirically simulated. In this paper, we give the exact null distribution of their test statistic, which makes it possible to calculate the critical values of the test.     

EDISON-WMW: Exact Dynamic Programing Solution of the Wilcoxon–Mann–Whitney Test

In many research disciplines, hypothesis tests are applied to evaluate whether findings are statistically significant or could be explained by chance. The Wilcoxon–Mann–Whitney(WMW) test is among the most popular hypothesis tests in medicine and life science to analyze if two groups of samples are equally distributed. This nonparametric statistical homogeneity test is commonly applied in molecular diagnosis. Generally, the solution of the WMW test takes a high combinatorial effort for large sample cohorts containing a significant number of ties. Hence, P value is frequently approximated by a normal distribution. We developed EDISON-WMW, a new approach to calculate the exact permutation of the two-tailed unpaired WMW test without any corrections required and allowing for ties. The method relies on dynamic programing to solve the combinatorial problem of the WMW test efficiently. Beyond a straightforward implementation of the algorithm, we presented different optimization strategies and developed a parallel solution. Using our program,the exact P value for large cohorts containing more than 1000 samples with ties can be calculated within minutes. We demonstrate the performance of this novel approach on randomly-generated data, benchmark it against 13 other commonly-applied approaches and moreover evaluate molecular biomarkers for lung carcinoma and chronic obstructive pulmonary disease(COPD). We foundthat approximated P values were generally higher than the exact solution provided by EDISONWMW. Importantly, the algorithm can also be applied to high-throughput omics datasets, where hundreds or thousands of features are included. To provide easy access to the multi-threaded version of EDISON-WMW, a web-based solution of our algorithm is freely available at http://www.ccb.uni-saarland.de/software/wtest/.     

On Exact Unconditional Test for Linear Trend in Dose‐Response Studies

The one‐degree‐of‐freedom Cochran‐Armitage (C‐A) test statistic for linear trend has been widely applied in various dose‐response studies (e.g., anti‐ulcer medications and short‐term antibiotics, animal carcinogenicity bioassays and occupational toxicant studies). This approximate statistic relies, however, on asymptotic theory that is reliable only when the sample sizes are reasonably large and well balanced across dose levels. For small, sparse, or skewed data, the asymptotic theory is suspect and exact conditional method (based on the C‐A statistic) seems to provide a dependable alternative. Unfortunately, the exact conditional method is only practical for the linear logistic model from which the sufficient statistics for the regression coefficients can be obtained explicitly. In this article, a simple and efficient recursive polynomial multiplication algorithm for exact unconditional test (based on the C‐A statistic) for detecting a linear trend in proportions is derived. The method is applicable for all choices of the model with monotone trend including logistic, probit, arcsine, extreme value and one hit. We also show that this algorithm can be easily extended to exact unconditional power calculation for studies with up to a moderately large sample size. A real example is given to illustrate the applicability of the proposed method.     

A Confidence‐Limit‐Based Approach to the Assessment of Hardy–Weinberg Equilibrium

The classical χ²‐procedure for the assessment of Hardy–Weinberg equilibrium (HWE) is tailored for detecting violations of HWE. However, many applications in genetic epidemiology require approximate compatibility with HWE. In a previous contribution to the field (Wellek, S. (2004). Biometrics, 60 , 694–703), the methodology of statistical equivalence testing was exploited for the construction of tests for problems in which the assumption of approximate compatibility of a given genotype distribution with HWE plays the role of the alternative hypothesis one aims to establish. In this article, we propose a procedure serving the same purpose but relying on confidence limits rather than critical bounds of a significance test. Interval estimation relates to essentially the same parametric function that was previously chosen as the target parameter for constructing an exact conditional UMPU test for equivalence with a HWE conforming genotype distribution. This population parameter is shown to have a direct genetic interpretation as a measure of relative excess heterozygosity. Confidence limits are constructed using both asymptotic and exact methods. The new approach is illustrated by reanalyzing genotype distributions obtained from published genetic association studies, and detailed guidance for choosing the equivalence margin is provided. The methods have been implemented in freely available SAS macros.     

Testing the Equality of the Intra‐Subject Treatment Covariance Matrices in a Crossover Study

In a 2 × 2 crossover bioavailability study, the sets of estimates of the pharmacokinetic parameters quite often have a symmetric covariance structure between the two treatments. For testing the equality of the intra‐subject covariance matrices for the two treatments in such studies, we suggest in this paper some statistical tests. When the response vectors are bivariate, we propose an exact test. Since the statistical procedures depend on the assumption of a symmetric covariance structure between the two treatments, we put forth some statistical tests for this assumption. We then apply the discussed tests to real data from a crossover bioavailability trial.     

Tests for establishing compatibility of an observed genotype distribution with Hardy-Weinberg equilibrium in the case of a biallelic locus

The classical chi(2)-procedure for the assessment of genetic equilibrium is tailored for establishing lack rather than goodness of fit of an observed genotype distribution to a model satisfying the Hardy-Weinberg law, and the same is true for the exact competitors to the large-sample procedure, which have been proposed in the biostatistical literature since the late 1930s. In this contribution, the methodology of statistical equivalence testing is adopted for the construction of tests for problems in which the assumption of approximate compatibility of the genotype distribution actually sampled with Hardy-Weinberg equilibrium (HWE) plays the role of the alternative hypothesis one aims to establish. The result of such a construction highly depends on the choice of a measure of distance to be used for defining an indifference zone containing those genotype distributions whose degree of disequilibrium shall be considered irrelevant. The first such measure proposed here is the Euclidean distance of the true parameter vector from that of a genotype distribution with identical allele frequencies being in strict HWE. The second measure is based on the (scalar) parameter of the distribution first introduced into the present context by Stevens (1938, Annals of Eugenics 8, 377-383). The first approach leads to a nonconditional test (which nevertheless can be carried out in a numerically exact way), the second to an exact conditional test shown to be uniformly most powerful unbiased (UMPU) for the associated pair of hypotheses. Both tests are compared in terms of the exact power attained against the class of those specific alternatives under which HWE is strictly satisfied.     

Conditional Maximum Likelihood Estimate and Exact Test of the Common Relative Difference in Combination of 2 × 2 Tables under Inverse Sampling

On the basis of the conditional distribution, given the marginal totals of non-cases fixed for each of independent 2 × 2 tables under inverse sampling, this paper develops the conditional maximum likelihood (CMLE) estimator of the underlying common relative difference (RD) and its asymptotic conditional variance. This paper further provides for the RD an exact interval calculation procedure, of which the coverage probability is always larger than or equal to the desired confidence level and for investigating whether the underlying common RD equals any specified value an exact test procedure, of which Type I error is always less than or equal to the nominal α-level. These exact interval estimation and exact hypothesis testing procedures are especially useful for the situation in which the number of index subjects in a study is small and the asymptotically approximate methods may not be appropriate for use. This paper also notes the condition under which the CMLE of RD uniquely exists and includes a simple example to illustrate use of these techniques.     

Comparison of independent samples of high-dimensional data by pairwise distance measures

Pairwise distance or association measures of sample elements are often used as a basis for hierarchical cluster analyses. They can also be used in tests for the comparison of pre-defined subgroups of the total sample. Usually this is done with permutation tests In this paper, we compare such a procedure with alternative tests for high-dimensional data based on spherically distributed scores in simulation experiments and with real data. The tests based on the pairwise distance or similarity measures perform quite well in this comparison. As the number of possible permutations is small in very small samples, this might restrict the use of the test. Therefore, we propose an exact parametric small sample version of the test using randomly rotated samples.     

Small Sample Properties of the Mantel-Haenszel Test Statistic for Density Follow-Up Studies

For the analysis of combinations of 2×2 non-contingency tables as obtained from density follow-up studies (relating a number of events to a number of person-years of follow-up) an analogue of the Mantel-Haenszel test for 2×2 contingency tables is widely used. In this paper the small sample properties of this test, both with and without continuity correction, are evaluated. Also the improvement of the test-statistic by using the first four cumulants via the Edgeworth expansion was studied. Results on continuity correction agree with similar studies on the Mantel-Haenszel statistic for 2×2 contingency tables: Continuity correction gives a p-value which approximates the exact p-value better than the p-value obtained without this correction; both the exact test and its approximations show considerable conservatism in small samples; the uncorrected Mantel-Haenszel test statistic gives a p-value that agrees more with the nominal significance level, but can be anti-conservative. The p-value based on the first four cumulants gives a better approximation of the exact p-value than the continuity corrected test, especially when the distribution has marked skewness.     

Exact Calculation of Permutational Distributions for Two Independent Samples

A simple Fortran Subroutine is given for the exact calculation of permutational distributions for two independent samples. Important special cases are the Fisher-Pitman randomization test, the Mann-Whitney U-test, the Mantel-Haenszel-test and the exact test for 2×k contingency tables. The algorithm needs polynomial time and is of use even for personal- or microcomputers.     

Testing the non-unity of rate ratio under inverse sampling

Inverse sampling is considered to be a more appropriate sampling scheme than the usual binomial sampling scheme when subjects arrive sequentially, when the underlying response of interest is acute, and when maximum likelihood estimators of some epidemiologic indices are undefined. In this article, we study various statistics for testing non-unity rate ratios in case-control studies under inverse sampling. These include the Wald, unconditional score, likelihood ratio and conditional score statistics. Three methods (the asymptotic, conditional exact, and Mid-P methods) are adopted for P-value calculation. We evaluate the performance of different combinations of test statistics and P-value calculation methods in terms of their empirical sizes and powers via Monte Carlo simulation. In general, asymptotic score and conditional score tests are preferable for their actual type I error rates are well controlled around the pre-chosen nominal level, and their powers are comparatively the largest. The exact version of Wald test is recommended if one wants to control the actual type I error rate at or below the pre-chosen nominal level. If larger power is expected and fluctuation of sizes around the pre-chosen nominal level are allowed, then the Mid-P version of Wald test is a desirable alternative. We illustrate the methodologies with a real example from a heart disease study.     

Multiple McNemar Tests

Summary Methods for performing multiple tests of paired proportions are described. A broadly applicable method using McNemar's exact test and the exact distributions of all test statistics is developed; the method controls the familywise error rate in the strong sense under minimal assumptions. A closed form (not simulation‐based) algorithm for carrying out the method is provided. A bootstrap alternative is developed to account for correlation structures. Operating characteristics of these and other methods are evaluated via a simulation study. Applications to multiple comparisons of predictive models for disease classification and to postmarket surveillance of adverse events are given.     

An Exact Paired Rank Test

An exact rank test for two dependent samples based on overall mid‐ranks is discussed which can be applied to metric as well as to ordinal data. The exact conditional distribution of the test statistic given the observed vector of rank differences is determined. A recursion formula is given as well as a fast shift algorithm in SAS/IML code. Moreover, it is demonstrated that the paired rank test can be more powerful than other tests for paired samples by means of a simulation study. Finally, the test is applied to a psychiatric trial with longitudinal ordinal data.