Is the Exact Test Better than the Asymptotic Test for Testing Marginal Homogeneity in 2 × 2 Tables?

McNemar test is commonly used to test for the marginal homogeneity in 2 × 2 contingency tables. McNemar test is an asymptotic test based either on standard normal distribution or on the chi‐square distribution. When the total sample size is small, an exact version of McNemar test is available based on the binomial probabilities. The example in the paper came from a clinical study to investigate the effect of epidermal growth factor for children who had microvillus inclusion diseases. There were only six observations available. The test results differ between the exact test and the asymptotic test. It is a common belief that with this small sample size the exact test be used. However, we claim that McNemar test performs better than the exact test even when the sample size is small. In order to investigate the performances of McNemar test and the exact test, we identify the parameters that affect the test results and then perform sensitivity analysis. In addition, through Monte Carlo simulation studies we compare the empirical sizes and powers of these tests as well as other asymptotic tests such as Wald test and the likelihood ratio test.     

Confidence Interval for Rate Ratio in a 2 × 2 Table with Structural Zero: An Application in Assessing False-Negative Rate Ratio When Combining Two Diagnostic Tests

Summary . In this article, we consider problems with correlated data that can be summarized in a 2 × 2 table with structural zero in one of the off‐diagonal cells. Data of this kind sometimes appear in infectious disease studies and two‐step procedure studies. Lui (1998, Biometrics 54, 706–711) considered confidence interval estimation of rate ratio based on Fieller‐type, Wald‐type, and logarithmic transformation statistics. We reexamine the same problem under the context of confidence interval construction on false‐negative rate ratio in diagnostic performance when combining two diagnostic tests. We propose a score statistic for testing the null hypothesis of nonunity false‐negative rate ratio. Score test–based confidence interval construction for false‐negative rate ratio will also be discussed. Simulation studies are conducted to compare the performance of the new derived score test statistic and existing statistics for small to moderate sample sizes. In terms of confidence interval construction, our asymptotic score test–based confidence interval estimator possesses significantly shorter expected width with coverage probability being close to the anticipated confidence level. In terms of hypothesis testing, our asymptotic score test procedure has actual type I error rate close to the pre‐assigned nominal level. We illustrate our methodologies with real examples from a clinical laboratory study and a cancer study.     

Homogeneity test of rate ratios in stratified matched-pair studies

This paper investigates homogeneity test of rate ratios in stratified matched-pair studies on the basis of asymptotic and bootstrap-resampling methods. Based on the efficient score approach, we develop a simple and computationally tractable score test statistic. Several other homogeneity test statistics are also proposed on the basis of the weighted least-squares estimate and logarithmic transformation. Sample size formulae are derived to guarantee a pre-specified power for the proposed tests at the pre-given significance level. Empirical results confirm that (i) the modified score statistic based on the bootstrap-resampling method performs better in the sense that its empirical type I error rate is much closer to the pre-specified nominal level than those of other tests and its power is greater than those of other tests, and is hence recommended, whilst the statistics based on the weighted least-squares estimate and logarithmic transformation are slightly conservative under some of the considered settings; (ii) the derived sample size formulae are rather accurate in the sense that their empirical powers obtained from the estimated sample sizes are very close to the pre-specified nominal powers. A real example is used to illustrate the proposed methodologies.     

Exact and Asymptotic Power Determination For Dose‐Response Studies With Ordinal Response

In this article, we describe a conditional score test for detecting a monotone dose‐response relationship with ordinal response data. We consider three different versions of this test: asymptotic, conditional exact, and mid‐P conditional score test. Exact and asymptotic power formulae based on these tests will be studied. Asymptotic sample size formulae based on the asymptotic conditional score test will be derived. The proposed formulae are applied to a vaccination study and a developmental toxicity study for illustrative purposes. Actual significance level and exact power properties of these tests are compared in a small empirical study. The mid‐P conditional score test is observed to be the most powerful test with actual significance level close to the pre‐specified nominal level.     

Testing approaches for overdispersion in poisson regression versus the generalized poisson model

Overdispersion is a common phenomenon in Poisson modeling, and the negative binomial (NB) model is frequently used to account for overdispersion. Testing approaches (Wald test, likelihood ratio test (LRT), and score test) for overdispersion in the Poisson regression versus the NB model are available. Because the generalized Poisson (GP) model is similar to the NB model, we consider the former as an alternate model for overdispersed count data. The score test has an advantage over the LRT and the Wald test in that the score test only requires that the parameter of interest be estimated under the null hypothesis. This paper proposes a score test for overdispersion based on the GP model and compares the power of the test with the LRT and Wald tests. A simulation study indicates the score test based on asymptotic standard Normal distribution is more appropriate in practical application for higher empirical power, however, it underestimates the nominal significance level, especially in small sample situations, and examples illustrate the results of comparing the candidate tests between the Poisson and GP models. A bootstrap test is also proposed to adjust the underestimation of nominal level in the score statistic when the sample size is small. The simulation study indicates the bootstrap test has significance level closer to nominal size and has uniformly greater power than the score test based on asymptotic standard Normal distribution. From a practical perspective, we suggest that, if the score test gives even a weak indication that the Poisson model is inappropriate, say at the 0.10 significance level, we advise the more accurate bootstrap procedure as a better test for comparing whether the GP model is more appropriate than Poisson model. Finally, the Vuong test is illustrated to choose between GP and NB2 models for the same dataset.     

A Comparison of the Three Conditional Exact Tests in Two‐way Contingency Tables Using the Unconditional Exact Power

The conditional exact tests of homogeneity of two binomial proportions are often used in small samples, because the exact tests guarantee to keep the size under the nominal level. The Fisher's exact test, the exact chi‐squared test and the exact likelihood ratio test are popular and can be implemented in software StatXact. In this paper we investigate which test is the best in small samples in terms of the unconditional exact power. In equal sample cases it is proved that the three tests produce the same unconditional exact power. A symmetry of the unconditional exact power is also found. In unequal sample cases the unconditional exact powers of the three tests are computed and compared. In most cases the Fisher's exact test turns out to be best, but we characterize some cases in which the exact likelihood ratio test has the highest unconditional exact power. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

A comparison of exact, mid-P, and score tests for matched case-control studies

A recently developed algorithm for generating the distribution of sufficient statistics for conditional logistic models can be put to a twofold use. First, it provides an avenue for performing inference for matched case-control studies that does not rely on the assumption of a large sample size. Second, joint distributions generated by this algorithm can be used to make comparisons of various inferential procedures that are free from Monte Carlo sampling errors. In this paper, these two features of the algorithm are utilized to compare small-sample properties of the exact, mid-P value, and score tests for a conditional logistic model with two unmatched binary covariates. Both uniparametric and multiparametric tests, performed at a nominal significance level of .05, were studied. It was found that the actual significance levels of the mid-P test tend to be closer to the nominal level when compared with those of the other two tests.     

Conditional Maximum Likelihood Estimate and Exact Test of the Common Relative Difference in Combination of 2 × 2 Tables under Inverse Sampling

On the basis of the conditional distribution, given the marginal totals of non-cases fixed for each of independent 2 × 2 tables under inverse sampling, this paper develops the conditional maximum likelihood (CMLE) estimator of the underlying common relative difference (RD) and its asymptotic conditional variance. This paper further provides for the RD an exact interval calculation procedure, of which the coverage probability is always larger than or equal to the desired confidence level and for investigating whether the underlying common RD equals any specified value an exact test procedure, of which Type I error is always less than or equal to the nominal α-level. These exact interval estimation and exact hypothesis testing procedures are especially useful for the situation in which the number of index subjects in a study is small and the asymptotically approximate methods may not be appropriate for use. This paper also notes the condition under which the CMLE of RD uniquely exists and includes a simple example to illustrate use of these techniques.     

The use of score tests for inference on variance components

Whenever inference for variance components is required, the choice between one-sided and two-sided tests is crucial. This choice is usually driven by whether or not negative variance components are permitted. For two-sided tests, classical inferential procedures can be followed, based on likelihood ratios, score statistics, or Wald statistics. For one-sided tests, however, one-sided test statistics need to be developed, and their null distribution derived. While this has received considerable attention in the context of the likelihood ratio test, there appears to be much confusion about the related problem for the score test. The aim of this paper is to illustrate that classical (two-sided) score test statistics, frequently advocated in practice, cannot be used in this context, but that well-chosen one-sided counterparts could be used instead. The relation with likelihood ratio tests will be established, and all results are illustrated in an analysis of continuous longitudinal data using linear mixed models.     

Trend Test for Overdispersed Proportions

The Cochran-Armitage test has commonly been used for a trend test in binomial proportions. The quasi-likelihood method provides a simple approach to model extra-binomial proportions. Two versions of the score and Wald tests using different parameterizations for the extra-binomial variance were investigated: one in terms of intercluster correlation, and another in terms of variance. The Monte Carlo simulation was used to evaluate the performance of the each version of the score test and the Wald test, and the Cochran-Armitage test. The simulation shows that the Cochran-Armitage test has the proper size only for the binomial sample data, and the test is no longer valid when applied to the extra-binomial data. The Wald test is more likely to exceed the nominal level than the score test under either intercluster correlation model or variance model. Both score tests performed very well even with the binomial data; the tests control the type I error and in the meantime maintain the power of detecting the dose effects. Based on the design considered in this paper, the two scores test are comparable. The score test based on the intercluster correlations model seems better controlling the Type I error but appears less powerful than that based on the variance model. An example from a developmental toxicity experiment is given.     

Non-inferiority testing with a variable margin

There has been growing interest, when comparing an experimental treatment with an active control with respect to a binary outcome, in allowing the non-inferiority margin to depend on the unknown success rate in the control group. It does not seem universally recognized, however, that the statistical test should appropriately adjust for the uncertainty surrounding the non-inferiority margin. In this paper, we inspect a naive procedure that treats an "observed margin" as if it were fixed a priori, and explain why it might not be valid. We then derive a class of tests based on the delta method, including the Wald test and the score test, for a smooth margin. An alternative derivation is given for the asymptotic distribution of the likelihood ratio statistic, again for a smooth margin. We discuss the asymptotic behavior of these tests when applied to a piecewise smooth margin. A simple condition on the margin function is given which allows the likelihood ratio test to carry over to a piecewise smooth margin using the same critical value as for a smooth margin. Simulation experiments are conducted, under a smooth margin and a piecewise linear margin, to evaluate the finite-sample performance of the asymptotic tests studied.     

Confidence interval for rate ratio in a 2 x 2 table with structural zero: an application in assessing false-negative rate ratio when combining two diagnostic tests

In this article, we consider problems with correlated data that can be summarized in a 2 x 2 table with structural zero in one of the off-diagonal cells. Data of this kind sometimes appear in infectious disease studies and two-step procedure studies. Lui (1998, Biometrics54, 706-711) considered confidence interval estimation of rate ratio based on Fieller-type, Wald-type, and logarithmic transformation statistics. We reexamine the same problem under the context of confidence interval construction on false-negative rate ratio in diagnostic performance when combining two diagnostic tests. We propose a score statistic for testing the null hypothesis of nonunity false-negative rate ratio. Score test-based confidence interval construction for false-negative rate ratio will also be discussed. Simulation studies are conducted to compare the performance of the new derived score test statistic and existing statistics for small to moderate sample sizes. In terms of confidence interval construction, our asymptotic score test-based confidence interval estimator possesses significantly shorter expected width with coverage probability being close to the anticipated confidence level. In terms of hypothesis testing, our asymptotic score test procedure has actual type I error rate close to the pre-assigned nominal level. We illustrate our methodologies with real examples from a clinical laboratory study and a cancer study.     

On small‐sample inference in group randomized trials with binary outcomes and cluster‐level covariates

Group randomized trials (GRTs) randomize groups, or clusters, of people to intervention or control arms. To test for the effectiveness of the intervention when subject‐level outcomes are binary, and while fitting a marginal model that adjusts for cluster‐level covariates and utilizes a logistic link, we develop a pseudo‐Wald statistic to improve inference. Alternative Wald statistics could employ bias‐corrected empirical sandwich standard error estimates, which have received limited attention in the GRT literature despite their broad utility and applicability in our settings of interest. The test could also be carried out using popular approaches based upon cluster‐level summary outcomes. A simulation study covering a variety of realistic GRT settings is used to compare the accuracy of these methods in terms of producing nominal test sizes. Tests based upon the pseudo‐Wald statistic and a cluster‐level summary approach utilizing the natural log of observed cluster‐level odds worked best. Due to weighting, some popular cluster‐level summary approaches were found to lead to invalid inference in many settings. Finally, although use of bias‐corrected empirical sandwich standard error estimates did not consistently result in nominal sizes, they did work well, thus supporting the applicability of marginal models in GRT settings.     

The Wald Statistic in Proportional Hazards Hypothesis Testing

In survivorship modelling using the proportional hazards model of Cox (1972, Journal of the Royal Statistical Society, Series B, 34, 187–220), it is often desired to test a subset of the vector of unknown regression parameters β in the expression for the hazard rate at time t. The likelihood ratio test statistic is well behaved in most situations but may be expensive to calculate. The Wald (1943, Transactions of the American Mathematical Society 54, 426–482) test statistic is easier to calculate, but has some drawbacks. In testing a single parameter in a binomial logit model, Hauck and Donner (1977, Journal of the American Statistical Association 72, 851–853) show that the Wald statistic decreases to zero the further the parameter estimate is from the null and that the asymptotic power of the test decreases to the significance level. The Wald statistic is extensively used in statistical software packages for survivorship modelling and it is therefore important to understand its behavior. The present work examines empirically the behavior of the Wald statistic under various departures from the null hypothesis and under the presence of Type I censoring and covariates in the model. It is shown via examples that the Wald statistic's behavior is not as aberrant as found for the logistic model. For the single parameter case, the asymptotic non-null distribution of the Wald statistic is examined.     

Homogeneity score test for the intraclass version of the kappa statistics and sample-size determination in multiple or stratified studies

When the intraclass correlation coefficient or the equivalent version of the kappa agreement coefficient have been estimated from several independent studies or from a stratified study, we have the problem of comparing the kappa statistics and combining the information regarding the kappa statistics in a common kappa when the assumption of homogeneity of kappa coefficients holds. In this article, using the likelihood score theory extended to nuisance parameters (Tarone, 1988, Communications in Statistics-Theory and Methods 17(5), 1549-1556) we present an efficient homogeneity test for comparing several independent kappa statistics and, also, give a modified homogeneity score method using a noniterative and consistent estimator as an alternative. We provide the sample size using the modified homogeneity score method and compare it with that using the goodness-of-fit method (GOF) (Donner, Eliasziw, and Klar, 1996, Biometrics 52, 176-183). A simulation study for small and moderate sample sizes showed that the actual level of the homogeneity score test using the maximum likelihood estimators (MLEs) of parameters is satisfactorily close to the nominal and it is smaller than those of the modified homogeneity score and the goodness-of-fit tests. We investigated statistical properties of several noniterative estimators of a common kappa. The estimator (Donner et al., 1996) is essentially efficient and can be used as an alternative to the iterative MLE. An efficient interval estimation of a common kappa using the likelihood score method is presented.     

A comparison of tests of the difference in the proportion of patients who are cured.

We compared by simulation the likelihood ratio, Wald, and score tests based on a mixture model similar to that proposed by Farewell (1982, Biometrics 38, 1041-1046), and a simple nonparametric test based on the plateau value of the product-limit estimate, for testing the difference in cured proportions between two groups. The parametric tests obtained their asymptotic properties even in small samples provided that one could assume equal failure rates in the two groups. Otherwise, good agreement with predictions required that essentially all potential failures had been observed. The comparative properties of the parametric tests depended on whether the population survival functions crossed, with the power of the Wald test as good as or better than the others in the common situation when the survival functions do not cross. However, its size was sometimes less than nominal. The score test was often not defined and is therefore of limited value. The product-limit test often performed as well as the parametric tests, and despite being biased in some circumstances, may be a useful alternative to these, especially in small samples when some potential failures have not been observed.     

Analyzing multiply matched cohort studies with two different comparison groups: application to pregnancy rates among HIV+ women

We develop a new statistical method to analyze multiply matched cohort studies with two different comparison groups. We employ a linear-logistic model to describe the underlying log-odds ratios and use a conditional likelihood approach to conduct inference. Under the assumption of homogeneous log-odds ratios, we provide methods to construct both asymptotic and exact confidence regions of the two log-odds ratios in a simple case. We propose a score test to evaluate the assumption of homogeneous log-odds ratios across strata. While our methods are general, we develop them around a specific application, namely, the study of pregnancy rates in HIV-infected women. Our analyses suggest that HIV infection is associated with a decrease in pregnancy rates and that this decrease in fertility becomes significant after accounting for illicit drug use.     

A note on permutation tests for variance components in multilevel generalized linear mixed models

In many applications of generalized linear mixed models to multilevel data, it is of interest to test whether a random effects variance component is zero. It is well known that the usual asymptotic chi-square distribution of the likelihood ratio and score statistics under the null does not necessarily hold. In this note we propose a permutation test, based on randomly permuting the indices associated with a given level of the model, that has the correct Type I error rate under the null. Results from a simulation study suggest that it is more powerful than tests based on mixtures of chi-square distributions. The proposed test is illustrated using data on the familial aggregation of sleep disturbance.     

Improved exact confidence intervals for the odds ratio in two independent binomial samples

For two independent binomial samples, the usual exact confidence interval for the odds ratio based on the conditional approach can be very conservative. Recently, Agresti and Min (2002) showed that the unconditional intervals are preferable to conditional intervals with small sample sizes. We use the unconditional approach to obtain a modified interval, which has shorter length, and its coverage probability is closer to and at least the nominal confidence coefficient.     

Some alternatives to asymptotic tests for the analysis of pharmacogenetic data using nonlinear mixed effects models

Nonlinear mixed effects models allow investigating individual differences in drug concentration profiles (pharmacokinetics) and responses. Pharmacogenetics focuses on the genetic component of this variability. Two tests often used to detect a gene effect on a pharmacokinetic parameter are (1) the Wald test, assessing whether estimates for the gene effect are significantly different from 0 and (2) the likelihood ratio test comparing models with and without the genetic effect. Because those asymptotic tests show inflated type I error on small sample size and/or with unevenly distributed genotypes, we develop two alternatives and evaluate them by means of a simulation study. First, we assess the performance of the permutation test using the Wald and the likelihood ratio statistics. Second, for the Wald test we propose the use of the F-distribution with four different values for the denominator degrees of freedom. We also explore the influence of the estimation algorithm using both the first-order conditional estimation with interaction linearization-based algorithm and the stochastic approximation expectation maximization algorithm. We apply these methods to the analysis of the pharmacogenetics of indinavir in HIV patients recruited in the COPHAR2-ANRS 111 trial. Results of the simulation study show that the permutation test seems appropriate but at the cost of an additional computational burden. One of the four F-distribution-based approaches provides a correct type I error estimate for the Wald test and should be further investigated.