Improving sandwich variance estimation for marginal Cox analysis of cluster randomized trials

Cluster randomized trials (CRTs) frequently recruit a small number of clusters, therefore necessitating the application of small-sample corrections for valid inference. A recent systematic review indicated that CRTs reporting right-censored, time-to-event outcomes are not uncommon and that the marginal Cox proportional hazards model is one of the common approaches used for primary analysis. While small-sample corrections have been studied under marginal models with continuous, binary, and count outcomes, no prior research has been devoted to the development and evaluation of bias-corrected sandwich variance estimators when clustered time-to-event outcomes are analyzed by the marginal Cox model. To improve current practice, we propose nine bias-corrected sandwich variance estimators for the analysis of CRTs using the marginal Cox model and report on a simulation study to evaluate their small-sample properties. Our results indicate that the optimal choice of bias-corrected sandwich variance estimator for CRTs with survival outcomes can depend on the variability of cluster sizes and can also slightly differ whether it is evaluated according to relative bias or type I error rate. Finally, we illustrate the new variance estimators in a real-world CRT where the conclusion about intervention effectiveness differs depending on the use of small-sample bias corrections. The proposed sandwich variance estimators are implemented in an R package CoxBcv .     

Nonparametric estimation of bivariate failure time associations in the presence of a competing risk

Most research on the study of associations among paired failuretimes has either assumed time invariance or been based on complexmeasures or estimators. Little has accommodated competing risks.This paper targets the conditional cause-specific hazard ratio,henceforth called the cause-specific cross ratio, a recent modificationof the conditional hazard ratio designed to accommodate competingrisks data. Estimation is accomplished by an intuitive, nonparametricmethod that localizes Kendall's tau. Time variance is accommodatedthrough a partitioning of space into ‘bins’ betweenwhich the strength of association may differ. Inferential proceduresare developed, small-sample performance is evaluated, and themethods are applied to the investigation of familial associationin dementia onset.     

Avoiding model selection bias in small-sample genomic datasets

MOTIVATION: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data. RESULTS: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.     

Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review,reanalysis, and simulation study

BackgroundCluster randomised trials (CRTs) are commonly analysed using mixed-effects models or generalised estimating equations (GEEs). However, these analyses do not always perform well with the small number of clusters typical of most CRTs. They can lead to increased risk of a type I error (finding a statistically significant treatment effect when it does not exist) if appropriate corrections are not used.MethodsWe conducted a small simulation study to evaluate the impact of using small-sample corrections for mixed-effects models or GEEs in CRTs with a small number of clusters. We then reanalysed data from TRIGGER, a CRT with six clusters, to determine the effect of using an inappropriate analysis method in practice. Finally, we reviewed 100 CRTs previously identified by a search on PubMed in order to assess whether trials were using appropriate methods of analysis. Trials were classified as at risk of an increased type I error rate if they did not report using an analysis method which accounted for clustering, or if they had fewer than 40 clusters and performed an individual-level analysis without reporting the use of an appropriate small-sample correction.ResultsOur simulation study found that using mixed-effects models or GEEs without an appropriate correction led to inflated type I error rates, even for as many as 70 clusters. Conversely, using small-sample corrections provided correct type I error rates across all scenarios. Reanalysis of the TRIGGER trial found that inappropriate methods of analysis gave much smaller P values (P ≤ 0.01) than appropriate methods (P = 0.04–0.15). In our review, of the 99 trials that reported the number of clusters, 64 (65 %) were at risk of an increased type I error rate; 14 trials did not report using an analysis method which accounted for clustering, and 50 trials with fewer than 40 clusters performed an individual-level analysis without reporting the use of an appropriate correction.ConclusionsCRTs with a small or medium number of clusters are at risk of an inflated type I error rate unless appropriate analysis methods are used. Investigators should consider using small-sample corrections with mixed-effects models or GEEs to ensure valid results.     

Hypothesis Testing Procedures for a Series of Independent Fourfold Tables under Inverse Sampling

This paper considers four summary test statistics, including the one recently proposed by Bennett (1986, Biometrical Journal 28, 859–862), for hypothesis testing of association in a series of independent fourfold tables under inverse sampling. This paper provides a systematic and quantitative evaluation of the small-sample performance for these summary test statistics on the basis of a Monte Carlo simulation. This paper notes that the test statistic developed by Bennett (1986) can be conservative and thereby possibly lose the power when the underlying disease is not rare. This paper also finds that for given a fixed total number of cases in each table, the conditional test statistic is the best in controlling type I error among all test statistics considered here.     

Small-sample confidence regions in exponential families

This article presents an algorithm for small-sample conditional confidence regions for two or more parameters for any discrete regression model in the generalized linear interactive model family. Regions are constructed by careful inversion of conditional hypothesis tests. This method presupposes the use of approximate or exact techniques for enumerating the sample space for some components of the vector of sufficient statistics conditional on other components. Such enumeration may be performed exactly or by exact or approximate Monte Carlo, including the algorithms of Kolassa and Tanner (1994, Journal of the American Statistical Association 89, 697-702; 1999, Biometrics 55, 246-251). This method also assumes that one can compute certain conditional probabilities for a fixed value of the parameter vector. Because of a property of exponential families, one can use this set of conditional probabilities to directly compute the conditional probabilities associated with any other value of the vector of the parameters of interest. This observation dramatically reduces the computational effort required to invert the hypothesis test to obtain the confidence region. To construct a region with confidence level 1 - alpha, the algorithm begins with a grid of values for the parameters of interest. For each parameter vector on the grid (corresponding to the current null hypothesis), one transforms the initial set of conditional probabilities using exponential tilting and then calculates the p value for this current null hypothesis. The confidence region is the set of parameter values for which the p value is at least alpha.     

A comparison of exact, mid-P, and score tests for matched case-control studies

A recently developed algorithm for generating the distribution of sufficient statistics for conditional logistic models can be put to a twofold use. First, it provides an avenue for performing inference for matched case-control studies that does not rely on the assumption of a large sample size. Second, joint distributions generated by this algorithm can be used to make comparisons of various inferential procedures that are free from Monte Carlo sampling errors. In this paper, these two features of the algorithm are utilized to compare small-sample properties of the exact, mid-P value, and score tests for a conditional logistic model with two unmatched binary covariates. Both uniparametric and multiparametric tests, performed at a nominal significance level of .05, were studied. It was found that the actual significance levels of the mid-P test tend to be closer to the nominal level when compared with those of the other two tests.     

Unconditional small-sample confidence intervals for the odds ratio

The traditional approach to 'exact' small-sample interval estimation of the odds ratio for binomial, Poisson, or multinomial samples uses the conditional distribution to eliminate nuisance parameters. This approach can be very conservative. For two independent binomial samples, we study an unconditional approach with overall confidence level guaranteed to equal at least the nominal level. With small samples this interval tends to be shorter and have coverage probabilities nearer the nominal level.     

Exact unconditional inference for risk ratio in a correlated 2 x 2 table with structural zero

In this article, we consider small-sample statistical inference for rate ratio (RR) in a correlated 2 x 2 table with a structural zero in one of the off-diagonal cells. Existing Wald's test statistic and logarithmic transformation test statistic will be adopted for this purpose. Hypothesis testing and confidence interval construction based on large-sample theory will be reviewed first. We then propose reliable small-sample exact unconditional procedures for hypothesis testing and confidence interval construction. We present empirical results to evince the better confidence interval performance of our proposed exact unconditional procedures over the traditional large-sample procedures in small-sample designs. Unlike the findings given in Lui (1998, Biometrics 54, 706-711), our empirical studies show that the existing asymptotic procedures may not attain a prespecified confidence level even in moderate sample-size designs (e.g., n = 50). Our exact unconditional procedures on the other hand do not suffer from this problem. Hence, the asymptotic procedures should be applied with caution. We propose two approximate unconditional confidence interval construction methods that outperform the existing asymptotic ones in terms of coverage probability and expected interval width. Also, we empirically demonstrate that the approximate unconditional tests are more powerful than their associated exact unconditional tests. A real data set from a two-step tuberculosis testing study is used to illustrate the methodologies.     

Analysis of covariance with incomplete data via semiparametric model transformations

We propose a method for fitting semiparametric models such as the proportional hazards (PH), additive risks (AR), and proportional odds (PO) models. Each of these semiparametric models implies that some transformation of the conditional cumulative hazard function (at each t) depends linearly on the covariates. The proposed method is based on nonparametric estimation of the conditional cumulative hazard function, forming a weighted average over a range of t-values, and subsequent use of least squares to estimate the parameters suggested by each model. An approximation to the optimal weight function is given. This allows semiparametric models to be fitted even in incomplete data cases where the partial likelihood fails (e.g., left censoring, right truncation). However, the main advantage of this method rests in the fact that neither the interpretation of the parameters nor the validity of the analysis depend on the appropriateness of the PH or any of the other semiparametric models. In fact, we propose an integrated method for data analysis where the role of the various semiparametric models is to suggest the best fitting transformation. A single continuous covariate and several categorical covariates (factors) are allowed. Simulation studies indicate that the test statistics and confidence intervals have good small-sample performance. A real data set is analyzed.     

Likelihood analysis for the ratio of means of two independent log-normal distributions

Existing methods for comparing the means of two independent skewed log-normal distributions do not perform well in a range of small-sample settings such as a small-sample bioavailability study. In this article, we propose two likelihood-based approaches-the signed log-likelihood ratio statistic and modified signed log-likelihood ratio statistic-for inference about the ratio of means of two independent log-normal distributions. More specifically, we focus on obtaining p-values for testing the equality of means and also constructing confidence intervals for the ratio of means. The performance of the proposed methods is assessed through simulation studies that show that the modified signed log-likelihood ratio statistic is nearly an exact approach even for very small samples. The methods are also applied to two real-life examples.     

Scientific knowledge is possible with small-sample classification

A typical small-sample biomarker classification paper discriminates between types of pathology based on, say, 30,000 genes and a small labeled sample of less than 100 points. Some classification rule is used to design the classifier from this data, but we are given no good reason or conditions under which this algorithm should perform well. An error estimation rule is used to estimate the classification error on the population using the same data, but once again we are given no good reason or conditions under which this error estimator should produce a good estimate, and thus we do not know how well the classifier should be expected to perform. In fact, virtually, in all such papers the error estimate is expected to be highly inaccurate. In short, we are given no justification for any claims.Given the ubiquity of vacuous small-sample classification papers in the literature, one could easily conclude that scientific knowledge is impossible in small-sample settings. It is not that thousands of papers overtly claim that scientific knowledge is impossible in regard to their content; rather, it is that they utilize methods that preclude scientific knowledge. In this paper, we argue to the contrary that scientific knowledge in small-sample classification is possible provided there is sufficient prior knowledge. A natural way to proceed, discussed herein, is via a paradigm for pattern recognition in which we incorporate prior knowledge in the whole classification procedure (classifier design and error estimation), optimize each step of the procedure given available information, and obtain theoretical measures of performance for both classifiers and error estimators, the latter being the critical epistemological issue. In sum, we can achieve scientific validation for a proposed small-sample classifier and its error estimate.     

Is Bagging Effective in the Classification of Small-Sample Genomic and Proteomic Data?

There has been considerable interest recently in the application of bagging in the classification of both gene-expression data and protein-abundance mass spectrometry data. The approach is often justified by the improvement it produces on the performance of unstable, overfitting classification rules under small-sample situations. However, the question of real practical interest is whether the ensemble scheme will improve performance of those classifiers sufficiently to beat the performance of single stable, nonoverfitting classifiers, in the case of small-sample genomic and proteomic data sets. To investigate that question, we conducted a detailed empirical study, using publicly-available data sets from published genomic and proteomic studies. We observed that, under t-test and RELIEF filter-based feature selection, bagging generally does a good job of improving the performance of unstable, overfitting classifiers, such as CART decision trees and neural networks, but that improvement was not sufficient to beat the performance of single stable, nonoverfitting classifiers, such as diagonal and plain linear discriminant analysis, or 3-nearest neighbors. Furthermore, as expected, the ensemble method did not improve the performance of these classifiers significantly. Representative experimental results are presented and discussed in this work.     

ON THE EVOLUTION OF HARMING AND RECOGNITION IN FINITE PANMICTIC AND INFINITE STRUCTURED POPULATIONS

Natural selection may favor two very different types of social behaviors that have costs in vital rates (fecundity and/or survival) to the actor: helping behaviors, which increase the vital rates of recipients, and harming behaviors, which reduce the vital rates of recipients. Although social evolutionary theory has mainly dealt with helping behaviors, competition for limited resources creates ecological conditions in which an actor may benefit from expressing behaviors that reduce the vital rates of neighbors. This may occur if the reduction in vital rates decreases the intensity of competition experienced by the actor or that experienced by its offspring. Here, we explore the joint evolution of neutral recognition markers and marker-based costly conditional harming whereby actors express harming, conditional on actor and recipient bearing different conspicuous markers. We do so for two complementary demographic scenarios: finite panmictic and infinite structured populations. We find that marker-based conditional harming can evolve under a large range of recombination rates and group sizes under both finite panmictic and infinite structured populations. A direct comparison with results for the evolution of marker-based conditional helping reveals that, if everything else is equal, marker-based conditional harming is often more likely to evolve than marker-based conditional helping.     

Conditioning on subsets of the data: applications to ascertainment and other genetic problems.

I here consider the question of when to formulate a likelihood over the whole data set, as opposed to conditioning the likelihood on subsets of the data (i.e., joint vs. conditional likelihoods). I show that when certain conditions are met, these two likelihoods are guaranteed to be equivalent, and thus that it is generally preferable to condition on subsets, since that likelihood is mathematically and computationally simpler. However, I show that when these conditions are not met, conditioning on subsets of the data is equivalent to introducing additional df into our genetic model, df that we may not have been aware of. I discuss the implications of these facts for ascertainment corrections and other genetic problems.     

Tests of significance using regression models for ordered categorical data

Regression models of the type proposed by McCullagh (1980, Journal of the Royal Statistical Society, Series B 42, 109-142) are a general and powerful method of analyzing ordered categorical responses, assuming categorization of an (unknown) continuous response of a specified distribution type. Tests of significance with these models are generally based on likelihood-ratio statistics that have asymptotic chi 2 distributions; therefore, investigators with small data sets may be concerned with the small-sample behavior of these tests. In a Monte Carlo sampling study, significance tests based on the ordinal model are found to be powerful, but a modified test procedure (using an F distribution with a finite number of degrees of freedom for the denominator) is suggested such that the empirical significance level agrees more closely with the nominal significance level in small-sample situations. We also discuss the parallels between an ordinal regression model assuming underlying normality and conventional multiple regression. We illustrate the model with two data sets: one from a study investigating the relationship between phosphorus in soil and plant-available phosphorus in corn grown in that soil, and the other from a clinical trial comparing analgesic drugs.     

Conditional QTL underlying resistance to late blight in a diploid potato population

A large number of quantitative trait loci (QTL) for resistance to late blight of potato have been reported with a "conventional" method in which each phenotypic trait reflects the cumulative genetic effects for the duration of the disease process. However, as genes controlling response to disease may have unique contributions with specific temporal features, it is important to consider the phenotype as dynamic. Here, using the net genetic effects evidenced at consecutive time points during disease development, we report the first conditional mapping of QTL underlying late blight resistance in potato under five environments in Peru. Six conditional QTL were mapped, one each on chromosome 2, 7 and 12 and three on chromosome 9. These QTL represent distinct contributions to the phenotypic variation at different stages of disease development. By comparison, when conventional mapping was conducted, only one QTL was detected on chromosome 9. This QTL was the same as one of the conditional QTL. The results imply that conditional QTL reflect genes that function at particular stages during the host-pathogen interaction. The dynamics revealed by conditional QTL mapping could contribute to the understanding of the molecular mechanism of late blight resistance and these QTL could be used to target genes for marker development or manipulation to improve resistance.     

Bias correction for segregation ratio estimation in human genetics

Segregation ratio estimation has long been important in human genetics. A simple truncated binomial model is considered that assumes complete ascertainment and a deterministic genotype-phenotype relationship. A simple but intuitively appealing estimator of the segregation ratio, previously proposed, is shown to have a negative bias. It is also shown that the bias of this estimator can be largely reduced via a randomization device, resulting in a new estimator that has the same large-sample behavior but with a negligible bias (decaying at a geometric rate). Numerical results are given to show the small-sample performance of this new estimator. An extension to incomplete ascertainment is also considered.     

Simultaneous confidence intervals for comparing binomial parameters

SUMMARY: To compare proportions with several independent binomial samples, we recommend a method of constructing simultaneous confidence intervals that uses the studentized range distribution with a score statistic. It applies to a variety of measures, including the difference of proportions, odds ratio, and relative risk. For the odds ratio, a simulation study suggests that the method has coverage probability closer to the nominal value than ad hoc approaches such as the Bonferroni implementation of Wald or "exact" small-sample pairwise intervals. It performs well even for the problematic but practically common case in which the binomial parameters are relatively small. For the difference of proportions, the proposed method has performance comparable to a method proposed by Piegorsch (1991, Biometrics 47, 45-52).