On small-sample confidence intervals for parameters in discrete distributions

The traditional definition of a confidence interval requires the coverage probability at any value of the parameter to be at least the nominal confidence level. In constructing such intervals for parameters in discrete distributions, less conservative behavior results from inverting a single two-sided test than inverting two separate one-sided tests of half the nominal level each. We illustrate for a variety of discrete problems, including interval estimation of a binomial parameter, the difference and the ratio of two binomial parameters for independent samples, and the odds ratio.     

Empirical likelihood inference for the area under the ROC curve

For a continuous-scale diagnostic test, the most commonly used summary index of the receiver operating characteristic curve (ROC) is the area under the curve (AUC) that measures the accuracy of the diagnostic test. In this article, we propose an empirical likelihood (EL) approach for the inference on the AUC. First we define an EL ratio for the AUC and show that its limiting distribution is a scaled chi-square distribution. We then obtain an EL-based confidence interval for the AUC using the scaled chi-square distribution. This EL inference for the AUC can be extended to stratified samples, and the resulting limiting distribution is a weighted sum of independent chi-square distributions. Additionally we conduct simulation studies to compare the relative performance of the proposed EL-based interval with the existing normal approximation-based intervals and bootstrap intervals for the AUC.     

Calculating confidence intervals for prediction error in microarray classification using resampling

Cross-validation based point estimates of prediction accuracy are frequently reported in microarray class prediction problems. However these point estimates can be highly variable, particularly for small sample numbers, and it would be useful to provide confidence intervals of prediction accuracy. We performed an extensive study of existing confidence interval methods and compared their performance in terms of empirical coverage and width. We developed a bootstrap case cross-validation (BCCV) resampling scheme and defined several confidence interval methods using BCCV with and without bias-correction. The widely used approach of basing confidence intervals on an independent binomial assumption of the leave-one-out cross-validation errors results in serious under-coverage of the true prediction error. Two split-sample based methods previously proposed in the literature tend to give overly conservative confidence intervals. Using BCCV resampling, the percentile confidence interval method was also found to be overly conservative without bias-correction, while the bias corrected accelerated (BCa) interval method of Efron returns substantially anti-conservative confidence intervals. We propose a simple bias reduction on the BCCV percentile interval. The method provides mildly conservative inference under all circumstances studied and outperforms the other methods in microarray applications with small to moderate sample sizes.     

Estimation of a parameter and its exact confidence interval following sequential sample size reestimation trials

For confirmatory trials of regulatory decision making, it is important that adaptive designs under consideration provide inference with the correct nominal level, as well as unbiased estimates, and confidence intervals for the treatment comparisons in the actual trials. However, naive point estimate and its confidence interval are often biased in adaptive sequential designs. We develop a new procedure for estimation following a test from a sample size reestimation design. The method for obtaining an exact confidence interval and point estimate is based on a general distribution property of a pivot function of the Self-designing group sequential clinical trial by Shen and Fisher (1999, Biometrics55, 190-197). A modified estimate is proposed to explicitly account for futility stopping boundary with reduced bias when block sizes are small. The proposed estimates are shown to be consistent. The computation of the estimates is straightforward. We also provide a modified weight function to improve the power of the test. Extensive simulation studies show that the exact confidence intervals have accurate nominal probability of coverage, and the proposed point estimates are nearly unbiased with practical sample sizes.     

Estimating the variance of disease-prevalence estimates from population-based registries

We propose a new Poisson method to estimate the variance for prevalence estimates obtained by the counting method described by Gail et al. (1999, Biometrics 55, 1137-1144) and to construct a confidence interval for the prevalence. We evaluate both the Poisson procedure and the procedure based on the bootstrap proposed by Gail et al. in simulated samples generated by resampling real data. These studies show that both variance estimators usually perform well and yield coverages of confidence intervals at nominal levels. When the number of disease survivors is very small, however, confidence intervals based on the Poisson method have supranominal coverage, whereas those based on the procedure of Gail et al. tend to have below-nominal coverage. For these reasons, we recommend the Poisson method, which also reduces the computational burden considerably.     

Nonparametric statistical methods for cost-effectiveness analyses

Two measures often used in a cost-effectiveness analysis are the incremental cost-effectiveness ratio (ICER) and the net health benefit (NHB). Inferences on these two quantities are often hindered by highly skewed cost data. In this article, we derive the Edgeworth expansions for the studentized t-statistics for the two measures and show how they could be used to guide inferences. In particular, we use the expansions to study the theoretical performance of existing confidence intervals based on normal theory and to derive new confidence intervals for the ICER and the NHB. We conduct a simulation study to compare our new intervals with several existing methods. The methods evaluated include Taylor's interval, Fieller's interval, the bootstrap percentile interval, and the bootstrap bias-corrected acceleration interval. We found that our new intervals give good coverage accuracy and are narrower compared to the current recommended intervals.     

Evaluation of jackknife and bootstrap for defining confidence intervals for pairwise agreement measures

Several research fields frequently deal with the analysis of diverse classification results of the same entities. This should imply an objective detection of overlaps and divergences between the formed clusters. The congruence between classifications can be quantified by clustering agreement measures, including pairwise agreement measures. Several measures have been proposed and the importance of obtaining confidence intervals for the point estimate in the comparison of these measures has been highlighted. A broad range of methods can be used for the estimation of confidence intervals. However, evidence is lacking about what are the appropriate methods for the calculation of confidence intervals for most clustering agreement measures. Here we evaluate the resampling techniques of bootstrap and jackknife for the calculation of the confidence intervals for clustering agreement measures. Contrary to what has been shown for some statistics, simulations showed that the jackknife performs better than the bootstrap at accurately estimating confidence intervals for pairwise agreement measures, especially when the agreement between partitions is low. The coverage of the jackknife confidence interval is robust to changes in cluster number and cluster size distribution.     

Comparison of family-based association tests in chromosome regions selected by linkage-based confidence intervals

We use the Genetic Analysis Workshop 14 simulated data to explore the effectiveness of a two-stage strategy for mapping complex disease loci consisting of an initial genome scan with confidence interval construction for gene location, followed by fine mapping with family-based tests of association on a dense set of single-nucleotide polymorphisms. We considered four types of intervals: the 1-LOD interval, a basic percentile bootstrap confidence interval based on the position of the maximum Zlr score, and asymptotic and bootstrap confidence intervals based on a generalized estimating equations method. For fine mapping we considered two family-based tests of association: a test based on a likelihood ratio statistic and a transmission-disequilibrium-type test implemented in the software FBAT. In two of the simulation replicates, we found that the bootstrap confidence intervals based on the peak Zlr and the 1-LOD support interval always contained the true disease loci and that the likelihood ratio test provided further strong confirmatory evidence of the presence of disease loci in these regions.     

Likelihood-based experimental design for estimation of ED50

In selecting the best dosage choice for the estimation of ED50, it is natural to try to minimize the length of the confidence intervals. In this presentation, the dose allocation that minimizes the length of the likelihood-based confidence intervals is presented and compared with alternative allocations that have been proposed based on the length of different types of confidence intervals, such as those based on the asymptotic variance or on Fieller's Theorem. Effective strategies to deal with the parameter dependence of these allocations are explored. A series of experiments to evaluate the effect of small doses per fraction on the radiation tolerance of the rat cervical spinal cord provide the motivation and an illustration for the proposed procedures.     

A Two One-Sided Parametric Tolerance Interval Test for Control of Delivered Dose Uniformity. Part 1—Characterization of FDA Proposed Test

The FDA proposed a parametric tolerance interval (PTI) test at the October 2005 Advisory Committee meeting as a replacement of the attribute (counting) test for delivered dose uniformity (DDU), published in the 1998 draft guidance for metered dose inhalers (MDIs) and dry powder inhalers (DPIs) and the 2002 final guidance for inhalation sprays and intranasal products. This article (first in a series of three) focuses on the test named by the FDA “87.5% coverage.” Unlike a typical two-sided PTI test, which controls the proportion of the DDU distribution within a target interval (coverage), this test is comprised of two one-sided tests (TOST) designed to control the maximum amount of DDU values in either tail of the distribution above and below the target interval. Through simulations, this article characterizes the properties and performance of the proposed PTI-TOST under different scenarios. The results show that coverages of 99% or greater are needed for a batch to have acceptance probability 98% or greater with the test named by the FDA “87.5% coverage” (95% confidence level), while batches with 87.5% coverage have less than 1% probability of being accepted. The results also illustrate that with this PTI-TOST, the coverage requirement for a given acceptance probability increases as the batch mean deviates from target. The accompanying articles study the effects of changing test parameters and the test robustness to deviations from normality.     

Permutation inference methods for multivariate meta-analysis

Multivariate meta-analysis is gaining prominence in evidence synthesis research because it enables simultaneous synthesis of multiple correlated outcome data, and random-effects models have generally been used for addressing between-studies heterogeneities. However, coverage probabilities of confidence regions or intervals for standard inference methods for random-effects models (eg, restricted maximum likelihood estimation) cannot retain their nominal confidence levels in general, especially when the number of synthesized studies is small because their validities depend on large sample approximations. In this article, we provide permutation-based inference methods that enable exact joint inferences for average outcome measures without large sample approximations. We also provide accurate marginal inference methods under general settings of multivariate meta-analyses. We propose effective approaches for permutation inferences using optimal weighting based on the efficient score statistic. The effectiveness of the proposed methods is illustrated via applications to bivariate meta-analyses of diagnostic accuracy studies for airway eosinophilia in asthma and a network meta-analysis for antihypertensive drugs on incident diabetes, as well as through simulation experiments. In numerical evaluations performed via simulations, our methods generally provided accurate confidence regions or intervals under a broad range of settings, whereas the current standard inference methods exhibited serious undercoverage properties.     

Rectangular tolerance regions and multivariate normal reference regions in laboratory medicine

Reference intervals are widely used in the interpretation of results of biochemical and physiological tests of patients. When there are multiple biochemical analytes measured from each subject, a multivariate reference region is needed. Because of their greater specificity against false positives, such reference regions are more desirable than separate univariate reference intervals that disregard the cross-correlations between variables. Traditionally, under multivariate normality, reference regions have been constructed as ellipsoidal regions. This approach suffers from a major drawback: it cannot detect component-wise extreme observations. In the present work, procedures are developed to construct rectangular reference regions in the multivariate normal setup. The construction is based on the criteria for tolerance intervals. The problems addressed include the computation of a rectangular tolerance region and simultaneous tolerance intervals. Also addressed is the computation of mixed reference intervals that include both two-sided and one-sided limits, simultaneously. A parametric bootstrap approach is used in the computations, and the accuracy of the proposed methodology is assessed using estimated coverage probabilities. The problem of sample size determination is also addressed, and the results are illustrated using examples that call for the computation of reference regions.     

Sample size determination for establishing equivalence/noninferiority via ratio of two proportions in matched-pair design

In this article, we propose approximate sample size formulas for establishing equivalence or noninferiority of two treatments in match-pairs design. Using the ratio of two proportions as the equivalence measure, we derive sample size formulas based on a score statistic for two types of analyses: hypothesis testing and confidence interval estimation. Depending on the purpose of a study, these formulas can be used to provide a sample size estimate that guarantees a prespecified power of a hypothesis test at a certain significance level or controls the width of a confidence interval with a certain confidence level. Our empirical results confirm that these score methods are reliable in terms of true size, coverage probability, and skewness. A liver scan detection study is used to illustrate the proposed methods.     

Methods and results from the genome-wide association group at GAW20

Background

X chromosome inactivation (XCI) is an important gene regulation mechanism in females to equalize the expression levels of X chromosome between two sexes. Generally, one of two X chromosomes in females is randomly chosen to be inactivated. Nonrandom XCI (XCI skewing) is also observed in females, which has been reported to play an important role in many X-linked diseases. However, there is no statistical measure available for the degree of the XCI skewing based on family data in population genetics.

Results

In this article, we propose a statistical approach to measure the degree of the XCI skewing based on family trios, which is represented by a ratio of two genotypic relative risks in females. The point estimate of the ratio is obtained from the maximum likelihood estimates of two genotypic relative risks. When parental genotypes are missing in some family trios, the expectation-conditional-maximization algorithm is adopted to obtain the corresponding maximum likelihood estimates. Further, the confidence interval of the ratio is derived based on the likelihood ratio test. Simulation results show that the likelihood-based confidence interval has an accurate coverage probability under the situations considered. Also, we apply our proposed method to the rheumatoid arthritis data from USA for its practical use, and find out that a locus, rs2238907, may undergo the XCI skewing against the at-risk allele. But this needs to be further confirmed by molecular genetics.

Conclusions

The proposed statistical measure for the skewness of XCI is applicable to complete family trio data or family trio data with some paternal genotypes missing. The likelihood-based confidence interval has an accurate coverage probability under the situations considered. Therefore, our proposed statistical measure is generally recommended in practice for discovering the potential loci which undergo the XCI skewing.

     

Adjusting for publication bias in meta-analysis via inverse probability weighting using clinical trial registries

Publication bias is a major concern in conducting systematic reviews and meta-analyses. Various sensitivity analysis or bias-correction methods have been developed based on selection models, and they have some advantages over the widely used trim-and-fill bias-correction method. However, likelihood methods based on selection models may have difficulty in obtaining precise estimates and reasonable confidence intervals, or require a rather complicated sensitivity analysis process. Herein, we develop a simple publication bias adjustment method by utilizing the information on conducted but still unpublished trials from clinical trial registries. We introduce an estimating equation for parameter estimation in the selection function by regarding the publication bias issue as a missing data problem under the missing not at random assumption. With the estimated selection function, we introduce the inverse probability weighting (IPW) method to estimate the overall mean across studies. Furthermore, the IPW versions of heterogeneity measures such as the between-study variance and the I² measure are proposed. We propose methods to construct confidence intervals based on asymptotic normal approximation as well as on parametric bootstrap. Through numerical experiments, we observed that the estimators successfully eliminated bias, and the confidence intervals had empirical coverage probabilities close to the nominal level. On the other hand, the confidence interval based on asymptotic normal approximation is much wider in some scenarios than the bootstrap confidence interval. Therefore, the latter is recommended for practical use.     

Fidelity Estimation for a Hierarchical Classifier

To estimate the correct classification rate of a classifier, many different methods exist (test sample, bootstrap, cross validation). The test sample is a method with very small expense. Sometimes, only a small number of objects is available (seldom diseases, high costs for experiments). When we split the sample in training set and test set, we get good or bad fidelity estimations but, unfortunately, vice versa a big or small confidence interval for the estimation. Overcoming this dilemma is only possible for simple classifiers. Such a simple classifier is investigated and a direct fidelity estimation is proposed.     

Confidence Intervals in Qtl Mapping by Bootstrapping

The determination of empirical confidence intervals for the location of quantitative trait loci (QTLs) was investigated using simulation. Empirical confidence intervals were calculated using a bootstrap resampling method for a backcross population derived from inbred lines. Sample sizes were either 200 or 500 individuals, and the QTL explained 1, 5, or 10% of the phenotypic variance. The method worked well in that the proportion of empirical confidence intervals that contained the simulated QTL was close to expectation. In general, the confidence intervals were slightly conservatively biased. Correlations between the test statistic and the width of the confidence interval were strongly negative, so that the stronger the evidence for a QTL segregating, the smaller the empirical confidence interval for its location. The size of the average confidence interval depended heavily on the population size and the effect of the QTL. Marker spacing had only a small effect on the average empirical confidence interval. The LOD drop-off method to calculate empirical support intervals gave confidence intervals that generally were too small, in particular if confidence intervals were calculated only for samples above a certain significance threshold. The bootstrap method is easy to implement and is useful in the analysis of experimental data.     

Inferences on the common mean of several log-normal populations: the generalized variable approach

This paper proposes a novel approach for the confidence interval estimation and hypothesis testing of the common mean of several log-normal populations using the concept of generalized variable. Simulation studies demonstrate that the proposed approach can provide confidence intervals with satisfying coverage probabilities and can perform hypothesis testing with satisfying type-I error control even at small sample sizes. Overall, it is superior to the large sample approach. The proposed method is illustrated using two examples.     

Confidence intervals for the substitution number in the nucleotide substitution models

In the nucleotide substitution model for molecular evolution, a major task in the exploration of an evolutionary process is to estimate the substitution number per site of a protein or DNA sequence. The usual estimators are based on the observation of the difference proportion of the two nucleotide sequences. However, a more objective approach is to report a confidence interval with precision rather than only providing point estimators. The conventional confidence intervals used in the literature for the substitution number are constructed by the normal approximation. The performance and construction of confidence intervals for evolutionary models have not been much investigated in the literature. In this article, the performance of these conventional confidence intervals for one-parameter and two-parameter models are explored. Results show that the coverage probabilities of these intervals are unsatisfactory when the true substitution number is small. Since the substitution number may be small in many situations for an evolutionary process, the conventional confidence interval cannot provide accurate information for these cases. Improved confidence intervals for the one-parameter model with desirable coverage probability are proposed in this article. A numerical calculation shows the substantial improvement of the new confidence intervals over the conventional confidence intervals.     

Estimating the species accumulation curve using mixtures

As a significant tool in ecological studies, the species accumulation curve or the collector's curve is the graph of the expected number of detected species as a function of sampling effort. The problem of estimating the species accumulation curve based on an empirical data set arising from quadrat sampling is studied in a nonparametric binomial mixture model. It will be shown that estimating the species accumulation curve not only is independent of the unknown number of species but also includes estimating the number of species as a limiting case. For the purpose of interpolation, moment-based estimators, associated with asymptotic confidence intervals, are developed from several points of view. A likelihood-based procedure is developed for the purpose of extrapolation, associated with bootstrap confidence intervals. The proposed methods are illustrated by ecological data sets.