The empirical coverage of confidence intervals: Point estimates and confidence intervals for confidence levels

Many confidence intervals calculated in practice are potentially not exact, either because the requirements for the interval estimator to be exact are known to be violated, or because the (exact) distribution of the data is unknown. If a confidence interval is approximate, the crucial question is how well its true coverage probability approximates its intended coverage probability. In this paper we propose to use the bootstrap to calculate an empirical estimate for the (true) coverage probability of a confidence interval. In the first instance, the empirical coverage can be used to assess whether a given type of confidence interval is adequate for the data at hand. More generally, when planning the statistical analysis of future trials based on existing data pools, the empirical coverage can be used to study the coverage properties of confidence intervals as a function of type of data, sample size, and analysis scale, and thus inform the statistical analysis plan for the future trial. In this sense, the paper proposes an alternative to the problematic pretest of the data for normality, followed by selection of the analysis method based on the results of the pretest. We apply the methodology to a data pool of bioequivalence studies, and in the selection of covariance patterns for repeated measures data.     

Constructing confidence intervals for selected parameters

In large-scale problems, it is common practice to select important parameters by a procedure such as the Benjamini and Hochberg procedure and construct confidence intervals (CIs) for further investigation while the false coverage-statement rate (FCR) for the CIs is controlled at a desired level. Although the well-known BY CIs control the FCR, they are uniformly inflated. In this paper, we propose two methods to construct shorter selective CIs. The first method produces shorter CIs by allowing a reduced number of selective CIs. The second method produces shorter CIs by allowing a prefixed proportion of CIs containing the values of uninteresting parameters. We theoretically prove that the proposed CIs are uniformly shorter than BY CIs and control the FCR asymptotically for independent data. Numerical results confirm our theoretical results and show that the proposed CIs still work for correlated data. We illustrate the advantage of the proposed procedures by analyzing the microarray data from a HIV study.     

Simultaneous small-sample multivariate Bernoulli confidence intervals

A technique based on the bootstrap is presented for assessing the simultaneous confidence level of k small-sample confidence intervals for multivariate Bernoulli marginal frequencies. The small-sample intervals used are those of Clopper and Pearson (1934, Biometrika 26, 404-413) and require iterative computation. To estimate the simultaneous confidence level, the multivariate Bernoulli vectors are resampled via the bootstrap and the Clopper-Pearson intervals recomputed on each pseudosample. The bootstrap estimate is then the proportion of times (computed via Monte Carlo) that all the k intervals computed by resampling contain the original sample frequencies. The technique is applied to single-sample HLA data.     

A study on confidence intervals for incremental cost-effectiveness ratios

In health policy and economics studies, the incremental cost-effectiveness ratio (ICER) has long been used to compare the economic consequences relative to the health benefits of therapies. Due to the skewed distributions of the costs and ICERs, much research has been done on how to obtain confidence intervals of ICERs, using either parametric or nonparametric methods, with or without the presence of censoring. In this paper, we will examine and compare the finite sample performance of many approaches via simulation studies. For the special situation when the health effect of the treatment is not statistically significant, we will propose a new bootstrapping approach to improve upon the bootstrap percentile method that is currently available. The most efficient way of constructing confidence intervals will be identified and extended to the censored data case. Finally, a data example from a cardiovascular clinical trial is used to demonstrate the application of these methods.     

Simultaneous confidence intervals for comparing binomial parameters

SUMMARY: To compare proportions with several independent binomial samples, we recommend a method of constructing simultaneous confidence intervals that uses the studentized range distribution with a score statistic. It applies to a variety of measures, including the difference of proportions, odds ratio, and relative risk. For the odds ratio, a simulation study suggests that the method has coverage probability closer to the nominal value than ad hoc approaches such as the Bonferroni implementation of Wald or "exact" small-sample pairwise intervals. It performs well even for the problematic but practically common case in which the binomial parameters are relatively small. For the difference of proportions, the proposed method has performance comparable to a method proposed by Piegorsch (1991, Biometrics 47, 45-52).     

Bootstrap confidence intervals for adaptive cluster sampling

Consider a collection of spatially clustered objects where the clusters are geographically rare. Of interest is estimation of the total number of objects on the site from a sample of plots of equal size. Under these spatial conditions, adaptive cluster sampling of plots is generally useful in improving efficiency in estimation over simple random sampling without replacement (SRSWOR). In adaptive cluster sampling, when a sampled plot meets some predefined condition, neighboring plots are added to the sample. When populations are rare and clustered, the usual unbiased estimators based on small samples are often highly skewed and discrete in distribution. Thus, confidence intervals based on asymptotic normal theory may not be appropriate. We investigated several nonparametric bootstrap methods for constructing confidence intervals under adaptive cluster sampling. To perform bootstrapping, we transformed the initial sample in order to include the information from the adaptive portion of the sample yet maintain a fixed sample size. In general, coverages of bootstrap percentile methods were closer to nominal coverage than the normal approximation.     

Stepwise confidence intervals for monotone dose-response studies

Summary .   In dose–response studies, one of the most important issues is the identification of the minimum effective dose (MED), where the MED is defined as the lowest dose such that the mean response is better than the mean response of a zero-dose control by a clinically significant difference. Dose–response curves are sometimes monotonic in nature. To find the MED, various authors have proposed step-down test procedures based on contrasts among the sample means. In this article, we improve upon the method of Marcus and Peritz (1976, Journal of the Royal Statistical Society, Series B 38 , 157–165) and implement the dose–response method of Hsu and Berger (1999, Journal of the American Statistical Association 94 , 468–482) to construct the lower confidence bound for the difference between the mean response of any nonzero-dose level and that of the control under the monotonicity assumption to identify the MED. The proposed method is illustrated by numerical examples, and simulation studies on power comparisons are presented.     

Unconditional small-sample confidence intervals for the odds ratio

The traditional approach to 'exact' small-sample interval estimation of the odds ratio for binomial, Poisson, or multinomial samples uses the conditional distribution to eliminate nuisance parameters. This approach can be very conservative. For two independent binomial samples, we study an unconditional approach with overall confidence level guaranteed to equal at least the nominal level. With small samples this interval tends to be shorter and have coverage probabilities nearer the nominal level.     

Measurement error and confidence intervals for ROC curves

Measurement error in a continuous test variable may bias estimates of the summary properties of receiver operating characteristics (ROC) curves. Typically, unbiased measurement error will reduce the diagnostic potential of a continuous test variable. This paper explores the effects of possibly heterogenous measurement error on estimated ROC curves for binormal test variables. Corrected estimators for specific points on the curve are derived under the assumption of known or estimated measurement variances for individual test results. These estimators and associated confidence intervals do not depend on normal assumptions for the distribution of the measurement error and are shown to be approximately unbiased for moderate size samples in a simulation study. An application from a study of emerging imaging modalities in breast cancer is used to demonstrate the new techniques.