Sample size calculation for studies comparing binary outcomes using historical controls

In historical control trials (HCTs), the experimental therapy is compared with a control therapy that has been evaluated in a previously conducted trial. Makuch and Simon developed a sample size formula where the observations from the HC group were considered not subject to sampling variability. Many researchers have pointed out that the Makuch–Simon sample size formula does not preserve the nominal power and type I error. We develop a sample size calculation approach that properly accounts for the uncertainty in the true response rate of the HC group. We demonstrate that the empirical power and type I error, obtained over the simulated HC data, have extremely skewed distributions. We then derive a closed‐form sample size formula that enables researchers to control percentiles, instead of means, of the power and type I error accounting for the skewness of the distributions. A simulation study demonstrates that this approach preserves the operational characteristics in a more realistic scenario where the true response rate of the HC group is unknown. We also show that the controlling percentiles can be used to describe the joint behavior of the power and type I error. It provides a new perspective on the assessment of HCTs.