Guidelines for the design and statistical analysis of experiments in papers submitted to ATLA

In vitro experiments need to be well designed and correctly analysed if they are to achieve their full potential to replace the use of animals in research. An "experiment" is a procedure for collecting scientific data in order to answer a hypothesis, or to provide material for generating new hypotheses, and differs from a survey because the scientist has control over the treatments that can be applied. Most experiments can be classified into one of a few formal designs, the most common being completely randomised, and randomised block designs. These are quite common with in vitro experiments, which are often replicated in time. Some experiments involve a single independent (treatment) variable, while other "factorial" designs simultaneously vary two or more independent variables, such as drug treatment and cell line. Factorial designs often provide additional information at little extra cost. Experiments need to be carefully planned to avoid bias, be powerful yet simple, provide for a valid statistical analysis and, in some cases, have a wide range of applicability. Virtually all experiments need some sort of statistical analysis in order to take account of biological variation among the experimental subjects. Parametric methods using the t test or analysis of variance are usually more powerful than non-parametric methods, provided the underlying assumptions of normality of the residuals and equal variances are approximately valid. The statistical analyses of data from a completely randomised design, and from a randomised-block design are demonstrated in Appendices 1 and 2, and methods of determining sample size are discussed in Appendix 3. Appendix 4 gives a checklist for authors submitting papers to ATLA.     

Importance of randomization in microarray experimental designs with Illumina platforms

Measurements of gene expression from microarray experiments are highly dependent on experimental design. Systematic noise can be introduced into the data at numerous steps. On Illumina BeadChips, multiple samples are assayed in an ordered series of arrays. Two experiments were performed using the same samples but different hybridization designs. An experiment confounding genotype with BeadChip and treatment with array position was compared to another experiment in which these factors were randomized to BeadChip and array position. An ordinal effect of array position on intensity values was observed in both experiments. We demonstrate that there is increased rate of false-positive results in the confounded design and that attempts to correct for confounded effects by statistical modeling reduce power of detection for true differential expression. Simple analysis models without post hoc corrections provide the best results possible for a given experimental design. Normalization improved differential expression testing in both experiments but randomization was the most important factor for establishing accurate results. We conclude that lack of randomization cannot be corrected by normalization or by analytical methods. Proper randomization is essential for successful microarray experiments.     

Restricted randomization designs in clinical trials.

Though therapeutic clinical trials are often categorized as using either "randomization" or "historical controls" as a basis for treatment evaluation, pure random assignment of treatments is rarely employed. Instead various restricted randomization designs are used. The restrictions include the balancing of treatment assignments over time and the stratification of the assignment with regard to covariates that may affect response. Restricted randomization designs for clinical trials differ from those of other experimental areas because patients arrive sequentially and a balanced design cannot be ensured. The major restricted randomization designs and arguments concerning the proper role of stratification are reviewed here. The effect of randomization restrictions on the validity of significance tests is discussed.     

An efficient sample size adaptation strategy with adjustment of randomization ratio

In clinical trials, sample size reestimation is a useful strategy for mitigating the risk of uncertainty in design assumptions and ensuring sufficient power for the final analysis. In particular, sample size reestimation based on unblinded interim effect size can often lead to sample size increase, and statistical adjustment is usually needed for the final analysis to ensure that type I error rate is appropriately controlled. In current literature, sample size reestimation and corresponding type I error control are discussed in the context of maintaining the original randomization ratio across treatment groups, which we refer to as “proportional increase.” In practice, not all studies are designed based on an optimal randomization ratio due to practical reasons. In such cases, when sample size is to be increased, it is more efficient to allocate the additional subjects such that the randomization ratio is brought closer to an optimal ratio. In this research, we propose an adaptive randomization ratio change when sample size increase is warranted. We refer to this strategy as “nonproportional increase,” as the number of subjects increased in each treatment group is no longer proportional to the original randomization ratio. The proposed method boosts power not only through the increase of the sample size, but also via efficient allocation of the additional subjects. The control of type I error rate is shown analytically. Simulations are performed to illustrate the theoretical results.     

Design and Analysis of Crossover Trials for Absorbing Binary Endpoints

Summary The crossover is a popular and efficient trial design used in the context of patient heterogeneity to assess the effect of treatments that act relatively quickly and whose benefit disappears with discontinuation. Each patient can serve as her own control as within‐individual treatment and placebo responses are compared. Conventional wisdom is that these designs are not appropriate for absorbing binary endpoints, such as death or HIV infection. We explore the use of crossover designs in the context of these absorbing binary endpoints and show that they can be more efficient than the standard parallel group design when there is heterogeneity in individuals' risks. We also introduce a new two‐period design where first period “survivors” are rerandomized for the second period. This design combines the crossover design with the parallel design and achieves some of the efficiency advantages of the crossover design while ensuring that the second period groups are comparable by randomization. We discuss the validity of the new designs and evaluate both a mixture model and a modified Mantel–Haenszel test for inference. The mixture model assumes no carryover or period effects while the Mantel–Haenszel approach conditions out period effects. Simulations are used to compare the different designs and an example is provided to explore practical issues in implementation.     

Guidelines for the design and statistical analysis of experiments using laboratory animals

For ethical and economic reasons, it is important to design animal experiments well, to analyze the data correctly, and to use the minimum number of animals necessary to achieve the scientific objectives---but not so few as to miss biologically important effects or require unnecessary repetition of experiments. Investigators are urged to consult a statistician at the design stage and are reminded that no experiment should ever be started without a clear idea of how the resulting data are to be analyzed. These guidelines are provided to help biomedical research workers perform their experiments efficiently and analyze their results so that they can extract all useful information from the resulting data. Among the topics discussed are the varying purposes of experiments (e.g., exploratory vs. confirmatory); the experimental unit; the necessity of recording full experimental details (e.g., species, sex, age, microbiological status, strain and source of animals, and husbandry conditions); assigning experimental units to treatments using randomization; other aspects of the experiment (e.g., timing of measurements); using formal experimental designs (e.g., completely randomized and randomized block); estimating the size of the experiment using power and sample size calculations; screening raw data for obvious errors; using the t-test or analysis of variance for parametric analysis; and effective design of graphical data.     

Role of ancillary variables in the design,analysis, and interpretation of animal experiments

During the course of an experiment using animals, many variables (e.g., age, body weight at several times, food and water consumption, hematology, and clinical biochemistry) and other characteristics are often recorded in addition to the primary response variable(s) specified by the experimenter. These additional variables have an important role in the design and interpretation of the experiment. They may be formally incorporated into the design and/or analysis and thus increase precision and power. However, even if these variables are not incorporated into the primary statistical design or into the formal analysis of the experiment, they may nevertheless be used in an ancillary or exploratory way to provide valuable information about the experiment, as shown by various examples. Used in this way, ancillary variables may improve analysis and interpretation by providing an assessment of the randomization process and an approach to the identification of outliers, lead to the generation of new hypotheses, and increase generality of results or account for differences in results when compared across different experiments. Thus, appropriate use of additional variables may lead to reduction in the number of animals required to achieve the aims of the experiment and may provide additional scientific information as an extra benefit. Unfortunately, this type of information is sometimes effectively discarded because its potential value is not recognized. Guidelines for use of animals include, in addition to the obligation to follow humane procedures, the obligation to use no more animals than necessary. Ethical experimental practice thus requires that all information be properly used and reported.     

Bayesian Adaptive Randomization and Trial Monitoring with Predictive Probability for Time-to-Event Endpoint

There has been much development in Bayesian adaptive designs in clinical trials. In the Bayesian paradigm, the posterior predictive distribution characterizes the future possible outcomes given the currently observed data. Based on the interim time-to-event data, we develop a new phase II trial design by combining the strength of both Bayesian adaptive randomization and the predictive probability. By comparing the mean survival times between patients assigned to two treatment arms, more patients are assigned to the better treatment on the basis of adaptive randomization. We continuously monitor the trial using the predictive probability for early termination in the case of superiority or futility. We conduct extensive simulation studies to examine the operating characteristics of four designs: the proposed predictive probability adaptive randomization design, the predictive probability equal randomization design, the posterior probability adaptive randomization design, and the group sequential design. Adaptive randomization designs using predictive probability and posterior probability yield a longer overall median survival time than the group sequential design, but at the cost of a slightly larger sample size. The average sample size using the predictive probability method is generally smaller than that of the posterior probability design.     

Blinded and unblinded sample size reestimation in crossover trials balanced for period

The determination of the sample size required by a crossover trial typically depends on the specification of one or more variance components. Uncertainty about the value of these parameters at the design stage means that there is often a risk a trial may be under‐ or overpowered. For many study designs, this problem has been addressed by considering adaptive design methodology that allows for the re‐estimation of the required sample size during a trial. Here, we propose and compare several approaches for this in multitreatment crossover trials. Specifically, regulators favor reestimation procedures to maintain the blinding of the treatment allocations. We therefore develop blinded estimators for the within and between person variances, following simple or block randomization. We demonstrate that, provided an equal number of patients are allocated to sequences that are balanced for period, the proposed estimators following block randomization are unbiased. We further provide a formula for the bias of the estimators following simple randomization. The performance of these procedures, along with that of an unblinded approach, is then examined utilizing three motivating examples, including one based on a recently completed four‐treatment four‐period crossover trial. Simulation results show that the performance of the proposed blinded procedures is in many cases similar to that of the unblinded approach, and thus they are an attractive alternative.     

A Bayesian model with application for adaptive platform trials having temporal changes

Temporal changes exist in clinical trials. Over time, shifts in patients' characteristics, trial conduct, and other features of a clinical trial may occur. In typical randomized clinical trials, temporal effects, that is, the impact of temporal changes on clinical outcomes and study analysis, are largely mitigated by randomization and usually need not be explicitly addressed. However, temporal effects can be a serious obstacle for conducting clinical trials with complex designs, including the adaptive platform trials that are gaining popularity in recent medical product development. In this paper, we introduce a Bayesian robust prior for mitigating temporal effects based on a hidden Markov model, and propose a particle filtering algorithm for computation. We conduct simulation studies to evaluate the performance of the proposed method and provide illustration examples based on trials of Ebola virus disease therapeutics and hemostat in vascular surgery.     

Quantifying the magnitude of baseline covariate imbalances resulting from selection bias in randomized clinical trials

Selection bias is most common in observational studies, when patients select their own treatments or treatments are assigned based on patient characteristics, such as disease severity. This first-order selection bias, as we call it, is eliminated by randomization, but there is residual selection bias that may occur even in randomized trials which occurs when, subconsciously or otherwise, an investigator uses advance knowledge of upcoming treatment allocations as the basis for deciding whom to enroll. For example, patients more likely to respond may be preferentially enrolled when the active treatment is due to be allocated, and patients less likely to respond may be enrolled when the control group is due to be allocated. If the upcoming allocations can be observed in their entirety, then we will call the resulting selection bias second-order selection bias. Allocation concealment minimizes the ability to observe upcoming allocations, yet upcoming allocations may still be predicted (imperfectly), or even determined with certainty, if at least some of the previous allocations are known, and if restrictions (such as randomized blocks) were placed on the randomization. This mechanism, based on prediction but not observation of upcoming allocations, is the third-order selection bias that is controlled by perfectly successful masking, but without perfect masking is not controlled even by the combination of advance randomization and allocation concealment. Our purpose is to quantify the magnitude of baseline imbalance that can result from third-order selection bias when the randomized block procedure is used. The smaller the block sizes, the more accurately one can predict future treatment assignments in the same block as known previous assignments, so this magnitude will depend on the block size, as well as on the level of certainty about upcoming allocations required to bias the patient selection. We find that a binary covariate can, on average, be up to 50% unbalanced by third-order selection bias.     

Bayesian Two-Stage Biomarker-Based Adaptive Design for Targeted Therapy Development

We propose a Bayesian two-stage biomarker-based adaptive randomization (AR) design for the development of targeted agents. The design has three main goals: (1) to test the treatment efficacy, (2) to identify prognostic and predictive markers for the targeted agents, and (3) to provide better treatment for patients enrolled in the trial. To treat patients better, both stages are guided by the Bayesian AR based on the individual patient’s biomarker profiles. The AR in the first stage is based on a known marker. A Go/No-Go decision can be made in the first stage by testing the overall treatment effects. If a Go decision is made at the end of the first stage, a two-step Bayesian lasso strategy will be implemented to select additional prognostic or predictive biomarkers to refine the AR in the second stage. We use simulations to demonstrate the good operating characteristics of the design, including the control of per-comparison type I and type II errors, high probability in selecting important markers, and treating more patients with more effective treatments. Bayesian adaptive designs allow for continuous learning. The designs are particularly suitable for the development of multiple targeted agents in the quest of personalized medicine. By estimating treatment effects and identifying relevant biomarkers, the information acquired from the interim data can be used to guide the choice of treatment for each individual patient enrolled in the trial in real time to achieve a better outcome. The design is being implemented in the BATTLE-2 trial in lung cancer at the MD Anderson Cancer Center.     

Design and analysis of phase I clinical trials

The Phase I clinical trial is a study intended to estimate the so-called maximum tolerable dose (MTD) of a new drug. Although there exists more or less a standard type of design for such trials, its development has been largely ad hoc. As usually implemented, the trial design has no intrinsic property that provides a generally satisfactory basis for estimation of the MTD. In this paper, the standard design and several simple alternatives are compared with regard to the conservativeness of the design and with regard to point and interval estimation of an MTD (33rd percentile) with small sample sizes. Using a Markov chain representation, we found several designs to be nearly as conservative as the standard design in terms of the proportion of patients entered at higher dose levels. In Monte Carlo simulations, two two-stage designs are found to provide reduced bias in maximum likelihood estimation of the MTD in less than ideal dose-response settings. Of the three methods considered for determining confidence intervals--the delta method, a method based on Fieller's theorem, and a likelihood ratio method--none was able to provide both usefully narrow intervals and coverage probabilities close to nominal.     

A comparison of experimental designs for selection in breeding trials with nested treatment structure

Plant breeders frequently evaluate large numbers of entries in field trials for selection. Generally, the tested entries are related by pedigree. The simplest case is a nested treatment structure, where entries fall into groups or families such that entries within groups are more closely related than between groups. We found that some plant breeders prefer to plant close relatives next to each other in the field. This contrasts with common experimental designs such as the α-design, where entries are fully randomized. A third design option is to randomize in such a way that entries of the same group are separated as much as possible. The present paper compares these design options by simulation. Another important consideration is the type of model used for analysis. Most of the common experimental designs were optimized assuming that the model used for analysis has fixed treatment effects. With many entries that are related by pedigree, analysis based on a model with random treatment effects becomes a competitive alternative. In simulations, we therefore study the properties of best linear unbiased predictions (BLUP) of genetic effects based on a nested treatment structure under these design options for a range of genetic parameters. It is concluded that BLUP provides efficient estimates of genetic effects and that resolvable incomplete block designs such as the α-design with restricted or unrestricted randomization can be recommended.     

R. A. Fisher and his advocacy of randomization

The requirement of randomization in experimental design was first stated by R. A. Fisher, statistician and geneticist, in 1925 in his book Statistical Methods for Research Workers. Earlier designs were systematic and involved the judgment of the experimenter; this led to possible bias and inaccurate interpretation of the data. Fisher’s dictum was that randomization eliminates bias and permits a valid test of significance. Randomization in experimenting had been used by Charles Sanders Peirce in 1885 but the practice was not continued. Fisher developed his concepts of randomizing as he considered the mathematics of small samples, in discussions with “Student,” William Sealy Gosset. Fisher published extensively. His principles of experimental design were spread worldwide by the many “voluntary workers” who came from other institutions to Rothamsted Agricultural Station in England to learn Fisher’s methods.     

The devil lies in details: reply to Stuart Hurlbert

As pointed out in Stuart Hurlbert's recent article, ecologists still at times design their experiments sloppily, creating a situation where various forms of ‘non‐demonic intrusion’ could account for the documented contrasts between treatments and controls. If such contrasts are nevertheless presented to the reader as if they were statistically demonstrated treatment effects, then pseudoreplication is not a pseudoissue and the use of a stigmatizing label of is entirely warranted, as pointed out by Hurlbert. The problems with Hurlbert's concepts start in the context of studies, where the scope of the experiment is to provoke a chain of dramatic and a priori extremely unlikely events, which a given conjecture predicts to happen as a consequence of a given manipulation. As the essence of these experiments is to trigger large dynamical responses in a biological system, they often require much space and/or special conditions, allowing for efficient isolation of the experimental system from potential sources of contamination. These constraints can be incompatible with standard designs (randomization, replication and treatment‐control interspersion). In the context of experiments, where it has been necessary to sacrifice randomization, replication or treatment‐control interspersion, the logic of inferring treatment effects is the same as used when interpreting causes of spontaneous events or events triggered by manipulations with practical purposes. The observed contrasts can be reasonably interpreted as effects of the treatment if and only if their magnitudes and the timing of their emergence makes alternative explanations utterly implausible (which is up to the reader to judge). If the logic of inference is clearly explained and no claim of statistically demonstrated treatment effect is made, the use of stigmatizing labels like ‘pseudoreplication’ is unwarranted. However, it might clarify the literature if such imperfectly designed experiments are referred to as experimental events, to be distinguished from perfectly designed experiments, where mechanical interpretation of contrasts between treatments and controls as treatment effects can be regarded as socially acceptable.     

A new u-statistic with superior design sensitivity in matched observational studies

Summary In an observational or nonrandomized study of treatment effects, a sensitivity analysis indicates the magnitude of bias from unmeasured covariates that would need to be present to alter the conclusions of a naïve analysis that presumes adjustments for observed covariates suffice to remove all bias. The power of sensitivity analysis is the probability that it will reject a false hypothesis about treatment effects allowing for a departure from random assignment of a specified magnitude; in particular, if this specified magnitude is “no departure” then this is the same as the power of a randomization test in a randomized experiment. A new family of u‐statistics is proposed that includes Wilcoxon's signed rank statistic but also includes other statistics with substantially higher power when a sensitivity analysis is performed in an observational study. Wilcoxon's statistic has high power to detect small effects in large randomized experiments—that is, it often has good Pitman efficiency—but small effects are invariably sensitive to small unobserved biases. Members of this family of u‐statistics that emphasize medium to large effects can have substantially higher power in a sensitivity analysis. For example, in one situation with 250 pair differences that are Normal with expectation 1/2 and variance 1, the power of a sensitivity analysis that uses Wilcoxon's statistic is 0.08 while the power of another member of the family of u‐statistics is 0.66. The topic is examined by performing a sensitivity analysis in three observational studies, using an asymptotic measure called the design sensitivity, and by simulating power in finite samples. The three examples are drawn from epidemiology, clinical medicine, and genetic toxicology.     

Inference from Stratified Samples: Study Design,Bias and Graphical Model Representations

Stratification is a widely used strategy in empirical research to improve efficiency of the sampling design. One concern of stratification is that ignoring it on analysis may bias the relationship between variables. A weighted analysis can only be carried out when sampling weights are known. When these are unknown, valid inference on the relationship between variables then depends on the ignorability of the design, which may be difficult to establish. Here, graphical representations of multivariate dependencies and independencies are used to find necessary conditions for ignorability of stratified sampling designs for inference on conditional and marginal relationships between variables.     

Using Inverse Probability Bootstrap Sampling to Eliminate Sample Induced Bias in Model Based Analysis of Unequal Probability Samples

In ecology, as in other research fields, efficient sampling for population estimation often drives sample designs toward unequal probability sampling, such as in stratified sampling. Design based statistical analysis tools are appropriate for seamless integration of sample design into the statistical analysis. However, it is also common and necessary, after a sampling design has been implemented, to use datasets to address questions that, in many cases, were not considered during the sampling design phase. Questions may arise requiring the use of model based statistical tools such as multiple regression, quantile regression, or regression tree analysis. However, such model based tools may require, for ensuring unbiased estimation, data from simple random samples, which can be problematic when analyzing data from unequal probability designs. Despite numerous method specific tools available to properly account for sampling design, too often in the analysis of ecological data, sample design is ignored and consequences are not properly considered. We demonstrate here that violation of this assumption can lead to biased parameter estimates in ecological research. In addition, to the set of tools available for researchers to properly account for sampling design in model based analysis, we introduce inverse probability bootstrapping (IPB). Inverse probability bootstrapping is an easily implemented method for obtaining equal probability re-samples from a probability sample, from which unbiased model based estimates can be made. We demonstrate the potential for bias in model-based analyses that ignore sample inclusion probabilities, and the effectiveness of IPB sampling in eliminating this bias, using both simulated and actual ecological data. For illustration, we considered three model based analysis tools—linear regression, quantile regression, and boosted regression tree analysis. In all models, using both simulated and actual ecological data, we found inferences to be biased, sometimes severely, when sample inclusion probabilities were ignored, while IPB sampling effectively produced unbiased parameter estimates.     

Surveying Hard-to-Reach Groups Through Sampled Respondents in a Social Network

The sampling frame in most social science surveys misses members of certain groups, such as the homeless or individuals living with HIV. These groups are known as hard-to-reach groups. One strategy for learning about these groups, or subpopulations, involves reaching hard-to-reach group members through their social network. In this paper we compare the efficiency of two common methods for subpopulation size estimation using data from standard surveys. These designs are examples of mental link tracing designs. These designs begin with a randomly sampled set of network members (nodes) and then reach other nodes indirectly through questions asked to the sampled nodes. Mental link tracing designs cost significantly less than traditional link tracing designs, yet introduce additional sources of potential bias. We examine the influence of one such source of bias using simulation studies. We then demonstrate our findings using data from the General Social Survey collected in 2004 and 2006. Additionally, we provide survey design suggestions for future surveys incorporating such designs.