Multiple comparisons distortions of parameter estimates

In experiments involving many variables, investigators typically use multiple comparisons procedures to determine differences that are unlikely to be the result of chance. However, investigators rarely consider how the magnitude of the greatest observed effect sizes may have been subject to bias resulting from multiple testing. These questions of bias become important to the extent investigators focus on the magnitude of the observed effects. As an example, such bias can lead to problems in attempting to validate results, if a biased effect size is used to power a follow-up study. An associated important consequence is that confidence intervals constructed using standard distributions may be badly biased. A bootstrap approach is used to estimate and adjust for the bias in the effect sizes of those variables showing strongest differences. This bias is not always present; some principles showing what factors may lead to greater bias are given and a proof of the convergence of the bootstrap distribution is provided.     

Sensitivity analyses informed by tests for bias in observational studies

In an observational study, the treatment received and the outcome exhibited may be associated in the absence of an effect caused by the treatment, even after controlling for observed covariates. Two tactics are common: (i) a test for unmeasured bias may be obtained using a secondary outcome for which the effect is known and (ii) a sensitivity analysis may explore the magnitude of unmeasured bias that would need to be present to explain the observed association as something other than an effect caused by the treatment. Can such a test for unmeasured bias inform the sensitivity analysis? If the test for bias does not discover evidence of unmeasured bias, then ask: Are conclusions therefore insensitive to larger unmeasured biases? Conversely, if the test for bias does find evidence of bias, then ask: What does that imply about sensitivity to biases? This problem is formulated in a new way as a convex quadratically constrained quadratic program and solved on a large scale using interior point methods by a modern solver. That is, a convex quadratic function of N variables is minimized subject to constraints on linear and convex quadratic functions of these variables. The quadratic function that is minimized is a statistic for the primary outcome that is a function of the unknown treatment assignment probabilities. The quadratic function that constrains this minimization is a statistic for subsidiary outcome that is also a function of these same unknown treatment assignment probabilities. In effect, the first statistic is minimized over a confidence set for the unknown treatment assignment probabilities supplied by the unaffected outcome. This process avoids the mistake of interpreting the failure to reject a hypothesis as support for the truth of that hypothesis. The method is illustrated by a study of the effects of light daily alcohol consumption on high-density lipoprotein (HDL) cholesterol levels. In this study, the method quickly optimizes a nonlinear function of $N=800$ variables subject to linear and quadratic constraints. In the example, strong evidence of unmeasured bias is found using the subsidiary outcome, but, perhaps surprisingly, this finding makes the primary comparison insensitive to larger biases.     

Evaluating SNP ascertainment bias and its impact on population assignment in Atlantic cod, Gadus morhua

The increasing use of single nucleotide polymorphisms (SNPs) in studies of nonmodel organisms accentuates the need to evaluate the influence of ascertainment bias on accurate ecological or evolutionary inference. Using a panel of 1641 expressed sequence tag-derived SNPs developed for northwest Atlantic cod (Gadus morhua), we examined the influence of ascertainment bias and its potential impact on assignment of individuals to populations ranging widely in origin. We hypothesized that reductions in assignment success would be associated with lower diversity in geographical regions outside the location of ascertainment. Individuals were genotyped from 13 locations spanning much of the contemporary range of Atlantic cod. Diversity, measured as average sample heterozygosity and number of polymorphic loci, declined (c. 30%) from the western (H(e) = 0.36) to eastern (H(e) = 0.25) Atlantic, consistent with a signal of ascertainment bias. Assignment success was examined separately for pools of loci representing differing degrees of reductions in diversity. SNPs displaying the largest declines in diversity produced the most accurate assignment in the ascertainment region (c. 83%) and the lowest levels of correct assignment outside the ascertainment region (c. 31%). Interestingly, several isolated locations showed no effect of assignment bias and consistently displayed 100% correct assignment. Contrary to expectations, estimates of accurate assignment range-wide using all loci displayed remarkable similarity despite reductions in diversity. Our results support the use of large SNP panels in assignment studies of high geneflow marine species. However, our evidence of significant reductions in assignment success using some pools of loci suggests that ascertainment bias may influence assignment results and should be evaluated in large-scale assignment studies.     

The Use of Adaptive Blocks in the Design of Clinical Trials

This paper introduces a class of data-dependent allocation rules for use in sequential clinical trials designed to choose the better of two competing treatments, or to decide that they are of equal efficacy. These readily understood and easily implemented rules are shown to reduce, substantially the number of tests with the poorer treatment for a broad category of experimental situations. Allocation rules of this type are applied both to trials with an instantaneous binomial response and to delayed response trials where interest centers on exponentially distributed survival time. In each case, a comparison of this design with alternative designs given in the literature shows that the proposed design is superior with respect to ease of application and is comparable to the alternatives regarding inferior treatment number and average sample number. In addition, the proposed rules mitigate many of the difficulties generally associated with adaptive assignment rules, such as selection and systematic bias.     

Assessing the power of informative subsets of loci for population assignment: standard methods are upwardly biased

It is well known that statistical classification procedures should be assessed using data that are separate from those used to train the classifier. This principle is commonly overlooked when the classification procedure in question is population assignment using a set of genetic markers that were chosen specifically on the basis of their allele frequencies from amongst a larger number of candidate markers. This oversight leads to a systematic upward bias in the predicted accuracy of the chosen set of markers for population assignment. Three widely used software programs for selecting markers informative for population assignment suffer from this bias. The extent of this bias is documented through a small set of simulations. The relative effect of the bias is largest when screening many candidate loci from poorly differentiated populations. Simple unbiased methods are presented and their use encouraged.     

Monte Carlo sensitivity analysis for unmeasured confounding in dynamic treatment regimes

Data-driven methods for personalizing treatment assignment have garnered much attention from clinicians and researchers. Dynamic treatment regimes formalize this through a sequence of decision rules that map individual patient characteristics to a recommended treatment. Observational studies are commonly used for estimating dynamic treatment regimes due to the potentially prohibitive costs of conducting sequential multiple assignment randomized trials. However, estimating a dynamic treatment regime from observational data can lead to bias in the estimated regime due to unmeasured confounding. Sensitivity analyses are useful for assessing how robust the conclusions of the study are to a potential unmeasured confounder. A Monte Carlo sensitivity analysis is a probabilistic approach that involves positing and sampling from distributions for the parameters governing the bias. We propose a method for performing a Monte Carlo sensitivity analysis of the bias due to unmeasured confounding in the estimation of dynamic treatment regimes. We demonstrate the performance of the proposed procedure with a simulation study and apply it to an observational study examining tailoring the use of antidepressant medication for reducing symptoms of depression using data from Kaiser Permanente Washington.     

A reanalysis of thyroid neoplasms in the Israeli tinea capitis study accounting for dose uncertainties

In the 1940s and 1950s, children in Israel were treated for tinea capitis by irradiation to the scalp to induce epilation. Follow-up studies of these patients and of other radiation- exposed populations show an increased risk of malignant and benign thyroid tumors. Those analyses, however, assume that thyroid dose for individuals is estimated precisely without error. Failure to account for uncertainties in dosimetry may affect standard errors and bias dose-response estimates. For the Israeli tinea capitis study, we discuss sources of uncertainties and adjust dosimetry for uncertainties in the prediction of true dose from X-ray treatment parameters. We also account for missing ages at exposure for patients with multiple X-ray treatments, since only ages at first treatment are known, and for missing data on treatment center, which investigators use to define exposure. Our reanalysis of the dose response for thyroid cancer and benign thyroid tumors indicates that uncertainties in dosimetry have minimal effects on dose-response estimation and for inference on the modifying effects of age at first exposure, time since exposure, and other factors. Since the components of the dose uncertainties we describe are likely to be present in other epidemiological studies of patients treated with radiation, our analysis may provide a model for considering the potential role of these uncertainties.     

Sex ratios observed in 80 species of parrots

A number of potential evolutionary and physiological factors may be involved in avian sex ratio bias so that under certain conditions a sex ratio bias may favour males or females within a population. In addition different factors may be important in manipulating sex ratio bias through the different life stages. In this study sex ratio bias was examined in a total of 16 570 captive parrots, representing 80 species, many of which are endangered in the wild, using database records originating form commercial laboratories that offer genetic sexing. Within the species examined 72% showed a male bias this was significant in three species, when adjusted for multiple comparisons. This preliminary study is limited due to lack of data on the age of the individuals sampled. However, the large dataset do suggest that this phenomenon should be further considered by investigators working at a species level where such data can be collected.     

Simple methods for the estimation and sensitivity analysis of principal strata effects using marginal structural models: Application to a bone fracture prevention trial

In randomized clinical trials, it is often of interest to estimate the effect of treatment on quality of life (QOL), in addition to those on the event itself. When an event occurs in some patients prior to QOL score assessment, investigators may compare QOL scores between patient subgroups defined by the event after randomization. However, owing to postrandomization selection bias, this analysis can mislead investigators about treatment efficacy and result in paradoxical findings. The recent Japanese Osteoporosis Intervention Trial (JOINT‐02), which compared the benefits of a combination therapy for fracture prevention with those of a monotherapy, exemplifies the case in point; the average QOL score was higher in the combination therapy arm for the unfractured subgroup but was lower for the fractured subgroup. To address this issue, principal strata effects (PSEs), which are treatment effects estimated within subgroups of individuals stratified by potential intermediate variable, have been discussed in the literature. In this paper, we describe a simple procedure for estimating the PSEs using marginal structural models. This procedure utilizes SAS code for the estimation. In addition, we present a simple sensitivity analysis method for examining the resulting estimates. The analyses of JOINT‐02 data using these methods revealed that QOL scores were higher in the combination therapy arm than in the monotherapy arm for both subgroups.     

Semiparametric Contrasts of Cumulative Pre-Treatment Mortality in the Presence of Dependent Censoring

In clinical settings, the necessity of treatment is often measured in terms of the patient’s prognosis in the absence of treatment. Along these lines, it is often of interest to compare subgroups of patients (e.g., based on underlying diagnosis) with respect to pre-treatment survival. Such comparisons may be complicated by at least two important issues. First, mortality contrasts by subgroup may differ over follow-up time, as opposed to being constant, and may follow a form that is difficult to model parametrically. Moreover, in settings where the proportional hazards assumption fails, investigators tend to be more interested in cumulative (as opposed to instantaneous) effects on mortality. Second, pre-treatment death is censored by the receipt of treatment and in settings where treatment assignment depends on time-dependent factors that also affect mortality, such censoring is likely to be informative. We propose semiparametric methods for contrasting subgroup-specific cumulative mortality in the presence of dependent censoring. The proposed estimators are based on the cumulative hazard function, with pre-treatment mortality assumed to follow a stratified Cox model. No functional form is assumed for the nature of the non-proportionality. Asymptotic properties of the proposed estimators are derived, and simulation studies show that the proposed methods are applicable to practical sample sizes. The methods are then applied to contrast pre-transplant mortality for acute versus chronic End-Stage Liver Disease patients.     

Effects of exercise on the biomechanical,biochemical and structural properties of tendons

Tendon has been shown to undergo remodeling in response to strength or endurance training, however, compared to muscle, studies of the effects of exercise on tendon are limited and the information is inconsistent. Exercise may influence the structure, chemical composition and/or mechanical properties of tendon. Studies that have examined mechanical changes of tendon in response to endurance training suggest that ultimate failure strength and stiffness increase with training. Available reports indicate that increases in tensile strength and stiffness are probably not associated with increases in collagen concentration or with tendon hypertrophy. The paucity of data renders it impossible to evaluate the response of other structural, chemical and mechanical parameters to training. Furthermore, few investigators have included discrete measures of structural, biomechanical and biochemical variables within a single study. The lack of integrative studies makes it difficult to definitively associate changes in the mechanical properties of tendon with chemical composition and structure.     

The prevalence of negative studies with inadequate statistical power: an analysis of the plastic surgery literature.

Studies published in the medical literature often neglect to consider the statistical power needed to detect a meaningful difference between study groups. Small sample sizes tend to produce negative results because of low statistical power. Studies that cannot make conclusive statements about their hypotheses can waste resources, deter further research, and impede advances in clinical treatment. The current study reviewed three of the most frequently read plastic surgery journals from 1976 to 1996 to determine the prevalence of inadequately (<80 percent) powered clinical trials and experimental studies that found no difference (negative studies) in the response variable of interest between comparison groups. The statistical power of 54 negative studies using continuous response variables was calculated to detect a difference of 1 SD (+/-1 SD) in means between the comparative groups. The power of another 57 negative studies with dichotomous response (yes/no) variables was calculated to detect a relative change in proportions of 25 percent and 50 percent from the experimental to the control group. It was found that 85 percent of the studies with continuous response variables had inadequate power to detect the desired mean difference of +/-1 SD. In studies with dichotomous response variables, 98 percent had inadequate power to detect a desired 25 percent relative change in proportions, and 74 percent had inadequate power to detect a desired 50 percent relative change in proportions. These results indicate that many of the studies in the plastic surgery literature lack adequate power to detect a moderate-to-large difference between groups. The lack of power makes the interpretation of the studies with negative findings inconclusive. Proper study design dictates that investigators consider a priori the difference between groups that is of clinical interest, and the sample size per group that is needed to provide adequate statistical power to detect the desired difference.     

Propensity Score Matching in Randomized Clinical Trials

Summary Cluster randomization trials with relatively few clusters have been widely used in recent years for evaluation of health‐care strategies. On average, randomized treatment assignment achieves balance in both known and unknown confounding factors between treatment groups, however, in practice investigators can only introduce a small amount of stratification and cannot balance on all the important variables simultaneously. The limitation arises especially when there are many confounding variables in small studies. Such is the case in the INSTINCT trial designed to investigate the effectiveness of an education program in enhancing the tPA use in stroke patients. In this article, we introduce a new randomization design, the balance match weighted (BMW) design, which applies the optimal matching with constraints technique to a prospective randomized design and aims to minimize the mean squared error (MSE) of the treatment effect estimator. A simulation study shows that, under various confounding scenarios, the BMW design can yield substantial reductions in the MSE for the treatment effect estimator compared to a completely randomized or matched‐pair design. The BMW design is also compared with a model‐based approach adjusting for the estimated propensity score and Robins‐Mark‐Newey E‐estimation procedure in terms of efficiency and robustness of the treatment effect estimator. These investigations suggest that the BMW design is more robust and usually, although not always, more efficient than either of the approaches. The design is also seen to be robust against heterogeneous error. We illustrate these methods in proposing a design for the INSTINCT trial.     

Emotional stress and diabetic control: a postulated model for the effect of emotional distress upon intermediary metabolism in the diabetic

The findings of a number of clinical investigators across a wide variety of situations and using a wide variety of observational procedures agree on the disruptive effects of emotional distress upon diabetic control. An integrative model of coordinated neural and hormonal reactions to emotional stress and their additive and potentiating effects on intermediary metabolism is proposed. The end result of these effects, as part of a general fight/flight or defense-alarm reaction, is a strong energy mobilization response (namely, a sharp rise in blood glucose and free fatty acid levels resulting in increased levels of serum cholesterol, triglycerides, and ketone bodies), which the diabetic cannot readily counterregulate. It is further proposed that increased diabetic control, together with lower insulin requirements, may result from the inclusion of stress management procedures as an adjunct to traditional medical treatment. The findings of a large number of investigators supporting these propositions are presented and discussed.     

Model misspecification in proportional hazards regression

The proportional hazards model is frequently used to evaluatethe effect of treatment on failure time events in randomisedclinical trials. Concomitant variables are usually availableand may be considered for use in the primary analyses underthe assumption that incorporating them may reduce bias or improveefficiency. In this paper we consider two approaches to includingcovariate information: regression modelling and stratification.We focus on the setting where covariate effects are nonproportionaland we compare the bias, efficiency and coverage propertiesof these approaches. These results indicate that our intuitionbased on linear model analysis of covariance is misleading.Covariate adjustment in proportional hazards models has littleeffect on the variance but may significantly improve the accuracyof the treatment effect estimator.     

Comparing the effects of continuous and discrete covariate mismeasurement,with emphasis on the dichotomization of mismeasured predictors

It is well known that imprecision in the measurement of predictor variables typically leads to bias in estimated regression coefficients. We compare the bias induced by measurement error in a continuous predictor with that induced by misclassification of a binary predictor in the contexts of linear and logistic regression. To make the comparison fair, we consider misclassification probabilities for a binary predictor that correspond to dichotomizing an imprecise continuous predictor in lieu of its precise counterpart. On this basis, nondifferential binary misclassification is seen to yield more bias than nondifferential continuous measurement error. However, it is known that differential misclassification results if a binary predictor is actually formed by dichotomizing a continuous predictor subject to nondifferential measurement error. When the postulated model linking the response and precise continuous predictor is correct, this differential misclassification is found to yield less bias than continuous measurement error, in contrast with nondifferential misclassification, i.e., dichotomization reduces the bias due to mismeasurement. This finding, however, is sensitive to the form of the underlying relationship between the response and the continuous predictor. In particular, we give a scenario where dichotomization involves a trade-off between model fit and misclassification bias. We also examine how the bias depends on the choice of threshold in the dichotomization process and on the correlation between the imprecise predictor and a second precise predictor.     

Stratified or Multicentre Analysis with Extended Mantel–Haenszel Statistics

Since its introduction in 1959 the ability of the classical Mantel-Haenszel (M–H) procedure for combining the odds ratios of a set of I 2 × 2 tables has led to its use also in stratified or multicentre type clinical trials. A familiar application is the M–H logrank test in survival analysis. An extension of the M–H procedure covering the case of 2 × K contingency tables (MANTEL , 1963) with ordered levels retains the essential property of pooling the results of I homogeneous tables (i.e. in absence of qualitative interactions). The assignment of some score for the K columns of a table is essential for the use of the method (in comparing 2 treatments). Some possibilities of score assignment are discussed: for clinical outcome variables such as the degree of severity of a disease, pain and so on, the score is at hand in a natural way. A less well-known type of scoring consists in ranking the observations of a continuous variable, leading to cell sizes of 1 or 0. In this case, however, if equidistant ranking was used, the E–M–H procedure appears as an extension of Wilcoxon's rank sum test and represents a powerful non-parametric approach in stratified or multicentre type designs with non normally distributed outcome variables. The results of some Monte-Carlo simulations for 2 possible equidistant ranking procedures are presented, which indicate only a moderate gain in power as compared to Wilcoxon's rank sum test under the common situation of centre effects not exceeding treatment effects. Use of the E–M–H pro?edure is also recommended as a simple method to overcome the potential bias due to unequally distributed prognostic factors among treatment groups.     

Use and misuse of the reduced major axis for line‐fitting

Many investigators use the reduced major axis (RMA) instead of ordinary least squares (OLS) to define a line of best fit for a bivariate relationship when the variable represented on the X‐axis is measured with error. OLS frequently is described as requiring the assumption that X is measured without error while RMA incorporates an assumption that there is error in X. Although an RMA fit actually involves a very specific pattern of error variance, investigators have prioritized the presence versus the absence of error rather than the pattern of error in selecting between the two methods. Another difference between RMA and OLS is that RMA is symmetric, meaning that a single line defines the bivariate relationship, regardless of which variable is X and which is Y, while OLS is asymmetric, so that the slope and resulting interpretation of the data are changed when the variables assigned to X and Y are reversed. The concept of error is reviewed and expanded from previous discussions, and it is argued that the symmetry‐asymmetry issue should be the criterion by which investigators choose between RMA and OLS. This is a biological question about the relationship between variables. It is determined by the investigator, not dictated by the pattern of error in the data. If X is measured with error but OLS should be used because the biological question is asymmetric, there are several methods available for adjusting the OLS slope to reflect the bias due to error. RMA is being used in many analyses for which OLS would be more appropriate. Am J Phys Anthropol, 2009. © 2009 Wiley‐Liss, Inc.     

Unanticipated consequences of logarithmic transformation in bivariate allometry

Parameters in the two-parameter allometric equation are commonly estimated by fitting a straight line to logarithmic transformations of the original data and by back-transforming the resulting model to the arithmetic scale. However, log transformation distorts the relationship between the predictor and response variables, and this distortion may be sufficient to lead unsuspecting investigators to analyze data that actually are unsuited for allometric research. Two data sets from the current literature are re-examined here to illustrate instances in which log transformation caused ugly data to look deceptively good. One of the investigations focused on the scaling of metabolism to body mass in evolutionary transitions from prokaryotic to protistan to metazoan levels of organization whereas the other addressed the scaling of intestines to body size in rodents. In both instances investigators were led to conclusions that are not supported by the original data. Problems of the sort described here can readily be avoided simply by performing preliminary graphical analysis of observations expressed in the original units and by validating the final model in the arithmetic domain.     

Methods for estimating the parameters of a linear model for ordered categorical data.

In many empirical analyses, the response of interest is categorical with an ordinal scale attached. Many investigators prefer to formulate a linear model, assigning scores to each category of the ordinal response and treating it as continuous. When the covariates are categorical, Haber (1985, Computational Statistics and Data Analysis 3, 1-10) has developed a method to obtain maximum likelihood (ML) estimates of the parameters of the linear model using Lagrange multipliers. However, when the covariates are continuous, the only method we found in the literature is ordinary least squares (OLS), performed under the assumption of homogeneous variance. The OLS estimates are unbiased and consistent but, since variance homogeneity is violated, the OLS estimates of variance can be biased and may not be consistent. We discuss a variance estimate (White, 1980, Econometrica 48, 817-838) that is consistent for the true variance of the OLS parameter estimates. The possible bias encountered by using the naive OLS variance estimate is discussed. An estimated generalized least squares (EGLS) estimator is proposed and its efficiency relative to OLS is discussed. Finally, an empirical comparison of OLS, EGLS, and ML estimators is made.