When should one adjust for measurement error in baseline variables in observational studies?

Previously, we showed that in randomised experiments, correction for measurement error in a baseline variable induces bias in the estimated treatment effect, and conversely that ignoring measurement error avoids bias. In observational studies, non-zero baseline covariate differences between treatment groups may be anticipated. Using a graphical approach, we argue intuitively that if baseline differences are large, failing to correct for measurement error leads to a biased estimate of the treatment effect. In contrast, correction eliminates bias if the true and observed baseline differences are equal. If this equality is not satisfied, the corrected estimator is also biased, but typically less so than the uncorrected estimator. Contrasting these findings, we conclude that there must be a threshold for the true baseline difference, above which correction is worthwhile. We derive expressions for the bias of the corrected and uncorrected estimators, as functions of the correlation of the baseline variable with the study outcome, its reliability, the true baseline difference, and the sample sizes. Comparison of these expressions defines a theoretical decision threshold about whether to correct for measurement error. The results show that correction is usually preferred in large studies, and also in small studies with moderate baseline differences. If the group sample sizes are very disparate, correction is less advantageous. If the equivalent balanced sample size is less than about 25 per group, one should correct for measurement error if the true baseline difference is expected to exceed 0.2-0.3 standard deviation units. These results are illustrated with data from a cohort study of atherosclerosis.     

One-dimensional analyses of Rapoport's rule reviewed through meta-analysis

Aim  To analyse quantitatively the extent to which several methodological, geographical and taxonomic variables affect the magnitude of the tendency for the latitudinal ranges of species to increase with latitude (the Rapoport effect).
Location  Global.
Methods  A meta-analysis of 49 published studies was used to evaluate the effect of several methodological and biological moderator variables on the magnitude of the pattern.
Results  The method used to depict the latitudinal variation in range sizes is a strong moderator variable that accounts for differences in the magnitude of the pattern. In contrast, the extent of the study or the use of areal or linear estimations of range sizes does not affect the magnitude of the pattern. The effect of geography is more consistent than the effect of taxonomy in accounting for differences in the magnitude of the pattern. The Rapoport effect is indeed strong in Eurasia and North America. Weaker or non-significant latitudinal trends are found at the global scale, and in Australia, South America and the New World. There are no significant differences in the magnitude of the pattern between different habitats, however, the overall pattern is weaker in oceans than in terrestrial regions of the world.
Main conclusions  The Rapoport effect is indeed strong in continental landmasses of the Northern Hemisphere. The magnitude of the effect is primarily affected by methodological and biogeographical factors. Ecological and spatial scale effects seem to be less important. We suggest that not all methodological approaches may be equally useful for analysing the pattern.     

Bias in estimates of quantitative-trait-locus effect in genome scans: demonstration of the phenomenon and a method-of-moments procedure for reducing bias

An attractive feature of variance-components methods (including the Haseman-Elston tests) for the detection of quantitative-trait loci (QTL) is that these methods provide estimates of the QTL effect. However, estimates that are obtained by commonly used methods can be biased for several reasons. Perhaps the largest source of bias is the selection process. Generally, QTL effects are reported only at locations where statistically significant results are obtained. This conditional reporting can lead to a marked upward bias. In this article, we demonstrate this bias and show that its magnitude can be large. We then present a simple method-of-moments (MOM)-based procedure to obtain more-accurate estimates, and we demonstrate its validity via Monte Carlo simulation. Finally, limitations of the MOM approach are noted, and we discuss some alternative procedures that may also reduce bias.     

Bias in the effect of habitat structure on pitfall traps: An experimental evaluation

Habitat structure has been implicated as a source of bias for pitfall-trap data but most evidence is observational or anecdotal. This study used an experimental approach to quantify biases due to habitat structure. In a randomized block design, I manipulated native grassland to create three types of habitat structure and measured pitfall-trap catches of grassland ants. Small patches of modified habitat were surrounded by otherwise unmodified grassland with the assumption that population density remained unaffected by the modification and that the effects observed were due to changes in trappability. I assessed magnitude, direction, predictability, and consistency of bias for the following types of data: population abundance for single species, relative abundance among species, species composition of assemblages, and species richness. The magnitude of the bias in population abundance was large for most species. However, since the direction of the bias varied predictably with habitat structure, pitfall-trap data can be used to judge differences in population abundance in some situations. The magnitude of the bias in relative abundance was less than for abundance. However, there was inconsistency in the direction and magnitude of bias among species. Thus, interpretation of relative abundance data in pitfall-trap studies may be compromised. Species richness and species composition were biased by habitat structure but were affected significantly only when the groundcover was very dense, suggesting a threshold effect of habitat structure. To help to interpret survey data, pitfall-trap studies should routinely measure attributes of habitat structure and incorporate an experimental component to characterize the bias.     

REPLICATED ORIGIN OF FEMALE‐BIASED ADULT SEX RATIO IN INTRODUCED POPULATIONS OF THE TRINIDADIAN GUPPY (POECILIA RETICULATA)

There are many theoretical and empirical studies explaining variation in offspring sex ratio but relatively few that explain variation in adult sex ratio. Adult sex ratios are important because biased sex ratios can be a driver of sexual selection and will reduce effective population size, affecting population persistence and shapes how populations respond to natural selection. Previous work on guppies (Poecilia reticulata) gives mixed results, usually showing a female‐biased adult sex ratio. However, a detailed analysis showed that this bias varied dramatically throughout a year and with no consistent sex bias. We used a mark‐recapture approach to examine the origin and consistency of female‐biased sex ratio in four replicated introductions. We show that female‐biased sex ratio arises predictably and is a consequence of higher male mortality and longer female life spans with little effect of offspring sex ratio. Inconsistencies with previous studies are likely due to sampling methods and sampling design, which should be less of an issue with mark‐recapture techniques. Together with other long‐term mark‐recapture studies, our study suggests that bias in offspring sex ratio rarely contributes to adult sex ratio in vertebrates. Rather, sex differences in adult survival rates and longevity determine vertebrate adult sex ratio.     

Three-armed trials including placebo and no-treatment groups may be subject to publication bias: systematic review

Background

It has been argued that placebos may not have important clinical impacts in general. However, there is increasing evidence of a publication bias among trials published in journals. Therefore, we explored the potential for publication bias in randomized trials with active treatment, placebo, and no-treatment groups.

Methods

Three-armed randomized trials of acupuncture, acupoint stimulation, and transcutaneous electrical stimulation were obtained from electronic databases. Effect sizes between treatment and placebo groups were calculated for treatment effect, and effect sizes between placebo and no-treatment groups were calculated for placebo effect. All data were then analyzed for publication bias.

Results

For the treatment effect, small trials with fewer than 100 patients per arm showed more benefits than large trials with at least 100 patients per arm in acupuncture and acupoint stimulation. For the placebo effect, no differences were found between large and small trials. Further analyses showed that the treatment effect in acupuncture and acupoint stimulation may be subject to publication bias because study design and any known factors of heterogeneity were not associated with the small study effects. In the simulation, the magnitude of the placebo effect was smaller than that calculated after considering publication bias.

Conclusions

Randomized three-armed trials, which are necessary for estimating the placebo effect, may be subject to publication bias. If the magnitude of the placebo effect is assessed in an intervention, the potential for publication bias should be investigated using data related to the treatment effect.     

Prediction error estimation: a comparison of resampling methods

MOTIVATION: In genomic studies, thousands of features are collected on relatively few samples. One of the goals of these studies is to build classifiers to predict the outcome of future observations. There are three inherent steps to this process: feature selection, model selection and prediction assessment. With a focus on prediction assessment, we compare several methods for estimating the 'true' prediction error of a prediction model in the presence of feature selection. RESULTS: For small studies where features are selected from thousands of candidates, the resubstitution and simple split-sample estimates are seriously biased. In these small samples, leave-one-out cross-validation (LOOCV), 10-fold cross-validation (CV) and the .632+ bootstrap have the smallest bias for diagonal discriminant analysis, nearest neighbor and classification trees. LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis. Additionally, LOOCV, 5- and 10-fold CV, and the .632+ bootstrap have the lowest mean square error. The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios. Differences in performance among resampling methods are reduced as the number of specimens available increase. SUPPLEMENTARY INFORMATION: A complete compilation of results and R code for simulations and analyses are available in Molinaro et al. (2005) (http://linus.nci.nih.gov/brb/TechReport.htm).     

Locus-specific heritability estimation via the bootstrap in linkage scans for quantitative trait loci

BACKGROUND/AIMS: In genome-wide linkage analysis of quantitative trait loci (QTL), locus-specific heritability estimates are biased when the original data are used to both localize linkage and estimate effects, due to maximization of the LOD score over the genome. Positive bias is increased by adoption of stringent significance levels to control genome-wide type I error. We propose multi-locus bootstrap resampling estimators for bias reduction in the situation in which linkage peaks at more than one QTL are of interest. METHODS: Bootstrap estimates were based on repeated sample splitting in the original dataset. We conducted simulation studies in nuclear families with 0 to 5 QTLs and applied the methods in a genome-wide analysis of a blood pressure phenotype in extended pedigrees from the Framingham Heart Study (FHS). RESULTS: Compared to na?ve estimates in the original simulation samples, bootstrap estimates had reduced bias and smaller mean squared error. In the FHS pedigrees, the bootstrap yielded heritability estimates as much as 70% smaller than in the original sample. CONCLUSIONS: Because effect estimates obtained in an initial study are typically inflated relative to those expected in an independent replication study, successful replication will be more likely when sample size requirements are based on bias-reduced estimates.     

Adapting prediction error estimates for biased complexity selection in high-dimensional bootstrap samples

The bootstrap is a tool that allows for efficient evaluation of prediction performance of statistical techniques without having to set aside data for validation. This is especially important for high-dimensional data, e.g., arising from microarrays, because there the number of observations is often limited. For avoiding overoptimism the statistical technique to be evaluated has to be applied to every bootstrap sample in the same manner it would be used on new data. This includes a selection of complexity, e.g., the number of boosting steps for gradient boosting algorithms. Using the latter, we demonstrate in a simulation study that complexity selection in conventional bootstrap samples, drawn with replacement, is severely biased in many scenarios. This translates into a considerable bias of prediction error estimates, often underestimating the amount of information that can be extracted from high-dimensional data. Potential remedies for this complexity selection bias, such as alternatively using a fixed level of complexity or of using sampling without replacement are investigated and it is shown that the latter works well in many settings. We focus on high-dimensional binary response data, with bootstrap .632+ estimates of the Brier score for performance evaluation, and censored time-to-event data with .632+ prediction error curve estimates. The latter, with the modified bootstrap procedure, is then applied to an example with microarray data from patients with diffuse large B-cell lymphoma.     

An approximately unbiased test of phylogenetic tree selection

An approximately unbiased (AU) test that uses a newly devised multiscale bootstrap technique was developed for general hypothesis testing of regions in an attempt to reduce test bias. It was applied to maximum-likelihood tree selection for obtaining the confidence set of trees. The AU test is based on the theory of Efron et al. (Proc. Natl. Acad. Sci. USA 93:13429-13434; 1996), but the new method provides higher-order accuracy yet simpler implementation. The AU test, like the Shimodaira-Hasegawa (SH) test, adjusts the selection bias overlooked in the standard use of the bootstrap probability and Kishino-Hasegawa tests. The selection bias comes from comparing many trees at the same time and often leads to overconfidence in the wrong trees. The SH test, though safe to use, may exhibit another type of bias such that it appears conservative. Here I show that the AU test is less biased than other methods in typical cases of tree selection. These points are illustrated in a simulation study as well as in the analysis of mammalian mitochondrial protein sequences. The theoretical argument provides a simple formula that covers the bootstrap probability test, the Kishino-Hasegawa test, the AU test, and the Zharkikh-Li test. A practical suggestion is provided as to which test should be used under particular circumstances.     

General independent censoring in event-driven trials with staggered entry

Randomized clinical trials with time-to-event endpoints are frequently stopped after a prespecified number of events has been observed. This practice leads to dependent data and nonrandom censoring, which can in general not be solved by conditioning on the underlying baseline information. In case of staggered study entry, matters are complicated substantially. The present paper demonstrates that the study design at hand entails general independent censoring in the counting process sense, provided that the analysis is based on study time information only. To illustrate that the filtrations must not use abundant information, we simulated data of event-driven trials and evaluated them by means of Cox regression models with covariates for the calendar times. The Breslow curves of the cumulative baseline hazard showed considerable deviations, which implies that the analysis is disturbed by conditioning on the calendar time variables. A second simulation study further revealed that Efron's classical bootstrap, unlike the (martingale-based) wild bootstrap, may lead to biased results in the given setting, as the assumption of random censoring is violated. This is exemplified by an analysis of data on immunotherapy in patients with advanced, previously treated nonsmall cell lung cancer.     

Substitution rates in Drosophila nuclear genes: implications for translational selection

The relationships between synonymous and nonsynonymous substitution rates and between synonymous rate and codon usage bias are important to our understanding of the roles of mutation and selection in the evolution of Drosophila genes. Previous studies used approximate estimation methods that ignore codon bias. In this study we reexamine those relationships using maximum-likelihood methods to estimate substitution rates, which accommodate the transition/transversion rate bias and codon usage bias. We compiled a sample of homologous DNA sequences at 83 nuclear loci from Drosophila melanogaster and at least one other species of Drosophila. Our analysis was consistent with previous studies in finding that synonymous rates were positively correlated with nonsynonymous rates. Our analysis differed from previous studies, however, in that synonymous rates were unrelated to codon bias. We therefore conducted a simulation study to investigate the differences between approaches. The results suggested that failure to properly account for multiple substitutions at the same site and for biased codon usage by approximate methods can lead to an artifactual correlation between synonymous rate and codon bias. Implications of the results for translational selection are discussed.     

Publication bias in ecology and evolution: an empirical assessment using the 'trim and fill' method

Recent reviews of specific topics, such as the relationship between male attractiveness to females and fluctuating asymmetry or attractiveness and the expression of secondary sexual characters, suggest that publication bias might be a problem in ecology and evolution. In these cases, there is a significant negative correlation between the sample size of published studies and the magnitude or strength of the research findings (formally the ‘effect size’). If all studies that are conducted are equally likely to be published, irrespective of their findings, there should not be a directional relationship between effect size and sample size; only a decrease in the variance in effect size as sample size increases due to a reduction in sampling error. One interpretation of these reports of negative correlations is that studies with small sample sizes and weaker findings (smaller effect sizes) are less likely to be published. If the biological literature is systematically biased this could undermine the attempts of reviewers to summarise actual biology relationships by inflating estimates of average effect sizes. But how common is this problem? And does it really effect the general conclusions of literature reviews? Here, we examine data sets of effect sizes extracted from 40 peer‐reviewed, published meta‐analyses. We estimate how many studies are missing using the newly developed ‘trim and fill’ method. This method uses asymmetry in plots of effect size against sample size (‘funnel plots’) to detect ‘missing’ studies. For random‐effect models of meta‐analysis 38% (15/40) of data sets had a significant number of ‘missing’ studies. After correcting for potential publication bias, 21% (8/38) of weighted mean effects were no longer significantly greater than zero, and 15% (5/34) were no longer statistically robust when we used random‐effects models in a weighted meta‐analysis. The mean correlation between sample size and the magnitude of standardised effect size was also significantly negative (r_s=‐0.20, P < 0‐0001). Individual correlations were significantly negative (P < 0.10) in 35% (14/40) of cases. Publication bias may therefore effect the main conclusions of at least 15–21% of meta‐analyses. We suggest that future literature reviews assess the robustness of their main conclusions by correcting for potential publication bias using the ‘trim and fill’ method.     

Signaling diversity of mu- and delta- opioid receptor ligands: Re-evaluating the benefits of β-arrestin/G protein signaling bias

Opioid analgesics are elective for treating moderate to severe pain but their use is restricted by severe side effects. Signaling bias has been proposed as a viable means for improving this situation. To exploit this opportunity, continuous efforts are devoted to understand how ligand-specific modulations of receptor functions could mediate the different in vivo effects of opioids. Advances in the field have led to the development of biased agonists based on hypotheses that allocated desired and undesired effects to specific signaling pathways. However, the prevalent hypothesis associating β-arrestin to opioid side effects was recently challenged and multiple of the newly developed biased drugs may not display the superior side effects profile that was sought. Moreover, biased agonism at opioid receptors is now known to be time- and cell-dependent, which adds a new layer of complexity for bias estimation. Here, we first review the signaling mechanisms underlying desired and undesired effects of opioids. We then describe biased agonism at opioid receptors and discuss the different perspectives that support the desired and undesired effects of opioids in view of exploiting biased signaling for therapeutic purposes. Finally, we explore how signaling kinetics and cellular background can influence the magnitude and directionality of bias at those receptors.     

The effects of small sample size and sample bias on threshold selection and accuracy assessment of species distribution models

Species distribution models are used for a range of ecological and evolutionary questions, but often are constructed from few and/or biased species occurrence records. Recent work has shown that the presence‐only model Maxent performs well with small sample sizes. While the apparent accuracy of such models with small samples has been studied, less emphasis has been placed on the effect of small or biased species records on the secondary modeling steps, specifically accuracy assessment and threshold selection, particularly with profile (presence‐only) modeling techniques. When testing the effects of small sample sizes on distribution models, accuracy assessment has generally been conducted with complete species occurrence data, rather than similarly limited (e.g. few or biased) test data. Likewise, selection of a probability threshold – a selection of probability that classifies a model into discrete areas of presences and absences – has also generally been conducted with complete data. In this study we subsampled distribution data for an endangered rodent across multiple years to assess the effects of different sample sizes and types of bias on threshold selection, and examine the differences between apparent and actual accuracy of the models. Although some previously recommended threshold selection techniques showed little difference in threshold selection, the most commonly used methods performed poorly. Apparent model accuracy calculated from limited data was much higher than true model accuracy, but the true model accuracy was lower than it could have been with a more optimal threshold. That is, models with thresholds and accuracy calculated from biased and limited data had inflated reported accuracy, but were less accurate than they could have been if better data on species distribution were available and an optimal threshold were used.     

Observations below multiple lower limits of quantification: How to estimate the mean and variance

Multiple lower limits of quantification (MLOQs) result if various laboratories are involved in the analysis of concentration data and some observations are too low to be quantified. For normally distributed data under MLOQs there exists only the multiple regression method of Helsel to estimate the mean and variance. We propose a simple imputation method and two new maximum likelihood estimation methods: the multiple truncated sample method and the multiple censored sample method. A simulation study is conducted to compare the performances of the newly introduced methods to Helsel's via the criteria root mean squared error (RMSE) and bias of the parameter estimates. Two and four lower limits of quantification (LLOQs), various amounts of unquantifiable observations and two sample sizes are studied. Furthermore, the robustness is investigated under model misspecification. The methods perform with decreasing accuracy for increasing rates of unquantified observations. Increasing sample sizes lead to smaller bias. There is almost no change in the performance between two and four LLOQs. The magnitude of the variance impairs the performance of all methods. For a smaller variance, the multiple censored sample method leads to superior estimates regarding the RMSE and bias, whereas Helsel's method is superior regarding the bias for a larger variance. Under model misspecification, Helsel's method was inferior to the other methods. Estimating the mean, the multiple censored sample method performed better, whereas the multiple truncated sample method performs best in estimating the variance. Summarizing, for a large sample size and normally distributed data we recommend to use Helsel's method. Otherwise, the multiple censored sample method should be used to obtain estimates of the mean and variance of data including MLOQs.     

Predicting and correcting bias caused by measurement error in line transect sampling using multiplicative error models

Line transect sampling is one of the most widely used methods for animal abundance assessment. Standard estimation methods assume certain detection on the transect, no animal movement, and no measurement errors. Failure of the assumptions can cause substantial bias. In this work, the effect of error measurement on line transect estimators is investigated. Based on considerations of the process generating the errors, a multiplicative error model is presented and a simple way of correcting estimates based on knowledge of the error distribution is proposed. Using beta models for the error distribution, the effect of errors and of the proposed correction is assessed by simulation. Adequate confidence intervals for the corrected estimates are obtained using a bootstrap variance estimate for the correction and the delta method. As noted by Chen (1998, Biometrics 54, 899-908), even unbiased estimators of the distances might lead to biased density estimators, depending on the actual error distribution. In contrast with the findings of Chen, who used an additive model, unbiased estimation of distances, given a multiplicative model, lead to overestimation of density. Some error distributions result in observed distance distributions that make efficient estimation impossible, by removing the shoulder present in the original detection function. This indicates the need to improve field methods to reduce measurement error. An application of the new methods to a real data set is presented.     

Categorical variables with many categories are preferentially selected in bootstrap‐based model selection procedures for multivariable regression models

Automated variable selection procedures, such as backward elimination, are commonly employed to perform model selection in the context of multivariable regression. The stability of such procedures can be investigated using a bootstrap‐based approach. The idea is to apply the variable selection procedure on a large number of bootstrap samples successively and to examine the obtained models, for instance, in terms of the inclusion of specific predictor variables. In this paper, we aim to investigate a particular important problem affecting this method in the case of categorical predictor variables with different numbers of categories and to give recommendations on how to avoid it. For this purpose, we systematically assess the behavior of automated variable selection based on the likelihood ratio test using either bootstrap samples drawn with replacement or subsamples drawn without replacement from the original dataset. Our study consists of extensive simulations and a real data example from the NHANES study. Our main result is that if automated variable selection is conducted on bootstrap samples, variables with more categories are substantially favored over variables with fewer categories and over metric variables even if none of them have any effect. Importantly, variables with no effect and many categories may be (wrongly) preferred to variables with an effect but few categories. We suggest the use of subsamples instead of bootstrap samples to bypass these drawbacks.     

Effects of sample size on estimates of population growth rates calculated with matrix models

Background

Matrix models are widely used to study the dynamics and demography of populations. An important but overlooked issue is how the number of individuals sampled influences estimates of the population growth rate (λ) calculated with matrix models. Even unbiased estimates of vital rates do not ensure unbiased estimates of λ–Jensen''s Inequality implies that even when the estimates of the vital rates are accurate, small sample sizes lead to biased estimates of λ due to increased sampling variance. We investigated if sampling variability and the distribution of sampling effort among size classes lead to biases in estimates of λ.

Methodology/Principal Findings

Using data from a long-term field study of plant demography, we simulated the effects of sampling variance by drawing vital rates and calculating λ for increasingly larger populations drawn from a total population of 3842 plants. We then compared these estimates of λ with those based on the entire population and calculated the resulting bias. Finally, we conducted a review of the literature to determine the sample sizes typically used when parameterizing matrix models used to study plant demography.

Conclusions/Significance

We found significant bias at small sample sizes when survival was low (survival = 0.5), and that sampling with a more-realistic inverse J-shaped population structure exacerbated this bias. However our simulations also demonstrate that these biases rapidly become negligible with increasing sample sizes or as survival increases. For many of the sample sizes used in demographic studies, matrix models are probably robust to the biases resulting from sampling variance of vital rates. However, this conclusion may depend on the structure of populations or the distribution of sampling effort in ways that are unexplored. We suggest more intensive sampling of populations when individual survival is low and greater sampling of stages with high elasticities.     

Bias in phylogenetic tree reconciliation methods: implications for vertebrate genome evolution

Background

Comparative genomic studies are revealing frequent gains and losses of whole genes via duplication and pseudogenization. One commonly used method for inferring the number and timing of gene gains and losses reconciles the gene tree for each gene family with the species tree of the taxa considered. Recent studies using this approach have found a large number of ancient duplications and recent losses among vertebrate genomes.

Results

I show that tree reconciliation methods are biased when the inferred gene tree is not correct. This bias places duplicates towards the root of the tree and losses towards the tips of the tree. I demonstrate that this bias is present when tree reconciliation is conducted on both multiple mammal and Drosophila genomes, and that lower bootstrap cut-off values on gene trees lead to more extreme bias. I also suggest a method for dealing with reconciliation bias, although this method only corrects for the number of gene gains on some branches of the species tree.

Conclusion

Based on the results presented, it is likely that most tree reconciliation analyses show biases, unless the gene trees used are exceptionally well-resolved and well-supported. These results cast doubt upon previous conclusions that vertebrate genome history has been marked by many ancient duplications and many recent gene losses.