Systematic error detection in experimental high-throughput screening

Background  

High-throughput screening (HTS) is a key part of the drug discovery process during which thousands of chemical compounds are screened and their activity levels measured in order to identify potential drug candidates (i.e., hits). Many technical, procedural or environmental factors can cause systematic measurement error or inequalities in the conditions in which the measurements are taken. Such systematic error has the potential to critically affect the hit selection process. Several error correction methods and software have been developed to address this issue in the context of experimental HTS [1–7]. Despite their power to reduce the impact of systematic error when applied to error perturbed datasets, those methods also have one disadvantage - they introduce a bias when applied to data not containing any systematic error [6]. Hence, we need first to assess the presence of systematic error in a given HTS assay and then carry out systematic error correction method if and only if the presence of systematic error has been confirmed by statistical tests.     

Diet preferences based on sequence read count: The role of species interaction in tissue bias correction

High-throughput sequencing and metabarcoding techniques provide a unique opportunity to study predator–prey relationships. However, in animal dietary preference studies, how to properly correct tissue bias within the sequence read count and the role of interactions between co-occurring species in metabarcoding mixtures remain largely unknown. In this study, we propose two categories of tissue bias correction indices: sequence read count number per unit tissue (SCN) and its ratio form (SCN ratio). By constructing plant mock communities with different numbers of co-occurring species in metabarcoding mixtures and conducting feeding trails on captive sika deer (Cervus nippon), we demonstrate the features of the SCN and SCN ratio, evaluate their correction effects and assess the role of species interactions during tissue bias correction. Tissue differences between species are defined as the differential ability to generate sequence counts. Our study suggests that pure tissue differences among species without a species interaction is not an optimal correction index for many biomes with limited tissue differences among species. Species interactions in mixtures may amplify tissue differences, which is beneficial for tissue bias correction. However, caution must be taken because varied species interactions among communities may increase the risk of worse correction. Correction effects based on the SCN and SCN ratio are comparable, but the SCN is less influenced by control species than the SCN ratio. Based on our study, several suggestions are provided for future animal diet studies or other high-throughput sequencing studies containing tissue bias.     

Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments

In 2006, Peters et al. identified 86 systematic reviews (SRs) of laboratory animal experiments (LAEs). They found 46 LAE meta-analyses (MAs), often of poor quality. Six of these 46 MAs tried to assess publication bias. Publication bias is the phenomenon of an experiment's results determining its likelihood of publication, often over-representing positive findings. As such, publication bias is the Achilles heel of any SR. Since researchers increasingly become aware of the fact that SRs directly support the 'three Rs', we expect the number of SRs of LAEs will sharply increase. Therefore, it is useful to see how publication bias is dealt with. Our objective was to identify all SRs and MAs of LAEs where the purpose was to inform human health published between July 2005 and 2010 with special attention to MAs' quality features and publication bias. We systematically searched Medline, Embase, Toxline and ScienceDirect from July 2005 to 2010, updating Peters' review. LAEs not directly informing human health or concerning fundamental biology were excluded. We found 2780 references of which 163 met the inclusion criteria: 158 SRs, of which 30 performed an MA, and five MAs without an SR. The number of SRs roughly doubled every three years since 1997. The number of MAs roughly doubled every five years since 1999. Compared with before July 2005, more MAs were preceded by SR and reported on (quality) features of included studies and heterogeneity. A statistically significant proportion of MAs considered publication bias (26/35) and tried to formally assess it (21/35).     

When should one adjust for measurement error in baseline variables in observational studies?

Previously, we showed that in randomised experiments, correction for measurement error in a baseline variable induces bias in the estimated treatment effect, and conversely that ignoring measurement error avoids bias. In observational studies, non-zero baseline covariate differences between treatment groups may be anticipated. Using a graphical approach, we argue intuitively that if baseline differences are large, failing to correct for measurement error leads to a biased estimate of the treatment effect. In contrast, correction eliminates bias if the true and observed baseline differences are equal. If this equality is not satisfied, the corrected estimator is also biased, but typically less so than the uncorrected estimator. Contrasting these findings, we conclude that there must be a threshold for the true baseline difference, above which correction is worthwhile. We derive expressions for the bias of the corrected and uncorrected estimators, as functions of the correlation of the baseline variable with the study outcome, its reliability, the true baseline difference, and the sample sizes. Comparison of these expressions defines a theoretical decision threshold about whether to correct for measurement error. The results show that correction is usually preferred in large studies, and also in small studies with moderate baseline differences. If the group sample sizes are very disparate, correction is less advantageous. If the equivalent balanced sample size is less than about 25 per group, one should correct for measurement error if the true baseline difference is expected to exceed 0.2-0.3 standard deviation units. These results are illustrated with data from a cohort study of atherosclerosis.     

Segmented regression in the presence of covariate measurement error in main study/validation study designs

This article considers the problem of segmented regression in the presence of covariate measurement error in main study/validation study designs. First, we derive a closed and interpretable form for the full likelihood. After that, we use the likelihood results to compute the bias of the estimated changepoint in the case when the measurement error is ignored. We find the direction of the bias in the estimated changepoint to be determined by the design distribution of the observed covariates, and the bias can be in either direction. We apply the methodology to data from a nutritional study that investigates the relation between dietary folate and blood serum homocysteine levels and find that the analysis that ignores covariate measurement error would have indicated a much higher minimum daily dietary folate intake requirement than is obtained in the analysis that takes covariate measurement error into account.     

Publication bias and merit in ecology

Bias, or any set of factors that influence the general expression of merit, is common in science and is an inevitable by-product of an imperfect but otherwise reasonably objective human pursuit to understand the world we inhabit. In this paper, we explore the conceptual significance of a relatively tractable form of bias, namely publication and dissemination bias. A specific definition is developed, a working model of classification for publication bias is proposed, and an assessment of what we can measure is described. Finally, we offer expectations for ecologists with respect to the significance of bias in the publication process within our discipline. We argue that without explicit consideration of both the qualitative and quantitative aspects of publication bias in ecology, we limit our capacity to fairly assess and best use the science that we as a community produce.     

Meta-analysis of genetic association studies

Meta-analysis, a statistical tool for combining results across studies, is becoming popular as a method for resolving discrepancies in genetic association studies. Persistent difficulties in obtaining robust, replicable results in genetic association studies are almost certainly because genetic effects are small, requiring studies with many thousands of subjects to be detected. In this article, we describe how meta-analysis works and consider whether it will solve the problem of underpowered studies or whether it is another affliction visited by statisticians on geneticists. We show that meta-analysis has been successful in revealing unexpected sources of heterogeneity, such as publication bias. If heterogeneity is adequately recognized and taken into account, meta-analysis can confirm the involvement of a genetic variant, but it is not a substitute for an adequately powered primary study.     

Three-armed trials including placebo and no-treatment groups may be subject to publication bias: systematic review

Background

It has been argued that placebos may not have important clinical impacts in general. However, there is increasing evidence of a publication bias among trials published in journals. Therefore, we explored the potential for publication bias in randomized trials with active treatment, placebo, and no-treatment groups.

Methods

Three-armed randomized trials of acupuncture, acupoint stimulation, and transcutaneous electrical stimulation were obtained from electronic databases. Effect sizes between treatment and placebo groups were calculated for treatment effect, and effect sizes between placebo and no-treatment groups were calculated for placebo effect. All data were then analyzed for publication bias.

Results

For the treatment effect, small trials with fewer than 100 patients per arm showed more benefits than large trials with at least 100 patients per arm in acupuncture and acupoint stimulation. For the placebo effect, no differences were found between large and small trials. Further analyses showed that the treatment effect in acupuncture and acupoint stimulation may be subject to publication bias because study design and any known factors of heterogeneity were not associated with the small study effects. In the simulation, the magnitude of the placebo effect was smaller than that calculated after considering publication bias.

Conclusions

Randomized three-armed trials, which are necessary for estimating the placebo effect, may be subject to publication bias. If the magnitude of the placebo effect is assessed in an intervention, the potential for publication bias should be investigated using data related to the treatment effect.     

Correcting for measurement error in individual ancestry estimates in structured association tests

We present theoretical explanations and show through simulation that the individual admixture proportion estimates obtained by using ancestry informative markers should be seen as an error-contaminated measurement of the underlying individual ancestry proportion. These estimates can be used in structured association tests as a control variable to limit type I error inflation or reduce loss of power due to population stratification observed in studies of admixed populations. However, the inclusion of such error-containing variables as covariates in regression models can bias parameter estimates and reduce ability to control for the confounding effect of admixture in genetic association tests. Measurement error correction methods offer a way to overcome this problem but require an a priori estimate of the measurement error variance. We show how an upper bound of this variance can be obtained, present four measurement error correction methods that are applicable to this problem, and conduct a simulation study to compare their utility in the case where the admixed population results from the intermating between two ancestral populations. Our results show that the quadratic measurement error correction (QMEC) method performs better than the other methods and maintains the type I error to its nominal level.     

Revisiting the association between alcohol drinking and oral cancer in nonsmoking and betel quid non-chewing individuals

Background: Alcohol drinking is an oral cancer (OC) risk factor; tobacco smoking (TS) and betel quid chewing (BQC) are oral carcinogens and effect modifiers of drinking. Although the assessment of the independent effect of drinking on OC must necessarily account for effect modifiers, no observational study has included interaction terms between drinking, TS, BQC in regression analyses. In order to assess the independent association between drinking and OC, this pooled analysis focused on subjects who were not exposed to such effect modifiers. Methods: Case-control studies on OC, which discriminated non TS/non BQC drinkers from multiexposed drinkers were searched. Exposed subjects (≥1 drink daily, ≥10 years) were compared to unexposed subjects (non/occasional drinkers). Unadjusted odds ratios (ORs) were extracted/calculated. Pooled ORs were assessed with the random-effect method, which assumed high between-study heterogeneity (assessed with Cochran's Q). Robustness of estimates was investigated through use of adjusted ORs, correction for publication bias, sensitivity analysis to inclusion criteria. The drinking–TS interaction was assessed with the Interaction Contrast Ratio (ICR) and the Attributable Proportion due to Interaction (AP). Results: Sixteen studies were used, with substantially high heterogeneity. The pooled OR was 0.787 (95CI, 0.677–0.914). Use of adjusted ORs, correction for publication bias, sensitivity analysis corroborated these results. ICR and AP were 2.444 and 54.6%. Conclusions: Consistent with stratified analyses reporting non significant/negative associations between alcohol drinking and OC in non multiexposed subjects, an OC preventive activity of drinking is inferable. However, given the high prevalence and the oral carcinogenicity of concomitant drinking and smoking, drinking control policies remain essential.     

Biases incurred from nonrandom repeat testing of haemoglobin levels in blood donors: Selective testing and its implications

To help prevent anaemia, it is a requisite for blood donors to undergo a haemoglobin test to ensure levels are not too low before donation. It is therefore important to have an accurate testing device and strategy to ensure donors are not being inappropriately bled. A recent study in blood donors used a selective testing strategy where if a donor's haemoglobin level is below the level required for donation, then another reading is taken and if this occurs again, a third and final reading is used. This strategy can reduce the average number of readings required per donor compared to taking three measurements for all donors. However, the final decision‐making measurement will on average be higher than a single measurement. In this paper, a selective testing strategy is compared against other strategies. Individual‐level biases are derived for the selective strategy and are shown to depend on how close a donor's true haemoglobin level is to the donation threshold and the magnitude of error in the testing device. A simulation study was conducted using the distribution of haemoglobin levels from a large donor population to investigate the effects different strategies have on population performance. We consider scenarios based on varying the measurement device bias and error, including differential biases that depend on the underlying haemoglobin level. Discriminatory performance is shown to be affected when using the selective testing strategies, especially when measurement error is large and when differential bias is present in the device. We recommend that the average of a number of readings should be used in preference to selective testing strategies if multiple measurements are available.     

Human fluctuating asymmetry in relation to health and quality: a meta-analysis

Developmental instability (DI) reflects the inability of a developing organism to buffer its development against random perturbations, due either to frequent, large perturbations or to a poor buffering system. The primary measure used to assess DI experienced by an individual organism is fluctuating asymmetry (FA), asymmetry of bilateral features that are, on average in a population, symmetrical. A large literature on FA in humans in relation to measures of health and quality (close to 100 studies and nearly 300 individual effect size estimates) has accumulated. This paper presents the first quantitative meta-analysis of this literature. The mean effect size (scaled as Pearson r) was about 0.2. Effect sizes covaried negatively with sample size, consistent with effects of publication bias, the tendency for significant effects to be published. Conservative correction for this bias reduced the mean effect to about 0.1. Associations with FA underestimate effects of underlying DI due to imprecise measurement of the latter. A model-based best estimate of the mean effect of DI on outcomes is about 0.3, a theoretically meaningful, relatively large effect, albeit of moderate absolute size. The data are consistent, however, with a range of true effect sizes between 0.08 and 0.67, partly due to large study effects. Study-specific effect sizes in DI ranged between −0.2 and 1.0. A humbling and perhaps sobering conclusion is that, in spite of a large body of literature involving nearly 50 000 participants, we can only confidently state that there is on average a robust positive average effect size. An accurate estimate of that effect size was not possible, and between-study variation remained largely unexplained. We detected no robust variation across six broad categories of outcomes (health and disease, fetal outcomes, psychological maladaptation, reproduction, attractiveness and hormonal effects), though examination of narrower domains reveal some corrected effects close to 0.2 and others near zero. The meta-analysis suggests fruitful directions for future research and theory.     

Should Studies of Diabetes Treatment Stratification Correct for Baseline HbA1c?

AimsBaseline HbA1c is a major predictor of response to glucose lowering therapy and therefore a potential confounder in studies aiming to identify other predictors. However, baseline adjustment may introduce error if the association between baseline HbA1c and response is substantially due to measurement error and regression to the mean. We aimed to determine whether studies of predictors of response should adjust for baseline HbA1c.MethodsWe assessed the relationship between baseline HbA1c and glycaemic response in 257 participants treated with GLP-1R agonists and assessed whether it reflected measurement error and regression to the mean using duplicate ‘pre-baseline’ HbA1c measurements not included in the response variable. In this cohort and an additional 2659 participants treated with sulfonylureas we assessed the relationship between covariates associated with baseline HbA1c and treatment response with and without baseline adjustment, and with a bias correction using pre-baseline HbA1c to adjust for the effects of error in baseline HbA1c.ResultsBaseline HbA1c was a major predictor of response (R² = 0.19,β = -0.44,p<0.001).The association between pre-baseline and response was similar suggesting the greater response at higher baseline HbA1cs is not mainly due to measurement error and subsequent regression to the mean. In unadjusted analysis in both cohorts, factors associated with baseline HbA1c were associated with response, however these associations were weak or absent after adjustment for baseline HbA1c. Bias correction did not substantially alter associations.ConclusionsAdjustment for the baseline HbA1c measurement is a simple and effective way to reduce bias in studies of predictors of response to glucose lowering therapy.     

A cautionary comment on the generation of Berkson error in epidemiological studies

Exposure measurement error can be seen as one of the most important sources of uncertainty in studies in epidemiology. When the aim is to assess the effects of measurement error on statistical inference or to compare the performance of several methods for measurement error correction, it is indispensable to be able to generate different types of measurement error. This paper compares two approaches for the generation of Berkson error, which have recently been applied in radiation epidemiology, in their ability to generate exposure data that satisfy the properties of the Berkson model. In particular, it is shown that the use of one of the methods produces results that are not in accordance with two important properties of Berkson error.     

Phase Error Correction in Time-Averaged 3D Phase Contrast Magnetic Resonance Imaging of the Cerebral Vasculature

Purpose

Volume flow rate (VFR) measurements based on phase contrast (PC)-magnetic resonance (MR) imaging datasets have spatially varying bias due to eddy current induced phase errors. The purpose of this study was to assess the impact of phase errors in time averaged PC-MR imaging of the cerebral vasculature and explore the effects of three common correction schemes (local bias correction (LBC), local polynomial correction (LPC), and whole brain polynomial correction (WBPC)).

Methods

Measurements of the eddy current induced phase error from a static phantom were first obtained. In thirty healthy human subjects, the methods were then assessed in background tissue to determine if local phase offsets could be removed. Finally, the techniques were used to correct VFR measurements in cerebral vessels and compared statistically.

Results

In the phantom, phase error was measured to be <2.1 ml/s per pixel and the bias was reduced with the correction schemes. In background tissue, the bias was significantly reduced, by 65.6% (LBC), 58.4% (LPC) and 47.7% (WBPC) (p < 0.001 across all schemes). Correction did not lead to significantly different VFR measurements in the vessels (p = 0.997). In the vessel measurements, the three correction schemes led to flow measurement differences of -0.04 ± 0.05 ml/s, 0.09 ± 0.16 ml/s, and -0.02 ± 0.06 ml/s. Although there was an improvement in background measurements with correction, there was no statistical difference between the three correction schemes (p = 0.242 in background and p = 0.738 in vessels).

Conclusions

While eddy current induced phase errors can vary between hardware and sequence configurations, our results showed that the impact is small in a typical brain PC-MR protocol and does not have a significant effect on VFR measurements in cerebral vessels.     

Linear measurement error models with restricted sampling

The relationship between nutrient consumption and chronic disease risk is the focus of a large number of epidemiological studies where food frequency questionnaires (FFQ) and food records are commonly used to assess dietary intake. However, these self-assessment tools are known to involve substantial random error for most nutrients, and probably important systematic error as well. Study subject selection in dietary intervention studies is sometimes conducted in two stages. At the first stage, FFQ-measured dietary intakes are observed and at the second stage another instrument, such as a 4-day food record, is administered only to participants who have fulfilled a prespecified criterion that is based on the baseline FFQ-measured dietary intake (e.g., only those reporting percent energy intake from fat above a prespecified quantity). Performing analysis without adjusting for this truncated sample design and for the measurement error in the nutrient consumption assessments will usually provide biased estimates for the population parameters. In this work we provide a general statistical analysis technique for such data with the classical additive measurement error that corrects for the two sources of bias. The proposed technique is based on multiple imputation for longitudinal data. Results of a simulation study along with a sensitivity analysis are presented, showing the performance of the proposed method under a simple linear regression model.     

Helicobacter pylori infection and colorectal cancer risk: a meta-analysis

BACKGROUND: Several studies suggested an association between Helicobacter pylori infection and colorectal carcinoma or adenoma risk. However, different authors reported quite varying estimates. We carried out a systematic review and meta-analysis of published studies investigating this association and paid special attention to the possibility of publication bias and sources of heterogeneity between studies. Materials and METHODS: An extensive literature search and cross-referencing were performed to identify all published studies. Summary estimates were obtained using random-effects models. The presence of possible publication bias was assessed using different statistical approaches. RESULTS: In a meta-analysis of the 11 identified human studies, published between 1991 and 2002, a summary odds ratio of 1.4 (95% CI, 1.1-1.8) was estimated for the association between H. pylori infection and colorectal cancer risk. The graphical funnel plot appeared asymmetrical, but the formal statistical evaluations did not provide strong evidence of publication bias. The proportion of variation of study results because of heterogeneity was small (36.5%). CONCLUSIONS: The results of our meta-analysis are consistent with a possible small increase in risk of colorectal cancer because of H. pylori infection. However, the possibility of some publication bias cannot be ruled out, although it could not be statistically confirmed. Larger, better designed and better controlled studies are needed to clarify the situation.     

Testing small study effects in multivariate meta-analysis

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta-analyses. The most well-known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta-analysis have been extensively studied. However, detecting small study effects in a multivariate meta-analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta-analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.     

A measurement error model for time-series studies of air pollution and mortality

One barrier to interpreting the observational evidence concerning the adverse health effects of air pollution for public policy purposes is the measurement error inherent in estimates of exposure based on ambient pollutant monitors. Exposure assessment studies have shown that data from monitors at central sites may not adequately represent personal exposure. Thus, the exposure error resulting from using centrally measured data as a surrogate for personal exposure can potentially lead to a bias in estimates of the health effects of air pollution. This paper develops a multi-stage Poisson regression model for evaluating the effects of exposure measurement error on estimates of effects of particulate air pollution on mortality in time-series studies. To implement the model, we have used five validation data sets on personal exposure to PM10. Our goal is to combine data on the associations between ambient concentrations of particulate matter and mortality for a specific location, with the validation data on the association between ambient and personal concentrations of particulate matter at the locations where data have been collected. We use these data in a model to estimate the relative risk of mortality associated with estimated personal-exposure concentrations and make a comparison with the risk of mortality estimated with measurements of ambient concentration alone. We apply this method to data comprising daily mortality counts, ambient concentrations of PM10measured at a central site, and temperature for Baltimore, Maryland from 1987 to 1994. We have selected our home city of Baltimore to illustrate the method; the measurement error correction model is general and can be applied to other appropriate locations.Our approach uses a combination of: (1) a generalized additive model with log link and Poisson error for the mortality-personal-exposure association; (2) a multi-stage linear model to estimate the variability across the five validation data sets in the personal-ambient-exposure association; (3) data augmentation methods to address the uncertainty resulting from the missing personal exposure time series in Baltimore. In the Poisson regression model, we account for smooth seasonal and annual trends in mortality using smoothing splines. Taking into account the heterogeneity across locations in the personal-ambient-exposure relationship, we quantify the degree to which the exposure measurement error biases the results toward the null hypothesis of no effect, and estimate the loss of precision in the estimated health effects due to indirectly estimating personal exposures from ambient measurements.     

Ten categories of statistical errors: a guide for research in endocrinology and metabolism

A simple framework is introduced that defines ten categories of statistical errors on the basis of type of error, bias or imprecision, and source: sampling, measurement, estimation, hypothesis testing, and reporting. Each of these ten categories is illustrated with examples pertinent to research and publication in the disciplines of endocrinology and metabolism. Some suggested remedies are discussed, where appropriate. A review of recent issues of American Journal of Physiology: Endocrinology and Metabolism and of Endocrinology finds that very small sample sizes may be the most prevalent cause of statistical error in this literature.