Irreproducible text‐book “knowledge”: The effects of color bands on zebra finch fitness

Many fields of science—including behavioral ecology—currently experience a heated debate about the extent to which publication bias against null findings results in a misrepresentative scientific literature. Here, we show a case of an extreme mismatch between strong positive support for an effect in the literature and a failure to detect this effect across multiple attempts at replication. For decades, researchers working with birds have individually marked their study species with colored leg bands. For the zebra finch Taeniopygia guttata, a model organism in behavioral ecology, many studies over the past 35 years have reported effects of bands of certain colors on male or female attractiveness and further on behavior, physiology, life history, and fitness. Only eight of 39 publications presented exclusively null findings. Here, we analyze the results of eight experiments in which we quantified the fitness of a total of 730 color‐banded individuals from four captive populations (two domesticated and two recently wild derived). This sample size exceeds the combined sample size of all 23 publications that clearly support the “color‐band effect” hypothesis. We found that band color explains no variance in either male or female fitness. We also found no heterogeneity in color‐band effects, arguing against both context and population specificity. Analysis of unpublished data from three other laboratories strengthens the generality of our null finding. Finally, a meta‐analysis of previously published results is indicative of selective reporting and suggests that the effect size approaches zero when sample size is large. We argue that our field—and science in general—would benefit from more effective means to counter confirmation bias and publication bias.     

Conservative Tests under Satisficing Models of Publication Bias

Publication bias leads consumers of research to observe a selected sample of statistical estimates calculated by producers of research. We calculate critical values for statistical significance that could help to adjust after the fact for the distortions created by this selection effect, assuming that the only source of publication bias is file drawer bias. These adjusted critical values are easy to calculate and differ from unadjusted critical values by approximately 50%—rather than rejecting a null hypothesis when the t-ratio exceeds 2, the analysis suggests rejecting a null hypothesis when the t-ratio exceeds 3. Samples of published social science research indicate that on average, across research fields, approximately 30% of published t-statistics fall between the standard and adjusted cutoffs.     

Replication,Communication, and the Population Dynamics of Scientific Discovery

Many published research results are false (Ioannidis, 2005), and controversy continues over the roles of replication and publication policy in improving the reliability of research. Addressing these problems is frustrated by the lack of a formal framework that jointly represents hypothesis formation, replication, publication bias, and variation in research quality. We develop a mathematical model of scientific discovery that combines all of these elements. This model provides both a dynamic model of research as well as a formal framework for reasoning about the normative structure of science. We show that replication may serve as a ratchet that gradually separates true hypotheses from false, but the same factors that make initial findings unreliable also make replications unreliable. The most important factors in improving the reliability of research are the rate of false positives and the base rate of true hypotheses, and we offer suggestions for addressing each. Our results also bring clarity to verbal debates about the communication of research. Surprisingly, publication bias is not always an obstacle, but instead may have positive impacts—suppression of negative novel findings is often beneficial. We also find that communication of negative replications may aid true discovery even when attempts to replicate have diminished power. The model speaks constructively to ongoing debates about the design and conduct of science, focusing analysis and discussion on precise, internally consistent models, as well as highlighting the importance of population dynamics.     

Reanalysis of epidemiological evidence on lung cancer and passive smoking

ObjectiveTo assess the epidemiological evidence for an increase in the risk of lung cancer resulting from exposure to environmental tobacco smoke.DesignReanalysis of 37 published epidemiological studies previously included in a meta-analysis allowing for the possibility of publication bias.ResultsIf it is assumed that all studies that have ever been carried out are included, or that those selected for review are truly representative of all such studies, then the estimated excess risk of lung cancer is 24%, as previously reported (95% confidence interval 13% to 36%, P<0.001). However, a significant correlation between study outcome and study size suggests the presence of publication bias. Adjustment for such bias implies that the risk has been overestimated. For example, if only 60% of studies have been included, the estimate of excess risk falls from 24% to 15%.ConclusionA modest degree of publication bias leads to a substantial reduction in the relative risk and to a weaker level of significance, suggesting that the published estimate of the increased risk of lung cancer associated with environmental tobacco smoke needs to be interpreted with caution.

Key messages

• A systematic review of epidemiological studies on passive smoking estimated the increased risk of lung cancer as 24%
• There is clear evidence of publication bias in these studies
• Reanalysis of the data allowing for the possibility of publication bias substantially lowers the estimate of relative risk

     

Dissemination biases in ecology: effect sizes matter more than quality

Publication and citation decisions in ecology are likely influenced by many factors, potentially including journal impact factors, direction and magnitude of reported effects, and year of publication. Dissemination bias exists when publication or citation of a study depends on any of these factors. We defined several dissemination biases and determined their prevalence across many sub‐disciplines in ecology, then determined whether or not data quality also affected these biases. We identified dissemination biases in ecology by conducting a meta‐analysis of citation trends for 3867 studies included in 52 meta‐analyses. We correlated effect size, year of publication, impact factor and citation rate within each meta‐analysis. In addition, we explored how data quality as defined in meta‐analyses (sample size or variance) influenced each form of bias. We also explored how the direction of the predicted or observed effect, and the research field, influenced any biases. Year of publication did not influence citation rates. The first papers published in an area reported the strongest effects, and high impact factor journals published the most extreme effects. Effect size was more important than data quality for many publication and citation trends. Dissemination biases appear common in ecology, and although their magnitude was generally small many were associated with theory tenacity, evidenced as tendencies to cite papers that most strongly support our ideas. The consequences of this behavior are amplified by the fact that papers reporting strong effects were often of lower data quality than papers reporting much weaker effects. Furthermore, high impact factor journals published the strongest effects, generally in the absence of any correlation with data quality. Increasing awareness of the prevalence of theory tenacity, confirmation bias, and the inattention to data quality among ecologists is a first step towards reducing the impact of these biases on research in our field.     

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making.

Methodology/Principal Findings

In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Three-armed trials including placebo and no-treatment groups may be subject to publication bias: systematic review

Background

It has been argued that placebos may not have important clinical impacts in general. However, there is increasing evidence of a publication bias among trials published in journals. Therefore, we explored the potential for publication bias in randomized trials with active treatment, placebo, and no-treatment groups.

Methods

Three-armed randomized trials of acupuncture, acupoint stimulation, and transcutaneous electrical stimulation were obtained from electronic databases. Effect sizes between treatment and placebo groups were calculated for treatment effect, and effect sizes between placebo and no-treatment groups were calculated for placebo effect. All data were then analyzed for publication bias.

Results

For the treatment effect, small trials with fewer than 100 patients per arm showed more benefits than large trials with at least 100 patients per arm in acupuncture and acupoint stimulation. For the placebo effect, no differences were found between large and small trials. Further analyses showed that the treatment effect in acupuncture and acupoint stimulation may be subject to publication bias because study design and any known factors of heterogeneity were not associated with the small study effects. In the simulation, the magnitude of the placebo effect was smaller than that calculated after considering publication bias.

Conclusions

Randomized three-armed trials, which are necessary for estimating the placebo effect, may be subject to publication bias. If the magnitude of the placebo effect is assessed in an intervention, the potential for publication bias should be investigated using data related to the treatment effect.     

Worldwide Topology of the Scientific Subject Profile: A Macro Approach in the Country Level

Background

Models for the production of knowledge and systems of innovation and science are key elements for characterizing a country in view of its scientific thematic profile. With regard to scientific output and publication in journals of international visibility, the countries of the world may be classified into three main groups according to their thematic bias.

Methodology/Principal Findings

This paper aims to classify the countries of the world in several broad groups, described in terms of behavioural models that attempt to sum up the characteristics of their systems of knowledge and innovation. We perceive three clusters in our analysis: 1) the biomedical cluster, 2) the basic science & engineering cluster, and 3) the agricultural cluster. The countries are conceptually associated with the clusters via Principal Component Analysis (PCA), and a Multidimensional Scaling (MDS) map with all the countries is presented.

Conclusions/Significance

As we have seen, insofar as scientific output and publication in journals of international visibility is concerned, the countries of the world may be classified into three main groups according to their thematic profile. These groups can be described in terms of behavioral models that attempt to sum up the characteristics of their systems of knowledge and innovation.     

Expression of survivin and patients survival in non-small cell lung cancer: a meta-analysis of the published studies

Among new biological markers that could become useful prognostic factors for non-small cell lung cancer (NSCLC). Survivin is one of the most commonly over-expressed oncogenes, however, its role in NSCLC remains controversial. We performed a systematic review of the literature with meta-analysis to clarify this issue. Electronic databases were used to identify published studies before August 2011. Pooled hazard ratio (HR) with 95 % confidence interval (95 % CI) was used to estimate the strength of the association of survivin expression with survival of NSCLC patients. Heterogeneity and publication bias were also assessed. Overall 29 relevant published studies including 2,517 lung cancer patients were identified from electronic databases. We found that overexpression of survivin in NSCLC patients might be a poor prognostic factor for survival 1.95 (95 % CI: 1.65-2.29; P < 0.001). Heterogeneity testing indicated that there was heterogeneity among studies. When stratified by histology types, the heterogeneity was absent. We should point out that the publication bias may partly account for the result, but the conclusion might not be affected deeply by the publication bias. When we accounted for publication bias using the trim and fill method, the results remained significant (HR = 1.71, 95 % CI: 1.44–2.02, P < 0.001), suggesting the stability of our results. Therefore, our study suggested that survivin overexpression had a poor prognosis value in patients with NSCLC.     

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.

Methodology/Principal Findings

We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology

Temporal changes in the magnitude of research findings have recently been recognized as a general phenomenon in ecology, and have been attributed to the delayed publication of non-significant results and disconfirming evidence. Here we introduce a method of cumulative meta-analysis which allows detection of both temporal trends and publication bias in the ecological literature. To illustrate the application of the method, we used two datasets from recently conducted meta-analyses of studies testing two plant defence theories. Our results revealed three phases in the evolution of the treatment effects. Early studies strongly supported the hypothesis tested, but the magnitude of the effect decreased considerably in later studies. In the latest studies, a trend towards an increase in effect size was observed. In one of the datasets, a cumulative meta-analysis revealed publication bias against studies reporting disconfirming evidence; such studies were published in journals with a lower impact factor compared to studies with results supporting the hypothesis tested. Correlation analysis revealed neither temporal trends nor evidence of publication bias in the datasets analysed. We thus suggest that cumulative meta-analysis should be used as a visual aid to detect temporal trends and publication bias in research findings in ecology in addition to the correlative approach.     

生态学与医学中的整合分析(Meta-analysis)

整合分析是针对一系列独立研究结果进行定量综合分析的方法。自从 1976年 Glass在心理学研究中提出以来 ,该方法已经在许多学科特别是医学领域进行了广泛的应用。 2 0世纪 90年代 ,整合分析被引入生态学研究 ,引起了生态学家和统计学家的广泛关注。在我国 ,该方法也于 1998年被引入生态学。由于生态学研究自身的特点 ,整合分析在应用时出现了许多新问题 ,如不同研究类型的数据抽提与转换、效应值的构建、研究间相关性的估计、出版偏见的评估与修正等 ,为此以整合分析应用最活跃的医学领域进行对比和借鉴 ,分析该方法在两个研究领域应用的范围和特点 ,讨论影响其在生态学中应用的各种因素 ,并着重阐述和探讨其在生态学应用中存在的问题和发展前景     

Biosecurity and the review and publication of dual-use research of concern

Dual-use research of concern (DURC) is scientific research with significant potential for generating information that could be used to harm national security, the public health, or the environment. Editors responsible for journal policies and publication decisions play a vital role in ensuring that effective safeguards exist to cope with the risks of publishing scientific research with dual-use implications. We conducted an online survey of 127 chief editors of life science journals in 27 countries to examine their attitudes toward and experience with the review and publication of dual-use research of concern. Very few editors (11) had experience with biosecurity review, and no editor in our study reported having ever refused a submission on biosecurity grounds. Most respondents (74.8%) agreed that editors have a responsibility to consider biosecurity risks during the review process, but little consensus existed among editors on how to handle specific issues in the review and publication of research with potential dual-use implications. More work is needed to establish consensus on standards for the review and publication of dual-use research of concern in life science journals.     

A quantitative synthesis of pollen supplementation experiments highlights the contribution of resource reallocation to estimates of pollen limitation

Our understanding of pollen limitation depends on a realistic view of its magnitude. Previous reviews of pollen supplementation experiments concluded that a majority of plant species suffers from pollen limitation and that its magnitude is high. Here, we perform a meta-analysis and find evidence that publication bias, experimental design, and the response variable chosen all influence the magnitude of pollen limitation. Fail-safe numbers indicate that publication bias exists for some measures of pollen limitation; significant results are more likely to be published and therefore available for review. Moreover, experiments conducted on only a fraction of a plant's flowers and reproductive episodes report ~8-fold higher effect sizes than those on all flowers produced over the entire lifetime, likely because resource reallocation among flowers and across years contributes to estimates of pollen limitation. Studies measuring percentage fruit set report higher values of pollen limitation than those measuring other response variables, such as seeds per fruit, perhaps because many plant species will not produce fruits unless adequate pollen receipt occurs to fertilize most ovules. We offer suggestions for reducing the bias introduced by methodology in pollen supplementation experiments and discuss our results in the context of optimality theory.     

Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments

In 2006, Peters et al. identified 86 systematic reviews (SRs) of laboratory animal experiments (LAEs). They found 46 LAE meta-analyses (MAs), often of poor quality. Six of these 46 MAs tried to assess publication bias. Publication bias is the phenomenon of an experiment's results determining its likelihood of publication, often over-representing positive findings. As such, publication bias is the Achilles heel of any SR. Since researchers increasingly become aware of the fact that SRs directly support the 'three Rs', we expect the number of SRs of LAEs will sharply increase. Therefore, it is useful to see how publication bias is dealt with. Our objective was to identify all SRs and MAs of LAEs where the purpose was to inform human health published between July 2005 and 2010 with special attention to MAs' quality features and publication bias. We systematically searched Medline, Embase, Toxline and ScienceDirect from July 2005 to 2010, updating Peters' review. LAEs not directly informing human health or concerning fundamental biology were excluded. We found 2780 references of which 163 met the inclusion criteria: 158 SRs, of which 30 performed an MA, and five MAs without an SR. The number of SRs roughly doubled every three years since 1997. The number of MAs roughly doubled every five years since 1999. Compared with before July 2005, more MAs were preceded by SR and reported on (quality) features of included studies and heterogeneity. A statistically significant proportion of MAs considered publication bias (26/35) and tried to formally assess it (21/35).     

Publication bias: evidence of delayed publication in a cohort study of clinical research projects.

OBJECTIVES: To determine the extent to which publication is influenced by study outcome. DESIGN: A cohort of studies submitted to a hospital ethics committee over 10 years were examined retrospectively by reviewing the protocols and by questionnaire. The primary method of analysis was Cox''s proportional hazards model. SETTING: University hospital, Sydney, Australia. STUDIES: 748 eligible studies submitted to Royal Prince Alfred Hospital Ethics Committee between 1979 and 1988. MAIN OUTCOME MEASURES: Time to publication. RESULTS: Response to the questionnaire was received for 520 (70%) of the eligible studies. Of the 218 studies analysed with tests of significance, those with positive results (P < 0.05) were much more likely to be published than those with negative results (P > or = 0.10) (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66), P = 0.0003), with a significantly shorter time to publication (median 4.8 v 8.0 years). This finding was even stronger for the group of 130 clinical trials (hazard ratio 3.13 (1.76 to 5.58). P = 0.0001), with median times to publication of 4.7 and 8.0 years respectively. These results were not materially changed after adjusting for other significant predictors of publication. Studies with indefinite conclusions (0.05 < or = P < 0.10) tended to have an even lower publication rate and longer time to publication than studies with negative results (hazard ratio 0.39 (0.13 to 1.12), P = 0.08). For the 103 studies in which outcome was rated qualitatively, there was no clear cut evidence of publication bias, although the number of studies in this group was not large. CONCLUSIONS: This study confirms the evidence of publication bias found in other studies and identifies delay in publication as an additional important factor. The study results support the need for prospective registration of trials to avoid publication bias and also support restricting the selection of trials to those started before a common date in undertaking systematic reviews.     

Predicting population trends using citizen science data: do subsampling methods produce reliable estimates for mammals?

Accurate assessment of population trends is invaluable in wildlife management, particularly for identifying species which are of conservation concern, and consequently, reliable cost-effective methods for their determination are highly desirable. In a recent publication (Eur J Wildl Res 62:407–413, 2016), the authors apply a subsampling method, used in several studies to quantify population trends from citizen science data for butterflies, birds, and plants, to assess the status of West European hedgehogs (Erinaceus europaeus) in England. Whilst the findings may be in agreement with expert opinion, we argue that this type of approach does not adequately account for spatial bias common in mammal data and that without further evaluation it is unclear whether the result is reliable or simply coincidental. To explore this concern, we apply the method across a range of terrestrial mammal species and compare the resulting trends to other published studies. Our findings show that the method fails to reproduce the accepted qualitative trends for the majority of species. Furthermore, comparison of trends based on data obtained from different sources produced conflicting predictions suggesting that the method is indeed vulnerable to survey bias. We therefore conclude that at present, without additional modification to address survey bias, this is not a reliable method for predicting population trends for mammals. However, more generally, this raises questions about the validity of subsampling methods based on citizen science data, and we would urge future studies to exercise caution by performing analysis across a suite of species including those with known trends for validation.     

NCI's publication affiliation conundrum: Reframing innovation to incentivize an equitable path for advocate representation

Advocacy engagement has been at the forefront of National Cancer Institute (NCI) efforts to advance scientific discoveries and transform medical interventions. Nonetheless, the journey for advocates has been uneven. Case in Point: NCI publication affiliation rules of engagement pose unique equity challenges while raising questions about structural representation in biomedical research. Abiding by the core rationale that publication affiliation should be tailored to employment status, the NCI has systematically denied research advocate volunteers the opportunity to specifically list NCI as an institutional affiliation on academic publications. Unpacking advocate NCI publication affiliation restrictions and its links with advocacy heritage preservation and convergent science goals poses unique diversity, equity, and inclusion challenges and opportunities. Improving the quality of structural representation in biomedical research requires new theories of action and flexible planning to advance, promote and build capacity for strategic advocacy inclusion and equity within publication affiliation initiatives. Here we highlight several opportunities for how leadership might formulate a radically different vision for NCI's approach. This perspective interrogates the best way forward for ensuring that biomedical employee and volunteer advocate workforce publication affiliation intersections are characterized by increased creativity and representation parity. Imbuing the scientist and clinical researcher archetype with social dimensions, we join NCI critical thinkers in urging employees, funded academics, and volunteer citizen scientists to collectively assume the role as paladins of science and integrity who view the triumphs of making a difference in science alongside the social responsibility of promoting transdisciplinary professionalism and the democratization of science.     

Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias

Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that address similar questions. A major obstacle for meta-analyses arises from biases in reporting. In particular, it is speculated that findings which do not achieve formal statistical significance are less likely reported than statistically significant findings. Moreover, the patterns of bias can be complex and may also depend on the timing of the research results and their relationship with previously published work. In this paper, we present an approach that is specifically designed to analyze large-scale datasets on published results. Such datasets are currently emerging in diverse research fields, particularly in molecular medicine. We use our approach to investigate a dataset on Alzheimer's disease (AD) that covers 1167 results from case-control studies on 102 genetic markers. We observe that initial studies on a genetic marker tend to be substantially more biased than subsequent replications. The chances for initial, statistically non-significant results to be published are estimated to be about 44% (95% CI, 32% to 63%) relative to statistically significant results, while statistically non-significant replications have almost the same chance to be published as statistically significant replications (84%; 95% CI, 66% to 107%). Early replications tend to be biased against initial findings, an observation previously termed Proteus phenomenon: The chances for non-significant studies going in the same direction as the initial result are estimated to be lower than the chances for non-significant studies opposing the initial result (73%; 95% CI, 55% to 96%). Such dynamic patterns in bias are difficult to capture by conventional methods, where typically simple publication bias is assumed to operate. Our approach captures and corrects for complex dynamic patterns of bias, and thereby helps generating conclusions from published results that are more robust against the presence of different coexisting types of selective reporting.     

Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy

The consolidation of scientific knowledge proceeds through the interpretation and then distillation of data presented in research reports, first in review articles and then in textbooks and undergraduate courses, until truths become accepted as such both amongst “experts” and in the public understanding. Where data are collected but remain unpublished, they cannot contribute to this distillation of knowledge. If these unpublished data differ substantially from published work, conclusions may not reflect adequately the underlying biological effects being described. The existence and any impact of such “publication bias” in the laboratory sciences have not been described. Using the CAMARADES (Collaborative Approach to Meta-analysis and Review of Animal Data in Experimental Studies) database we identified 16 systematic reviews of interventions tested in animal studies of acute ischaemic stroke involving 525 unique publications. Only ten publications (2%) reported no significant effects on infarct volume and only six (1.2%) did not report at least one significant finding. Egger regression and trim-and-fill analysis suggested that publication bias was highly prevalent (present in the literature for 16 and ten interventions, respectively) in animal studies modelling stroke. Trim-and-fill analysis suggested that publication bias might account for around one-third of the efficacy reported in systematic reviews, with reported efficacy falling from 31.3% to 23.8% after adjustment for publication bias. We estimate that a further 214 experiments (in addition to the 1,359 identified through rigorous systematic review; non publication rate 14%) have been conducted but not reported. It is probable that publication bias has an important impact in other animal disease models, and more broadly in the life sciences.