Dissemination biases in ecology: effect sizes matter more than quality

Publication and citation decisions in ecology are likely influenced by many factors, potentially including journal impact factors, direction and magnitude of reported effects, and year of publication. Dissemination bias exists when publication or citation of a study depends on any of these factors. We defined several dissemination biases and determined their prevalence across many sub‐disciplines in ecology, then determined whether or not data quality also affected these biases. We identified dissemination biases in ecology by conducting a meta‐analysis of citation trends for 3867 studies included in 52 meta‐analyses. We correlated effect size, year of publication, impact factor and citation rate within each meta‐analysis. In addition, we explored how data quality as defined in meta‐analyses (sample size or variance) influenced each form of bias. We also explored how the direction of the predicted or observed effect, and the research field, influenced any biases. Year of publication did not influence citation rates. The first papers published in an area reported the strongest effects, and high impact factor journals published the most extreme effects. Effect size was more important than data quality for many publication and citation trends. Dissemination biases appear common in ecology, and although their magnitude was generally small many were associated with theory tenacity, evidenced as tendencies to cite papers that most strongly support our ideas. The consequences of this behavior are amplified by the fact that papers reporting strong effects were often of lower data quality than papers reporting much weaker effects. Furthermore, high impact factor journals published the strongest effects, generally in the absence of any correlation with data quality. Increasing awareness of the prevalence of theory tenacity, confirmation bias, and the inattention to data quality among ecologists is a first step towards reducing the impact of these biases on research in our field.     

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.

Methodology/Principal Findings

We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Statin Use and Risk of Prostate Cancer: A Meta-Analysis of Observational Studies

Background

Emerging evidence suggests that statins may decrease the risk of cancers. However, available evidence on prostate cancer (PCa) is conflicting. We therefore examined the association between statin use and risk of PCa by conducting a detailed meta-analysis of all observational studies published regarding this subject.

Methods

Literature search in PubMed database was undertaken through February 2012 looking for observational studies evaluating the association between statin use and risk of PCa. Before meta-analysis, the studies were evaluated for publication bias and heterogeneity. Pooled relative risk (RR) estimates and 95% confidence intervals (CIs) were calculated using random-effects model (DerSimonian and Laird method). Subgroup analyses, sensitivity analysis and cumulative meta-analysis were also performed.

Results

A total of 27 (15 cohort and 12 case-control) studies contributed to the analysis. There was heterogeneity among the studies but no publication bias. Statin use significantly reduced the risk of both total PCa by 7% (RR 0.93, 95% CI 0.87–0.99, p = 0.03) and clinically important advanced PCa by 20% (RR 0.80, 95% CI 0.70–0.90, p<0.001). Long-term statin use did not significantly affect the risk of total PCa (RR 0.94, 95% CI 0.84–1.05, p = 0.31). Stratification by study design did not substantially influence the RR. Furthermore, sensitivity analysis confirmed the stability of results. Cumulative meta-analysis showed a change in trend of reporting risk from positive to negative in statin users between 1993 and 2011.

Conclusions

Our meta-analysis provides evidence supporting the hypothesis that statins reduce the risk of both total PCa and clinically important advanced PCa. Further research is needed to confirm these findings and to identify the underlying biological mechanisms.     

Testing the immunocompetence handicap hypothesis: a review of the evidence

The immunocompetence handicap hypothesis was formulated 12 years ago in an attempt to offer a proximate mechanism by which female choice of males could be explained by endocrine control of honest signalling. The hypothesis suggested that testosterone has a dual effect in males of controlling the development of sexual signals while causing immunosuppression. Our purpose in this review is to examine the empirical evidence to date that has attempted to test the hypothesis, and to conduct a meta-analysis on two of the assumptions of the hypothesis, that testosterone reduces immunocompetence and increases parasitism, to ascertain any statistical trend in the data. There is some evidence to suggest that testosterone is responsible for the magnitude of trait expression or development of sexual traits, but this is by no means conclusive. The results of many studies attempting to find evidence for the supposed immunosuppressive qualities of testosterone are difficult to interpret since they are observational rather than experimental. Of the experimental studies, the data obtained are ambiguous, and this is reflected in the result of the meta-analysis. Overall, the meta-analysis found a significant suppressive effect of testosterone on immunity, in support of the hypothesis, but this effect disappeared when we controlled for multiple studies on the same species. There was no effect of testosterone on direct measures of immunity, but it did increase ectoparasite abundance in several studies, in particular in reptiles. A funnel analysis indicated that the results were robust to a publication bias. Alternative substances that interact with testosterone, such as glucocorticoids, may be important. Ultimately, a greater understanding is required of the complex relationships that exist both within and between the endocrine and immune systems and their consequences for mate choice decision making.     

Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution.

Both significant positive and negative relationships between the magnitude of research findings (their 'effect size') and their year of publication have been reported in a few areas of biology. These trends have been attributed to Kuhnian paradigm shifts, scientific fads and bias in the choice of study systems. Here we test whether or not these isolated cases reflect a more general trend. We examined the relationship using effect sizes extracted from 44 peer-reviewed meta-analyses covering a wide range of topics in ecological and evolutionary biology. On average, there was a small but significant decline in effect size with year of publication. For the original empirical studies there was also a significant decrease in effect size as sample size increased. However, the effect of year of publication remained even after we controlled for sampling effort. Although these results have several possible explanations, it is suggested that a publication bias against non-significant or weaker findings offers the most parsimonious explanation. As in the medical sciences, non-significant results may take longer to publish and studies with both small sample sizes and non-significant results may be less likely to be published.     

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making.

Methodology/Principal Findings

In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Meta-analysis: synthesizing research findings in ecology and evolution

The growing number of empirical studies performed in ecology and evolution creates a need for quantitative summaries of research domains to generate higher-order conclusions about general trends and patterns. Recent developments In meta-analysis (the area of statistics that is designed for summarizing and analyzing multiple independent studies) have opened up new and exciting possibilities. Unlike more traditional qualitative and narrative reviews, meta-analysis allows powerful quantitative analyses of the magnitude of effects and has a high degree of objectivity because it is based on a standardized set of statistical procedures. The first pioneering applications in ecology and evolution demonstrate that meta-analysis is both tractable and powerful.     

Requiring more empirical studies in ecology: A comparison of sample size in meta-analysis of ecology and medicine

Meta-analysis is an effective and popular tool for studies of systemic literature or research cases review. Compared with medicine, the ecological publications with meta-analysis typically addressed more questions, but with a less datasets. Therefore, in ecology the meta-analysis method should have higher credibility as it does in medicine. For the future of meta-analysis in ecology, the more important work is to implement more empirical studies, such as ‘coordinated distributed experiment’, to get more reliable datasets.     

Helicobacter pylori infection and colorectal cancer risk: a meta-analysis

BACKGROUND: Several studies suggested an association between Helicobacter pylori infection and colorectal carcinoma or adenoma risk. However, different authors reported quite varying estimates. We carried out a systematic review and meta-analysis of published studies investigating this association and paid special attention to the possibility of publication bias and sources of heterogeneity between studies. Materials and METHODS: An extensive literature search and cross-referencing were performed to identify all published studies. Summary estimates were obtained using random-effects models. The presence of possible publication bias was assessed using different statistical approaches. RESULTS: In a meta-analysis of the 11 identified human studies, published between 1991 and 2002, a summary odds ratio of 1.4 (95% CI, 1.1-1.8) was estimated for the association between H. pylori infection and colorectal cancer risk. The graphical funnel plot appeared asymmetrical, but the formal statistical evaluations did not provide strong evidence of publication bias. The proportion of variation of study results because of heterogeneity was small (36.5%). CONCLUSIONS: The results of our meta-analysis are consistent with a possible small increase in risk of colorectal cancer because of H. pylori infection. However, the possibility of some publication bias cannot be ruled out, although it could not be statistically confirmed. Larger, better designed and better controlled studies are needed to clarify the situation.     

Endothelial nitric oxide synthase gene polymorphisms and essential hypertension in Han Chinese

We examined the effect of polymorphisms in the endothelial nitric oxide synthase gene on the risk for essential hypertension in a Han Chinese population through a meta-analysis of data from 15 studies. Associations between increased risk for essential hypertension and 4b/a were obtained in a dominant model and allele contrast (aa + ab vs bb: odds ratio (OR)(FE) = 1.26, 95% confidence interval (CI) = 1.10-1.44; a vs b allele: OR(FE) = 1.23, 95%CI: 1.09-1.40). Four studies with sample sizes over 500 produced similar results. No evidence of publication bias was found. Also, no significant heterogeneity was observed among these studies. When we examined the G894T polymorphism, we found a marginally significant association for allele contrast and the recessive model when all the eligible studies were pooled together. However, there was no evidence for a significant association after the exclusion of two studies deviating from Hardy-Weinberg equilibrium in the control group. Heterogeneity among studies was observed. Results of cumulative and recursive cumulative meta-analysis indicated that more studies are needed to objectively determine the effects of these two polymorphisms.     

Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias

Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that address similar questions. A major obstacle for meta-analyses arises from biases in reporting. In particular, it is speculated that findings which do not achieve formal statistical significance are less likely reported than statistically significant findings. Moreover, the patterns of bias can be complex and may also depend on the timing of the research results and their relationship with previously published work. In this paper, we present an approach that is specifically designed to analyze large-scale datasets on published results. Such datasets are currently emerging in diverse research fields, particularly in molecular medicine. We use our approach to investigate a dataset on Alzheimer's disease (AD) that covers 1167 results from case-control studies on 102 genetic markers. We observe that initial studies on a genetic marker tend to be substantially more biased than subsequent replications. The chances for initial, statistically non-significant results to be published are estimated to be about 44% (95% CI, 32% to 63%) relative to statistically significant results, while statistically non-significant replications have almost the same chance to be published as statistically significant replications (84%; 95% CI, 66% to 107%). Early replications tend to be biased against initial findings, an observation previously termed Proteus phenomenon: The chances for non-significant studies going in the same direction as the initial result are estimated to be lower than the chances for non-significant studies opposing the initial result (73%; 95% CI, 55% to 96%). Such dynamic patterns in bias are difficult to capture by conventional methods, where typically simple publication bias is assumed to operate. Our approach captures and corrects for complex dynamic patterns of bias, and thereby helps generating conclusions from published results that are more robust against the presence of different coexisting types of selective reporting.     

Debating the greening vs. browning of the North American boreal forest: differences between satellite datasets

A number of remote sensing studies have evaluated the temporal trends of the normalized difference vegetation index (NDVI or vegetation greenness) in the North American boreal forest during the last two decades, often getting quite different results. To examine the effect that the use of different datasets might be having on the estimated trends, we compared the temporal trends of recently burned and unburned sites of boreal forest in central Canada calculated from two datasets: the Global Inventory, Monitoring, and Modeling Studies (GIMMS), which is the most commonly used 8 km dataset, and a new 1 km dataset developed by the Canadian Centre for Remote Sensing (CCRS). We compared the NDVI trends of both datasets along a fire severity gradient in order to evaluate the variance in regeneration rates. Temporal trends were calculated using the seasonal Mann–Kendall trend test, a rank‐based, nonparametric test, which is robust against seasonality, nonnormality, heteroscedasticity, missing values, and serial dependence. The results showed contrasting NDVI trends between the CCRS and the GIMMS datasets. The CCRS dataset showed NDVI increases in all recently burned sites and in 50% of the unburned sites. Surprisingly, the GIMMS dataset did not capture the NDVI recovery in most burned sites and even showed NDVI declines in some burned sites one decade after fire. Between 50% and 75% of GIMMS pixels showed NDVI decreases in the unburned forest compared with <1% of CCRS pixels. Being the most broadly used dataset for monitoring ecosystem and carbon balance changes, the bias towards negative trends in the GIMMS dataset in the North American boreal forest has broad implications for the evaluation of vegetation and carbon dynamics in this region and globally.     

Irreproducible text‐book “knowledge”: The effects of color bands on zebra finch fitness

Many fields of science—including behavioral ecology—currently experience a heated debate about the extent to which publication bias against null findings results in a misrepresentative scientific literature. Here, we show a case of an extreme mismatch between strong positive support for an effect in the literature and a failure to detect this effect across multiple attempts at replication. For decades, researchers working with birds have individually marked their study species with colored leg bands. For the zebra finch Taeniopygia guttata, a model organism in behavioral ecology, many studies over the past 35 years have reported effects of bands of certain colors on male or female attractiveness and further on behavior, physiology, life history, and fitness. Only eight of 39 publications presented exclusively null findings. Here, we analyze the results of eight experiments in which we quantified the fitness of a total of 730 color‐banded individuals from four captive populations (two domesticated and two recently wild derived). This sample size exceeds the combined sample size of all 23 publications that clearly support the “color‐band effect” hypothesis. We found that band color explains no variance in either male or female fitness. We also found no heterogeneity in color‐band effects, arguing against both context and population specificity. Analysis of unpublished data from three other laboratories strengthens the generality of our null finding. Finally, a meta‐analysis of previously published results is indicative of selective reporting and suggests that the effect size approaches zero when sample size is large. We argue that our field—and science in general—would benefit from more effective means to counter confirmation bias and publication bias.     

What determines species richness of parasitic organisms? A meta‐analysis across animal,plant and fungal hosts

Although a small set of external factors account for much of the spatial variation in plant and animal diversity, the search continues for general drivers of variation in parasite species richness among host species. Qualitative reviews of existing evidence suggest idiosyncrasies and inconsistent predictive power for all proposed determinants of parasite richness. Here, we provide the first quantitative synthesis of the evidence using a meta‐analysis of 62 original studies testing the relationship between parasite richness across animal, plant and fungal hosts, and each of its four most widely used presumed predictors: host body size, host geographical range size, host population density, and latitude. We uncover three universal predictors of parasite richness across host species, namely host body size, geographical range size and population density, applicable regardless of the taxa considered and independently of most aspects of study design. A proper match in the primary studies between the focal predictor and both the spatial scale of study and the level at which parasite species richness was quantified (i.e. within host populations or tallied across a host species' entire range) also affected the magnitude of effect sizes. By contrast, except for a couple of indicative trends in subsets of the full dataset, there was no strong evidence for an effect of latitude on parasite species richness; where found, this effect ran counter to the general latitude gradient in diversity, with parasite species richness tending to be higher further from the equator. Finally, the meta‐analysis also revealed a negative relationship between the magnitude of effect sizes and the year of publication of original studies (i.e. a time‐lag bias). This temporal bias may be due to the increasing use of phylogenetic correction in comparative analyses of parasite richness over time, as this correction yields more conservative effect sizes. Overall, these findings point to common underlying processes of parasite diversification fundamentally different from those controlling the diversity of free‐living organisms.     

A quantitative synthesis of pollen supplementation experiments highlights the contribution of resource reallocation to estimates of pollen limitation

Our understanding of pollen limitation depends on a realistic view of its magnitude. Previous reviews of pollen supplementation experiments concluded that a majority of plant species suffers from pollen limitation and that its magnitude is high. Here, we perform a meta-analysis and find evidence that publication bias, experimental design, and the response variable chosen all influence the magnitude of pollen limitation. Fail-safe numbers indicate that publication bias exists for some measures of pollen limitation; significant results are more likely to be published and therefore available for review. Moreover, experiments conducted on only a fraction of a plant's flowers and reproductive episodes report ~8-fold higher effect sizes than those on all flowers produced over the entire lifetime, likely because resource reallocation among flowers and across years contributes to estimates of pollen limitation. Studies measuring percentage fruit set report higher values of pollen limitation than those measuring other response variables, such as seeds per fruit, perhaps because many plant species will not produce fruits unless adequate pollen receipt occurs to fertilize most ovules. We offer suggestions for reducing the bias introduced by methodology in pollen supplementation experiments and discuss our results in the context of optimality theory.     

Circulating visfatin levels and cancers risk: A systematic review and meta-analysis

Visfatin levels have been reported to be abnormal in many types of cancers. However, epidemiological studies yielded inconsistent results. Therefore, a meta-analysis was performed to assess the association between circulating visfatin levels and cancer risk. A systematic search was conducted for relevant studies in health-related electronic databases up to March 2018. Data related to standard mean difference (SMD) and overall odds ratio (ORS) were collected and analyzed. Summary SMD and pooled OR with 95% CIs were calculated using a random-effect model. Funnel plot and Egger's linear regression test were conducted to examine the risk of publication bias. A total of 27 studies with 2,693 cases and 3,040 healthy controls were included in meta-analysis for pooling SMD analysis. The results of the meta-analysis showed a significant higher visfatin levels in patients with various cancers than in controls, with a pooled SMD of 0.88, 95% CI = 0.56–1.20, p = 0.000. In subgroup, metaregression, Galbraith plot, and sensitivity analysis showed no substantial difference among all the analyzed factors. Data from 14 studies were also used for pooling ORs analysis. Metaresults revealed that high visfatin levels were associated with cancer risk (OR = 1.24, 95% CI: 1.14–1.34, p = 0.000). No evidence of publication bias was observed for pooling ORs and SMD analysis. This meta-analysis indicated a significant association between high circulating visfatin levels and increased risk of various cancers. Visfatin may represent a potential biomarker for early detection of cancers who may benefit from preventive treatment.Note.     

Bilateral symmetry and sexual selection: a meta-analysis

A considerable body of primary research has accumulated over the last 10 yr testing the relationship between developmental instability in the form of fluctuating asymmetry and performance of individuals in mating success itself or sexual attractiveness. This research comprises 146 samples from 65 studies of 42 species of four major taxa. We present the results of a meta-analysis of these studies, which demonstrates that there is indeed an overall significant, moderate negative relationship: for studies, the overall mean Pearson's r or effect size = -.42, P <.0005; for species, the overall mean r = -.34, .01 < P < .025. Based on calculated fail-safe numbers, the effect-size estimates are highly robust against any publication or reporting bias that may exist. There is considerable evidence that the magnitude of the negative correlation between fluctuating asymmetry and success related to sexual selection is greater for males than for females, when a secondary sexual trait rather than an ordinary trait is studied, with experimentation compared with observation, and for traits not involved with mobility compared with traits affecting mobility. There is also limited evidence that higher taxa may differ in effect size and that intensity of sexual selection negatively correlates with effect size.     

Meta-analysis shows association between the tryptophan hydroxylase (TPH) gene and schizophrenia

A number of studies have suggested an association between schizophrenia and the tryptophan hydroxylase (TPH) and tyrosine hydroxylase (TH) genes. On the other hand, several studies attempting to replicate these findings have produced mixed results, possibly reflecting inadequate statistical power of the individual studies as well as the heterogeneity inherent in schizophrenia. In an attempt to clarify this inconsistency our meta-analysis has combined all the studies using multiple research methods published up to February 2006 to give a comprehensive picture of the role of three hydroxylase-related genes. The TPH A218C/A779C (OR = 1.18, 95% C.I. 1.06–1.33, P = 0.004) revealed a significant association with schizophrenia. However, the evidence for the TH and phenylalanine hydroxylase (PAH) genes was weak. No publication bias was detected in current studies. The findings, which may implicate the involvement of TPH in the pathogenesis of schizophrenia, have potentially important clinical, scientific and public health implications as well as providing a putative basis for the study of hydroxylase-related drugs. To our knowledge, this is the first meta-analysis of association between the three genes and schizophrenia.Electronic Supplementary Material Supplementary material is available to authorised users in the online version of this article at .     

Demography,education, and research trends in the interdisciplinary field of disease ecology

Micro‐ and macroparasites are a leading cause of mortality for humans, animals, and plants, and there is great need to understand their origins, transmission dynamics, and impacts. Disease ecology formed as an interdisciplinary field in the 1970s to fill this need and has recently rapidly grown in size and influence. Because interdisciplinary fields integrate diverse scientific expertise and training experiences, understanding their composition and research priorities is often difficult. Here, for the first time, we quantify the composition and educational experiences of a subset of disease ecology practitioners and identify topical trends in published research. We combined a large survey of self‐declared disease ecologists with a literature synthesis involving machine‐learning topic detection of over 18,500 disease ecology research articles. The number of graduate degrees earned by disease ecology practitioners has grown dramatically since the early 2000s. Similar to other science fields, we show that practitioners in disease ecology have diversified in the last decade in terms of gender identity and institution, with weaker diversification in race and ethnicity. Topic detection analysis revealed how the frequency of publications on certain topics has declined (e.g., HIV, serology), increased (e.g., the dilution effect, infectious disease in bats), remained relatively common (e.g., malaria ecology, influenza, vaccine research and development), or have consistently remained relatively infrequent (e.g., theoretical models, field experiments). Other topics, such as climate change, superspreading, emerging infectious diseases, and network analyses, have recently come to prominence. This study helps identify the major themes of disease ecology and demonstrates how publication frequency corresponds to emergent health and environmental threats. More broadly, our approach provides a framework to examine the composition and publication trends of other major research fields that cross traditional disciplinary boundaries.     

Methodological issues and advances in biological meta-analysis

Meta-analysis has changed the way researchers conduct literature reviews not only in medical and social sciences but also in biological sciences. Meta-analysis in biological sciences, especially in ecology and evolution (which we refer to as ‘biological’ meta-analysis) faces somewhat different methodological problems from its counterparts in medical and social sciences, where meta-analytic techniques were originally developed. The main reason for such differences is that biological meta-analysis often integrates complex data composed of multiple strata with, for example, different measurements and a variety of species. Here, we review methodological issues and advancements in biological meta-analysis, focusing on three topics: (1) non-independence arising from multiple effect sizes obtained in single studies and from phylogenetic relatedness, (2) detecting and accounting for heterogeneity, and (3) identifying publication bias and measuring its impact. We show how the marriage between mixed-effects (hierarchical/multilevel) models and phylogenetic comparative methods has resolved most of the issues under discussion. Furthermore, we introduce the concept of across-study and within-study meta-analysis, and propose how the use of within-study meta-analysis can improve many empirical studies typical of ecology and evolution.