Publication bias in ecology and evolution: an empirical assessment using the 'trim and fill' method

Recent reviews of specific topics, such as the relationship between male attractiveness to females and fluctuating asymmetry or attractiveness and the expression of secondary sexual characters, suggest that publication bias might be a problem in ecology and evolution. In these cases, there is a significant negative correlation between the sample size of published studies and the magnitude or strength of the research findings (formally the ‘effect size’). If all studies that are conducted are equally likely to be published, irrespective of their findings, there should not be a directional relationship between effect size and sample size; only a decrease in the variance in effect size as sample size increases due to a reduction in sampling error. One interpretation of these reports of negative correlations is that studies with small sample sizes and weaker findings (smaller effect sizes) are less likely to be published. If the biological literature is systematically biased this could undermine the attempts of reviewers to summarise actual biology relationships by inflating estimates of average effect sizes. But how common is this problem? And does it really effect the general conclusions of literature reviews? Here, we examine data sets of effect sizes extracted from 40 peer‐reviewed, published meta‐analyses. We estimate how many studies are missing using the newly developed ‘trim and fill’ method. This method uses asymmetry in plots of effect size against sample size (‘funnel plots’) to detect ‘missing’ studies. For random‐effect models of meta‐analysis 38% (15/40) of data sets had a significant number of ‘missing’ studies. After correcting for potential publication bias, 21% (8/38) of weighted mean effects were no longer significantly greater than zero, and 15% (5/34) were no longer statistically robust when we used random‐effects models in a weighted meta‐analysis. The mean correlation between sample size and the magnitude of standardised effect size was also significantly negative (r_s=‐0.20, P < 0‐0001). Individual correlations were significantly negative (P < 0.10) in 35% (14/40) of cases. Publication bias may therefore effect the main conclusions of at least 15–21% of meta‐analyses. We suggest that future literature reviews assess the robustness of their main conclusions by correcting for potential publication bias using the ‘trim and fill’ method.     

Ignoring non‐English‐language studies may bias ecological meta‐analyses

Meta‐analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta‐analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta‐analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by systematically searching published meta‐analyses and comparing effect‐size estimates between English‐ and Japanese‐language studies included in existing meta‐analyses. Of the 40 published ecological meta‐analysis articles authored by those affiliated to Japanese institutions, we find that three meta‐analysis articles searched for studies in the two languages and involved sufficient numbers of English‐ and Japanese‐language studies, resulting in four eligible meta‐analyses (i.e., four meta‐analyses conducted in the three meta‐analysis articles). In two of the four, effect sizes differ significantly between the English‐ and Japanese‐language studies included in the meta‐analyses, causing considerable changes in overall mean effect sizes and even their direction when Japanese‐language studies are excluded. The observed differences in effect sizes are likely attributable to systematic differences in reported statistical results and associated study characteristics, particularly taxa and ecosystems, between English‐ and Japanese‐language studies. Despite being based on a small sample size, our findings suggest that ignoring non‐English‐language studies may bias outcomes of ecological meta‐analyses, due to systematic differences in study characteristics and effect‐size estimates between English‐ and non‐English languages. We provide a list of actions that meta‐analysts could take in the future to reduce the risk of language bias.     

论文引用率影响因素——中外生态学期刊比较

本文选择8种有代表性中外生态学期刊,对其一定发表周期内的论文引用率进行分析,探讨生态学论文引用率的影响因素及中外生态学期刊的差异.结果表明：4种英文期刊的年均被引次数均远大于4种中文期刊;英文期刊1位作者的论文数量百分比相对较高;所有期刊的合著论文比例均较高,体现了合作性在现代生态学研究中的重要性;论文作者数量与引用率之间有一定的正相关关系,但不显著;英文期刊论文的长度显著高于中文论文;随着论文长度的增加,年均被引次数增多.对中外期刊论文的引用率变化动态进行分析表明,英文期刊中总被引次数高的论文其增长速率也较快,表明其持续影响力强于中文生态学论文.我们希望这些结果会对生态学者以及相关期刊工作者有所裨益.     

Formalizing the definition of meta‐analysis in Molecular Ecology

Meta‐analysis, the statistical synthesis of pertinent literature to develop evidence‐based conclusions, is relatively new to the field of molecular ecology, with the first meta‐analysis published in the journal Molecular Ecology in 2003 (Slate & Phua 2003). The goal of this article is to formalize the definition of meta‐analysis for the authors, editors, reviewers and readers of Molecular Ecology by completing a review of the meta‐analyses previously published in this journal. We also provide a brief overview of the many components required for meta‐analysis with a more specific discussion of the issues related to the field of molecular ecology, including the use and statistical considerations of Wright's F_ST and its related analogues as effect sizes in meta‐analysis. We performed a literature review to identify articles published as ‘meta‐analyses’ in Molecular Ecology, which were then evaluated by at least two reviewers. We specifically targeted Molecular Ecology publications because as a flagship journal in this field, meta‐analyses published in Molecular Ecology have the potential to set the standard for meta‐analyses in other journals. We found that while many of these reviewed articles were strong meta‐analyses, others failed to follow standard meta‐analytical techniques. One of these unsatisfactory meta‐analyses was in fact a secondary analysis. Other studies attempted meta‐analyses but lacked the fundamental statistics that are considered necessary for an effective and powerful meta‐analysis. By drawing attention to the inconsistency of studies labelled as meta‐analyses, we emphasize the importance of understanding the components of traditional meta‐analyses to fully embrace the strengths of quantitative data synthesis in the field of molecular ecology.     

'PUBLISH OR PERISH': THE PITFALLS OF DUPLICATE PUBLICATION

Abstract:  Duplication of previously published text or figures in the scientific literature without adequate citation is plagiarism or, in the case of an author's own work, self-plagiarism. It breaches the ethical standards that are expected in science and threatens the integrity of scientific journals. Three examples of duplication are noted, one of which involves Palaeontology . Redundant publication lowers the quality of scientific literature, damages the good standing of journals, and reduces the intellectual impact of a study. Multiple papers on a particular theme are only acceptable if each builds significantly upon previous work and contains only as much background information as necessary to put the new data and observations into perspective.     

Fifty years of invasion ecology – the legacy of Charles Elton

The publication, in 1958, of Charles Elton's book The ecology of invasions by animals and plants launched the systematic study of biological invasions. Invasion ecology has grown to become an important multi‐disciplinary subfield of ecology with growing links to many other disciplines. This paper examines the citation history of Elton's book using the Web of Science. We also examine Elton's influence in shaping the current research agenda in invasion ecology, for which we use the 28 papers in a special issue of Diversity and Distributions (Volume 14: 2) as a representative sample. After 50 years, Elton's book remains the most cited single source in the field (> 1500 citations), and is cited more often every year (> 100 times) than any other invasion‐related publication, including influential papers in journals. Most citations to Elton's book refer to particular topics/concepts covered in the book, rather than citing it as a general reference about invasions. The shift in the distribution of topics/concepts cited with reference to Elton over time follows the same trend as for biogeography and ecology in general (increasing emphasis on analytical studies, multi‐scale analyses, multi‐disciplinary studies, etc.). Some topics emphasized by Elton are still the focus of current research (dispersal and spread of invasive organisms, impact on biodiversity, role of disturbance and enemy release) but several prominent themes in modern studies were not addressed by Elton. The emergence of new themes can be attributed to a general change in approach and emphasis underpinning research questions in conservation biogeography and applied ecology over the last half century (risk analysis, multi‐scale comparisons, propagule pressure, experimental approaches) and to the recent emergence and increasing availability of large data sets on the distribution of introduced species and to the emergence of key technologies (e.g. geographic information systems, modelling techniques, including niche‐based modelling, and molecular methods). Half a century after its publication, Charles Elton's book on invasions remains influential, but massive changes in the status of invasions and other environmental issues worldwide, together with advances in technology, are reshaping the game rules and priorities of invasion ecology.     

A possible role of social activity to explain differences in publication output among ecologists

Publication output is the standard by which scientific productivity is evaluated. Despite a plethora of papers on the issue of publication and citation biases, no study has so far considered a possible effect of social activities on publication output. One of the most frequent social activities in the world is drinking alcohol. In Europe, most alcohol is consumed as beer and, based on well known negative effects of alcohol consumption on cognitive performance, I predicted negative correlations between beer consumption and several measures of scientific performance. Using a survey from the Czech Republic, that has the highest per capita beer consumption rate in the world, I show that increasing per capita beer consumption is associated with lower numbers of papers, total citations, and citations per paper (a surrogate measure of paper quality). In addition I found the same predicted trends in comparison of two separate geographic areas within the Czech Republic that are also known to differ in beer consumption rates. These correlations are consistent with the possibility that leisure time social activities might influence the quality and quantity of scientific work and may be potential sources of publication and citation biases.     

Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology

Temporal changes in the magnitude of research findings have recently been recognized as a general phenomenon in ecology, and have been attributed to the delayed publication of non-significant results and disconfirming evidence. Here we introduce a method of cumulative meta-analysis which allows detection of both temporal trends and publication bias in the ecological literature. To illustrate the application of the method, we used two datasets from recently conducted meta-analyses of studies testing two plant defence theories. Our results revealed three phases in the evolution of the treatment effects. Early studies strongly supported the hypothesis tested, but the magnitude of the effect decreased considerably in later studies. In the latest studies, a trend towards an increase in effect size was observed. In one of the datasets, a cumulative meta-analysis revealed publication bias against studies reporting disconfirming evidence; such studies were published in journals with a lower impact factor compared to studies with results supporting the hypothesis tested. Correlation analysis revealed neither temporal trends nor evidence of publication bias in the datasets analysed. We thus suggest that cumulative meta-analysis should be used as a visual aid to detect temporal trends and publication bias in research findings in ecology in addition to the correlative approach.     

Irreproducible text‐book “knowledge”: The effects of color bands on zebra finch fitness

Many fields of science—including behavioral ecology—currently experience a heated debate about the extent to which publication bias against null findings results in a misrepresentative scientific literature. Here, we show a case of an extreme mismatch between strong positive support for an effect in the literature and a failure to detect this effect across multiple attempts at replication. For decades, researchers working with birds have individually marked their study species with colored leg bands. For the zebra finch Taeniopygia guttata, a model organism in behavioral ecology, many studies over the past 35 years have reported effects of bands of certain colors on male or female attractiveness and further on behavior, physiology, life history, and fitness. Only eight of 39 publications presented exclusively null findings. Here, we analyze the results of eight experiments in which we quantified the fitness of a total of 730 color‐banded individuals from four captive populations (two domesticated and two recently wild derived). This sample size exceeds the combined sample size of all 23 publications that clearly support the “color‐band effect” hypothesis. We found that band color explains no variance in either male or female fitness. We also found no heterogeneity in color‐band effects, arguing against both context and population specificity. Analysis of unpublished data from three other laboratories strengthens the generality of our null finding. Finally, a meta‐analysis of previously published results is indicative of selective reporting and suggests that the effect size approaches zero when sample size is large. We argue that our field—and science in general—would benefit from more effective means to counter confirmation bias and publication bias.     

High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information

Do citations accumulate too slowly in the social sciences to be used to assess the quality of recent articles? I investigate whether this is the case using citation data for all articles in economics and political science published in 2006 and indexed in the Web of Science. I find that citations in the first two years after publication explain more than half of the variation in cumulative citations received over a longer period. Journal impact factors improve the correlation between the predicted and actual future ranks of journal articles when using citation data from 2006 alone but the effect declines sharply thereafter. Finally, more than half of the papers in the top 20% in 2012 were already in the top 20% in the year of publication (2006).     

Negative changes in the scientific publication process in ecology: potential causes and consequences

1. Increasingly viewed to have societal impact and value, science is affected by complex changes such as globalisation and the increasing dominance of commercial interest. As a result, technical advancements, financial concerns, institutional prestige and journal proliferation have created challenges for ecological and other scientific journals and affected the perception of both researchers and the public about the science that they publish. 2. Journals are now used for more than dissemination of scientific research. Institutions use journal rankings for a variety of purposes and often require a pre‐established number of articles in hiring and budgetary decisions. Consequently, journal impact factors have achieved greater importance, and the splitting of articles into smaller parcels of information (‘salami‐slicing’) to increase numbers of publications has become more frequent. 3. Journals may prescribe upper limits to article length, even though the average length of articles for several ecological journals examined has increased over time. There are clear signs, however, that journals without length limits for articles will become rarer. In contrast to ecological journals, taxonomic journals are not following this trend. 4. Two case histories demonstrate how splitting longer ecological articles into a series of shorter ones results in both redundancy of information and actually increases the journal space used overall. Furthermore, with current rejection rates of ecological journals (often ～80%), many thin salami‐sliced articles jam the peer‐review system much longer (through resubmission after rejection) than unsliced articles previously did (e.g. when rejection rates were ～50%). In our experience, the increased pressure to publish many articles in ‘high‐impact’ journals also may decrease the attractiveness of a future scientific career in ecology to young people. 5. ‘Gatekeeping’ of journal quality has shifted from editors to reviewers, and several recent trends are apparent including: bias about appropriate statistical methods; reviewers being more rigid overall; non‐native English writers being criticised for poor communications skills; and favourable reviews being signed more often than unfavourable ones. In terms of production, outsourcing of copy editing has increased the final error rate of published material. 6. We supplemented our perceptions with those of older colleagues (～100 experienced ecologists) that responded to an informal survey on this topic (response rate: 81%). In the opinion of almost 90% of our respondents, the overall review process has changed and for 20% among them the professional quality of reviews has declined. 7. We, and many older colleagues, are convinced there have been some negative changes in the scientific publication process. If younger colleagues share this concern, we can collectively counter this deteriorating situation, because we are the key to the publishing and evaluation process.     

Data archiving in ecology and evolution: best practices

Many ecology and evolution journals have recently adopted policies requiring that data from their papers be publicly archived. I present suggestions on how data generators, data re-users, and journals can maximize the fairness and scientific value of data archiving. Data should be archived with enough clarity and supporting information that they can be accurately interpreted by others. Re-users should respect their intellectual debt to the originators of data through citation both of the paper and of the data package. In addition, journals should consider requiring that all data for published papers be archived, just as DNA sequences must be deposited in GenBank. Data are another valuable part of the legacy of a scientific career and archiving them can lead to new scientific insights. Archiving also increases opportunities for credit to be given to the scientists who originally collected the data.     

Fifty-Year Fate and Impact of General Medical Journals

Background

Influential medical journals shape medical science and practice and their prestige is usually appraised by citation impact metrics, such as the journal impact factor. However, how permanent are medical journals and how stable is their impact over time?

Methods and Results

We evaluated what happened to general medical journals that were publishing papers half a century ago, in 1959. Data were retrieved from ISI Web of Science for citations and PubMed (Journals function) for journal history. Of 27 eligible journals publishing in 1959, 4 have stopped circulation (including two of the most prestigious journals in 1959) and another 7 changed name between 1959 and 2009. Only 6 of these 27 journals have been published continuously with their initial name since they started circulation. The citation impact of papers published in 1959 gives a very different picture from the current journal impact factor; the correlation between the two is non-significant and very close to zero. Only 13 of the 5,223 papers published in 1959 received at least 5 citations in 2009.

Conclusions

Journals are more permanent entities than single papers, but they are also subject to major change and their relative prominence can change markedly over time.     

Towards a reduction in publication bias.

Current practice results in the publication of many research studies in medical and related disciplines which may be criticised on the grounds of inadequate sample size and statistical power. Small studies continue to be carried out with little more than a blind hope of showing the desired effect. Nevertheless, papers based on such work are submitted for publication, especially if the results turn out to be statistically significant. There is confusion about what makes a result suitable for publication. Often there is a preference for statistically significant results at the peer review stage. Consequently published reports of small studies tend to contain too many false positive results and to exaggerate the true effects. The use of a criterion of a posteriori power does not eliminate the bias; a priori power is the criterion of choice. This could be implemented by peer review of study protocols at the planning stage by funding bodies and journals.