Impact beyond the impact factor

The journal impact factor is an annually calculated number for each scientific journal, based on the average number of times its articles published in the two preceding years have been cited. It was originally devised as a tool for librarians and publishers to provide information about the citation performance of a journal as a whole, but over the last few decades it has increasingly been used to assess the quality of specific articles and the research performance of individual investigators, institutions, and countries. In addition to this clear abuse of the journal impact factor, several conceptual and technical issues limit its usability as a measure of journal reputation, especially when journals are compared across different fields. An author’s decision regarding the suitability of a scholarly journal for publication should, therefore, be based on the impact that this journal makes in the field of research, rather than on the journal impact factor.     

From Excessive Journal Self-Cites to Citation Stacking: Analysis of Journal Self-Citation Kinetics in Search for Journals,Which Boost Their Scientometric Indicators

Bibliometric indicators increasingly affect careers, funding, and reputation of individuals, their institutions and journals themselves. In contrast to author self-citations, little is known about kinetics of journal self-citations. Here we hypothesized that they may show a generalizable pattern within particular research fields or across multiple fields. We thus analyzed self-cites to 60 journals from three research fields (multidisciplinary sciences, parasitology, and information science). We also hypothesized that the kinetics of journal self-citations and citations received from other journals of the same publisher may differ from foreign citations. We analyzed the journals published the American Association for the Advancement of Science, Nature Publishing Group, and Editura Academiei Române. We found that although the kinetics of journal self-cites is generally faster compared to foreign cites, it shows some field-specific characteristics. Particularly in information science journals, the initial increase in a share of journal self-citations during post-publication year 0 was completely absent. Self-promoting journal self-citations of top-tier journals have rather indirect but negligible direct effects on bibliometric indicators, affecting just the immediacy index and marginally increasing the impact factor itself as long as the affected journals are well established in their fields. In contrast, other forms of journal self-citations and citation stacking may severely affect the impact factor, or other citation-based indices. We identified here a network consisting of three Romanian physics journals Proceedings of the Romanian Academy, Series A, Romanian Journal of Physics, and Romanian Reports in Physics, which displayed low to moderate ratio of journal self-citations, but which multiplied recently their impact factors, and were mutually responsible for 55.9%, 64.7% and 63.3% of citations within the impact factor calculation window to the three journals, respectively. They did not receive nearly any network self-cites prior impact factor calculation window, and their network self-cites decreased sharply after the impact factor calculation window. Journal self-citations and citation stacking requires increased attention and elimination from citation indices.     

Evaluating research and impact: a bibliometric analysis of research by the NIH/NIAID HIV/AIDS clinical trials networks

Evaluative bibliometrics uses advanced techniques to assess the impact of scholarly work in the context of other scientific work and usually compares the relative scientific contributions of research groups or institutions. Using publications from the National Institute of Allergy and Infectious Diseases (NIAID) HIV/AIDS extramural clinical trials networks, we assessed the presence, performance, and impact of papers published in 2006-2008. Through this approach, we sought to expand traditional bibliometric analyses beyond citation counts to include normative comparisons across journals and fields, visualization of co-authorship across the networks, and assess the inclusion of publications in reviews and syntheses. Specifically, we examined the research output of the networks in terms of the a) presence of papers in the scientific journal hierarchy ranked on the basis of journal influence measures, b) performance of publications on traditional bibliometric measures, and c) impact of publications in comparisons with similar publications worldwide, adjusted for journals and fields. We also examined collaboration and interdisciplinarity across the initiative, through network analysis and modeling of co-authorship patterns. Finally, we explored the uptake of network produced publications in research reviews and syntheses. Overall, the results suggest the networks are producing highly recognized work, engaging in extensive interdisciplinary collaborations, and having an impact across several areas of HIV-related science. The strengths and limitations of the approach for evaluation and monitoring research initiatives are discussed.     

Impact of the journal Plastic and Reconstructive Surgery.

The "impact" of Plastic and Reconstructive Surgery was evaluated by bibliometric analysis. This technique considers the average frequency with which journal authors refer to articles published in Plastic and Reconstructive Surgery. The impact of Plastic and Reconstructive Surgery was compared with that of journals most commonly referred to within Plastic and Reconstructive Surgery as well as with that of journals read most commonly by Plastic and Reconstructive Surgery subscribers, such as the Annals of Plastic Surgery and the Journal of Hand Surgery (both American and British editions). The impact factor for Plastic and Reconstructive Surgery was at least twice as great as for any other journal related to this surgical subspecialty. With regard to other surgical subspecialties, while the impact factor for Plastic and Reconstructive Surgery was less than that of the Journal of Neurosurgery, Journal of Urology, and Journal of Ophthalmology, it was greater than those for orthopedics, otolaryngology, hand surgery, and dermatologic surgery.     

Comparison of Journal Self-Citation Rates between Some Chinese and Non-Chinese International Journals

Background

The past 3 decades have witnessed a boost in science development in China; in parallel, more and more Chinese scientific journals are indexed by the Journal Citation Reports issued by Thomson Reuters (SCI). Evaluation of the performance of these Chinese SCI journals is necessary and helpful to improve their quality. This study aimed to evaluate these journals by calculating various journal self-citation rates, which are important parameters influencing a journal impact factor.

Methodology/Principal Findings

We defined three journal self-citation rates, and studied these rates for 99 Chinese scientific journals, almost exhausting all Chinese SCI journals currently available. Likewise, we selected 99 non-Chinese international (abbreviated as ‘world’) journals, with each being in the same JCR subject category and having similar impact factors as their Chinese counterparts. Generally, Chinese journals tended to be higher in all the three self-citation rates than world journal counterparts. Particularly, a few Chinese scientific journals had much higher self-citation rates.

Conclusions/Significance

Our results show that generally Chinese scientific journals have higher self-citation rates than those of world journals. Consequently, Chinese scientific journals tend to have lower visibility and are more isolated in the relevant fields. Considering the fact that sciences are rapidly developing in China and so are Chinese scientific journals, we expect that the differences of journal self-citation rates between Chinese and world scientific journals will gradually disappear in the future. Some suggestions to solve the problems are presented.     

Inflated impact factors? The true impact of evolutionary papers in non-evolutionary journals

Amongst the numerous problems associated with the use of impact factors as a measure of quality are the systematic differences in impact factors that exist among scientific fields. While in theory this can be circumvented by limiting comparisons to journals within the same field, for a diverse and multidisciplinary field like evolutionary biology, in which the majority of papers are published in journals that publish both evolutionary and non-evolutionary papers, this is impossible. However, a journal's overall impact factor may well be a poor predictor for the impact of its evolutionary papers. The extremely high impact factors of some multidisciplinary journals, for example, are by many believed to be driven mostly by publications from other fields. Despite plenty of speculation, however, we know as yet very little about the true impact of evolutionary papers in journals not specifically classified as evolutionary. Here I present, for a wide range of journals, an analysis of the number of evolutionary papers they publish and their average impact. I show that there are large differences in impact among evolutionary and non-evolutionary papers within journals; while the impact of evolutionary papers published in multidisciplinary journals is substantially overestimated by their overall impact factor, the impact of evolutionary papers in many of the more specialized, non-evolutionary journals is significantly underestimated. This suggests that, for evolutionary biologists, publishing in high-impact multidisciplinary journals should not receive as much weight as it does now, while evolutionary papers in more narrowly defined journals are currently undervalued. Importantly, however, their ranking remains largely unaffected. While journal impact factors may thus indeed provide a meaningful qualitative measure of impact, a fair quantitative comparison requires a more sophisticated journal classification system, together with multiple field-specific impact statistics per journal.     

Conflicts of interest at medical journals: the influence of industry-supported randomised trials on journal impact factors and revenue - cohort study

Background

Transparency in reporting of conflict of interest is an increasingly important aspect of publication in medical journals. Publication of large industry-supported trials may generate many citations and journal income through reprint sales and thereby be a source of conflicts of interest for journals. We investigated industry-supported trials'' influence on journal impact factors and revenue.

Methods and Findings

We sampled six major medical journals (Annals of Internal Medicine, Archives of Internal Medicine, BMJ, JAMA, The Lancet, and New England Journal of Medicine [NEJM]). For each journal, we identified randomised trials published in 1996–1997 and 2005–2006 using PubMed, and categorized the type of financial support. Using Web of Science, we investigated citations of industry-supported trials and the influence on journal impact factors over a ten-year period. We contacted journal editors and retrieved tax information on income from industry sources. The proportion of trials with sole industry support varied between journals, from 7% in BMJ to 32% in NEJM in 2005–2006. Industry-supported trials were more frequently cited than trials with other types of support, and omitting them from the impact factor calculation decreased journal impact factors. The decrease varied considerably between journals, with 1% for BMJ to 15% for NEJM in 2007. For the two journals disclosing data, income from the sales of reprints contributed to 3% and 41% of the total income for BMJ and The Lancet in 2005–2006.

Conclusions

Publication of industry-supported trials was associated with an increase in journal impact factors. Sales of reprints may provide a substantial income. We suggest that journals disclose financial information in the same way that they require them from their authors, so that readers can assess the potential effect of different types of papers on journals'' revenue and impact. Please see later in the article for the Editors'' Summary     

Species Distribution Models and Impact Factor Growth in Environmental Journals: Methodological Fashion or the Attraction of Global Change Science

In this work, I evaluate the impact of species distribution models (SDMs) on the current status of environmental and ecological journals by asking the question to which degree development of SDMs in the literature is related to recent changes in the impact factors of ecological journals. The hypothesis evaluated states that research fronts are likely to attract research attention and potentially drive citation patterns, with journals concentrating papers related to the research front receiving more attention and benefiting from faster increases in their impact on the ecological literature. My results indicate a positive relationship between the number of SDM related articles published in a journal and its impact factor (IF) growth during the period 2000–09. However, the percentage of SDM related papers in a journal was strongly and positively associated with the percentage of papers on climate change and statistical issues. The results support the hypothesis that global change science has been critical in the development of SDMs and that interest in climate change research in particular, rather than the usage of SDM per se, appears as an important factor behind journal IF increases in ecology and environmental sciences. Finally, our results on SDM application in global change science support the view that scientific interest rather than methodological fashion appears to be the major driver of research attraction in the scientific literature.     

中国生态学期刊现状分析

通过对我国12种主要生态学期刊的出版指标、引用指标和网络传播能力的现状分析,得出《生态学报》、《应用生态学报》、《植物生态学报》和《生态学杂志》已经成为我国生态学期刊的第一方阵,出版能力和引用表现都要优于其他期刊,但是期刊特色不明显。生态学期刊已经不能满足生态学快速发展的需要,应该有新的期刊来承载新生分支学科的内容,同时生态学期刊需要进一步加强网络传播能力的建设。最重要的是生态学期刊应该从散落在其他学科的状态中聚集起来形成生态学期刊集群。     

Qualitative and quantitative aspects of recent research on helminth parasites

Relationships between the species diversity of different taxa, the mean number of articles published per year on each taxon, and the mean impact factor of the journals in which they appear, were examined across six taxa of helminths: Nematomorpha, Acanthocephala, Monogenea, Trematoda, Cestoda and Nematoda, the latter including only animal parasitic nematodes. The mean annual output of scientific articles per taxon was not related to the species diversity of these taxa or, at least, not significantly. Thus, the large volume of publications on nematodes is not merely a reflection of their estimated diversity. There were significant differences among taxa in the mean impact factor of the journals in which papers on each taxon appeared, with nematodes having the highest mean score, followed by trematodes and cestodes. In addition, across the six taxa, the mean journal impact factor correlated positively and significantly with the mean annual number of papers published: not only are there more papers published on nematodes and trematodes than on nematomorphs or acanthocephalans, but they are also generally published in higher-ranking journals. These results suggest that there is an increasing gap in the quantity and general importance of the research carried out on different helminth taxa.     

I Publish in I Edit? - Do Editorial Board Members of Urologic Journals Preferentially Publish Their Own Scientific Work?

Scientists who are members of an editorial board have been accused of preferentially publishing their scientific work in the journal where they serve as editor. Reputation and academic standing do depend on an uninterrupted flow of published scientific work and the question does arise as to whether publication mainly occurs in the self-edited journal. This investigation was designed to determine whether editorial board members of five urological journals were more likely to publish their research reports in their own rather than in other journals. A retrospective analysis was conducted for all original reports published from 2001–2010 by 65 editorial board members nominated to the boards of five impact leading urologic journals in 2006. Publications before editorial board membership, 2001–2005, and publications within the period of time as an editorial board member, 2006–2010, were identified. The impact factors of the journals were also recorded over the time period 2001–2010 to see whether a change in impact factor correlated with publication locality. In the five journals as a whole, scientific work was not preferentially published in the journal in which the scientists served as editor. However, significant heterogeneity among the journals was evident. One journal showed a significant increase in the amount of published papers in the ‘own’ journal after assumption of editorship, three journals showed no change and one journal showed a highly significant decrease in publishing in the ‘own’ journal after assumption of editorship.     

Dissemination biases in ecology: effect sizes matter more than quality

Publication and citation decisions in ecology are likely influenced by many factors, potentially including journal impact factors, direction and magnitude of reported effects, and year of publication. Dissemination bias exists when publication or citation of a study depends on any of these factors. We defined several dissemination biases and determined their prevalence across many sub‐disciplines in ecology, then determined whether or not data quality also affected these biases. We identified dissemination biases in ecology by conducting a meta‐analysis of citation trends for 3867 studies included in 52 meta‐analyses. We correlated effect size, year of publication, impact factor and citation rate within each meta‐analysis. In addition, we explored how data quality as defined in meta‐analyses (sample size or variance) influenced each form of bias. We also explored how the direction of the predicted or observed effect, and the research field, influenced any biases. Year of publication did not influence citation rates. The first papers published in an area reported the strongest effects, and high impact factor journals published the most extreme effects. Effect size was more important than data quality for many publication and citation trends. Dissemination biases appear common in ecology, and although their magnitude was generally small many were associated with theory tenacity, evidenced as tendencies to cite papers that most strongly support our ideas. The consequences of this behavior are amplified by the fact that papers reporting strong effects were often of lower data quality than papers reporting much weaker effects. Furthermore, high impact factor journals published the strongest effects, generally in the absence of any correlation with data quality. Increasing awareness of the prevalence of theory tenacity, confirmation bias, and the inattention to data quality among ecologists is a first step towards reducing the impact of these biases on research in our field.     

Open Access Meets Discoverability: Citations to Articles Posted to Academia.edu

Using matching and regression analyses, we measure the difference in citations between articles posted to Academia.edu and other articles from similar journals, controlling for field, impact factor, and other variables. Based on a sample size of 31,216 papers, we find that a paper in a median impact factor journal uploaded to Academia.edu receives 16% more citations after one year than a similar article not available online, 51% more citations after three years, and 69% after five years. We also found that articles also posted to Academia.edu had 58% more citations than articles only posted to other online venues, such as personal and departmental home pages, after five years.     

Effectiveness of journal ranking schemes as a tool for locating information

Background

The rise of electronic publishing [1], preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review [2]. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information [1]. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research.

Methodology/Principal Findings

Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.

Conclusions/Significance

Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.     

Impact factor of Revista Iberoamericana de Micología

The impact factor is a bibliometric indicator published annually in the Journal Citation Reports, and widely regarded as a quality ranking of the journals included in this database. The problem with this indicator is that the impact factor of several journals not listed in the Science Citation Index database is largely unknown. The aim of this study was to analyze the 2001 national and international impact factor of Revista Iberoamericana de Micología. The National impact factor of Revista Iberoamericana de Micología was obtained by adding the number of cites in 2001 from a total of 87 Spanish medical journals of greater scientific quality. Also, bibliographical references from Spanish journals indexed in the 2001 Journal Citation reports database have been included to determine the international impact factor of this analyzed journal. Revista Iberoamericana de Micología received a total of 62 cites from published articles in 1999 to 2001, coming from 20 different journals, being their self-citation index 10.1%. The journal with the highest number of cites to Revista Iberoamericana de Micología was Journal of Clinical Microbiology, with 12 cites (19.3%). According to this findings the national and international impact factor of Revista Iberoamericana de Micología was 0.266 and 0.606, respectively. The impact factor of Revista Iberoamericana de Micología, although not included in the Science Citation Index database, was higher than other Journal Citation Reports. Moreover, Revista Iberoamericana de Micología received most of its citations from high impact factor journals included in the Journal Citation Reports database. These data support the international recognition of the scientific level of the journal.     

Increasing recognition of NHJ: a first-time impact factor of 1.4!

A crucial step for a journal is the attainment of an acceptable impact factor. Impact factors of journals are calculated each year by Thomson Scientific (ISI) and reported in the Journal Citation Reports (JCR).     

An Analysis of Peer-Reviewed Scores and Impact Factors with Different Citation Time Windows: A Case Study of 28 Ophthalmologic Journals

Background

An important attribute of the traditional impact factor was the controversial 2-year citation window. So far, several scholars have proposed using different citation time windows for evaluating journals. However, there is no confirmation whether a longer citation time window would be better. How did the journal evaluation effects of 3IF, 4IF, and 6IF comparing with 2IF and 5IF? In order to understand these questions, we made a comparative study of impact factors with different citation time windows with the peer-reviewed scores of ophthalmologic journals indexed by Science Citation Index Expanded (SCIE) database.

Methods

The peer-reviewed scores of 28 ophthalmologic journals were obtained through a self-designed survey questionnaire. Impact factors with different citation time windows (including 2IF, 3IF, 4IF, 5IF, and 6IF) of 28 ophthalmologic journals were computed and compared in accordance with each impact factor’s definition and formula, using the citation analysis function of the Web of Science (WoS) database. An analysis of the correlation between impact factors with different citation time windows and peer-reviewed scores was carried out.

Results

Although impact factor values with different citation time windows were different, there was a high level of correlation between them when it came to evaluating journals. In the current study, for ophthalmologic journals’ impact factors with different time windows in 2013, 3IF and 4IF seemed the ideal ranges for comparison, when assessed in relation to peer-reviewed scores. In addition, the 3-year and 4-year windows were quite consistent with the cited peak age of documents published by ophthalmologic journals.

Research Limitations

Our study is based on ophthalmology journals and we only analyze the impact factors with different citation time window in 2013, so it has yet to be ascertained whether other disciplines (especially those with a later cited peak) or other years would follow the same or similar patterns.

Originality/ Value

We designed the survey questionnaire ourselves, specifically to assess the real influence of journals. We used peer-reviewed scores to judge the journal evaluation effect of impact factors with different citation time windows. The main purpose of this study was to help researchers better understand the role of impact factors with different citation time windows in journal evaluation.     

Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals

Journal policy on research data and code availability is an important part of the ongoing shift toward publishing reproducible computational science. This article extends the literature by studying journal data sharing policies by year (for both 2011 and 2012) for a referent set of 170 journals. We make a further contribution by evaluating code sharing policies, supplemental materials policies, and open access status for these 170 journals for each of 2011 and 2012. We build a predictive model of open data and code policy adoption as a function of impact factor and publisher and find higher impact journals more likely to have open data and code policies and scientific societies more likely to have open data and code policies than commercial publishers. We also find open data policies tend to lead open code policies, and we find no relationship between open data and code policies and either supplemental material policies or open access journal status. Of the journals in this study, 38% had a data policy, 22% had a code policy, and 66% had a supplemental materials policy as of June 2012. This reflects a striking one year increase of 16% in the number of data policies, a 30% increase in code policies, and a 7% increase in the number of supplemental materials policies. We introduce a new dataset to the community that categorizes data and code sharing, supplemental materials, and open access policies in 2011 and 2012 for these 170 journals.     

Temporal trends in the impact factor of European versus USA biomedical journals

Background

The impact factors of biomedical journals tend to rise over time. We sought to assess the trend in the impact factor, during the past decade, of journals published on behalf of United States (US) and European scientific societies, in four select biomedical subject categories (Biology, Cell Biology, Critical Care Medicine, and Infectious Diseases).

Methods

We identified all journals included in the above-mentioned subject categories of Thomson Reuters Journal Citation Reports® for the years 1999, 2002, 2005, and 2008. We selected those that were published on behalf of US or European scientific societies, as documented in journal websites.

Results

We included 167 journals (35 in the subject category of Biology, 79 in Cell Biology, 27 in Critical Care Medicine, and 26 in Infectious Diseases). Between 1999 and 2008, the percentage increase in the impact factor of the European journals was higher than for the US journals (73.7±110.0% compared with 39.7±70.0%, p = 0.049). Regarding specific subject categories, the percentage change in the factor of the European journals tended to be higher than the respective US journals for Cell Biology (61.7% versus 16.3%), Critical Care Medicine (212.4% versus 65.4%), Infectious Diseases (88.3% versus 48.7%), whereas the opposite was observed for journals in Biology (41.0% versus 62.5%).

Conclusion

Journals published on behalf of European scientific societies, in select biomedical fields, may tend to close the “gap” in impact factor compared with those of US societies.

What''s Already Known About This Topic?

The impact factors of biomedical journals tend to rise through years. The leading positions in productivity in biomedical research are held by developed countries, including those from North America and Western Europe.

What Does This Article Add?

The journals from European biomedical scientific societies tended, over the past decade, to increase their impact factor more than the respective US journals.