Towards objectivity in research evaluation using bibliometric indicators – A protocol for incorporating complexity

Publications are thought to be an integrative indicator best suited to measure the multifaceted nature of scientific performance. Therefore, indicators based on the publication record (citation analysis) are the primary tool for rapid evaluation of scientific performance. Nevertheless, it has to be questioned whether the indicators really do measure what they are intended to measure because people adjust to the indicator value system by optimizing their indicator rather than their performance. Thus, no matter how sophisticated an indicator may be, it will never be proof against manipulation. A literature review identifies the most critical problems of citation analysis: database-related problems, inflated citation records, bias in citation rates and crediting of multi-author papers. We present a step-by-step protocol to address these problems. By applying this protocol, reviewers can avoid most of the pitfalls associated with the pure numbers of indicators and achieve a fast but fair evaluation of a scientist's performance. We as ecologists should accept complexity not only in our research but also in our research evaluation and should encourage scientists of other disciplines to do so as well.     

A reverse engineering approach to the suppression of citation biases reveals universal properties of citation distributions

The large amount of information contained in bibliographic databases has recently boosted the use of citations, and other indicators based on citation numbers, as tools for the quantitative assessment of scientific research. Citations counts are often interpreted as proxies for the scientific influence of papers, journals, scholars, and institutions. However, a rigorous and scientifically grounded methodology for a correct use of citation counts is still missing. In particular, cross-disciplinary comparisons in terms of raw citation counts systematically favors scientific disciplines with higher citation and publication rates. Here we perform an exhaustive study of the citation patterns of millions of papers, and derive a simple transformation of citation counts able to suppress the disproportionate citation counts among scientific domains. We find that the transformation is well described by a power-law function, and that the parameter values of the transformation are typical features of each scientific discipline. Universal properties of citation patterns descend therefore from the fact that citation distributions for papers in a specific field are all part of the same family of univariate distributions.     

基于Web of Science的合成生物学文献计量分析

合成生物学是一个新兴的交叉学科,近年来得到了广泛关注。本文以1992-2012年期间Web of Science数据库收录的5012篇与合成生物学相关的论文为研究对象,进行年代分布、地域分布、机构分布、研究热点等方面的计量分析,以探究合成生物领域的研究实力分布、研究热点与发展动态。通过文献计量分析发现合成生物学领域近十年来发展迅猛,美国等发达国家占据主动。我国在该领域的论文产出数量可观,但学术影响力有待提高。研究的热点主要集中在基因调控网络构建、基因基因组合成、功能回路设计等方面。     

The Citation Wake of Publications Detects Nobel Laureates' Papers

For several decades, a leading paradigm of how to quantitatively assess scientific research has been the analysis of the aggregated citation information in a set of scientific publications. Although the representation of this information as a citation network has already been coined in the 1960s, it needed the systematic indexing of scientific literature to allow for impact metrics that actually made use of this network as a whole, improving on the then prevailing metrics that were almost exclusively based on the number of direct citations. However, besides focusing on the assignment of credit, the paper citation network can also be studied in terms of the proliferation of scientific ideas. Here we introduce a simple measure based on the shortest-paths in the paper''s in-component or, simply speaking, on the shape and size of the wake of a paper within the citation network. Applied to a citation network containing Physical Review publications from more than a century, our approach is able to detect seminal articles which have introduced concepts of obvious importance to the further development of physics. We observe a large fraction of papers co-authored by Nobel Prize laureates in physics among the top-ranked publications.     

Biased Citation Practice and Taxonomic Parochialism

Citation bias in scientific literature is as widespread as unwelcome. Among other drawbacks, it has a detrimental influence by the wide use of citation statistics for political decisions on resource distribution. Here I ask whether a taxonomic citation bias exists in behavioural studies. The analysis revealed that (1) the taxonomic citation bias is on average large, with nearly a quarter of articles in a haphazardly chosen sample referring to the studied taxon alone, and (2) behavioural studies on mammals and birds show a significantly larger bias than work on other taxa. In contrast, research themes and questions studied do not seem to influence the taxonomic citation bias. Authors of different regions of the world do not differ in their taxonomic citation bias, and the number of papers cited in an article does not relate to the degree of bias. I discuss potential reasons for the substantial citation bias revealed by this analysis and conclude that taxonomic parochialism is a likely cause.     

Contribution of case reports to brain metastases research: systematic review and analysis of pattern of citation

Research activity related to different aspects of prevention, prediction, diagnosis and treatment of brain metastases has increased during recent years. One of the major databases (Scopus) contains 942 scientific articles that were published during the 5-year time period 2006–2010. Of these, 195 (21%) reported on single patient cases and 12 (1%) were reports of 2 cases. Little is known about their influence on advancement of the field or scientific merits. Do brain metastases case reports attract attention and provide stimuli for further research or do they go largely unrecognized? Different measures of impact, visibility and quality of published research are available, each with its own pros and cons. For the present evaluation, article citation rate was chosen. The median number of citations overall and stratified by year of publication was 0, except for the year 2006 when it was 2. As compared to other articles, case reports remained more often without citation (p<0.05 except for 2006 data). All case reports with 10 or more citations (n = 6) reported on newly introduced anticancer drugs, which commonly are prescribed to treat extracranial metastases, and the responses observed in single patients with brain metastases. Average annual numbers of citations were also calculated. The articles with most citations per year were the same six case reports mentioned above (the only ones that obtained more than 2.0 citations per year). Citations appeared to gradually increase during the first two years after publication but remained on a generally low or modest level. It cannot be excluded that case reports without citation provide interesting information to some clinicians or researchers. Apparently, case reports describing unexpected therapeutic success gain more attention, at least in terms of citation, than others.     

Dissemination biases in ecology: effect sizes matter more than quality

Publication and citation decisions in ecology are likely influenced by many factors, potentially including journal impact factors, direction and magnitude of reported effects, and year of publication. Dissemination bias exists when publication or citation of a study depends on any of these factors. We defined several dissemination biases and determined their prevalence across many sub‐disciplines in ecology, then determined whether or not data quality also affected these biases. We identified dissemination biases in ecology by conducting a meta‐analysis of citation trends for 3867 studies included in 52 meta‐analyses. We correlated effect size, year of publication, impact factor and citation rate within each meta‐analysis. In addition, we explored how data quality as defined in meta‐analyses (sample size or variance) influenced each form of bias. We also explored how the direction of the predicted or observed effect, and the research field, influenced any biases. Year of publication did not influence citation rates. The first papers published in an area reported the strongest effects, and high impact factor journals published the most extreme effects. Effect size was more important than data quality for many publication and citation trends. Dissemination biases appear common in ecology, and although their magnitude was generally small many were associated with theory tenacity, evidenced as tendencies to cite papers that most strongly support our ideas. The consequences of this behavior are amplified by the fact that papers reporting strong effects were often of lower data quality than papers reporting much weaker effects. Furthermore, high impact factor journals published the strongest effects, generally in the absence of any correlation with data quality. Increasing awareness of the prevalence of theory tenacity, confirmation bias, and the inattention to data quality among ecologists is a first step towards reducing the impact of these biases on research in our field.     

Brassicaceae: Chromosome number index and database on CD-Rom

A summary of chromosome numbers has been prepared for the Brassicaceae (Cruciferae) family. Over 9000 chromosome counts have been compiled from the literature. Chromosome numbers are known from 232 of the 338 (68.6%) genera and 1558 of the 3709 (42.0%) of the species in the family. Counts in the excel database have been arranged according to currently accepted taxonomy, and includes for each count, country of origin and reference citation. Electronic supplementary material to this article is available at and is accessible for authorized users.     

Six authors in search of a citation: villains or victims of the Vancouver convention?

OBJECTIVES--To analyse trends in the number of authors per article over the past 10 years. DESIGN--Analysis of articles from random volumes of eight biomedical journals. SUBJECTS--Cell, Nature, Proceedings of the National Academy of Sciences USA (PNAS), Journal of Clinical Investigation (JCI), Biochemical and Biophysical Research Communications (BBRC), Journal of Clinical Oncology (JCO), New England Journal of Medicine (NEJM), Lancet. MAIN OUTCOME MEASURES--Median and modal numbers of authors. RESULTS--All journals except Cell and Nature showed a trend towards increasing authorship numbers over the study period. The trend was most noticeable in journals such as JCO which feature clinical research. General medical journals (Lancet, NEJM) with a median of six to seven authors per article published far fewer seven author than six author studies, which suggests that author number may be influenced by the Vancouver convention which precludes citation of more than six authors. CONCLUSIONS--The phenomenon of expanding authorship in biomedical journal articles is not explained by the hypothesis that newer research technologies have necessitated more extensive collaboration. Rather, the data suggest that conferral of authorship may sometimes have a volitional component which contributes to rising author numbers. It is proposed that replacement of the Vancouver convention with a "first author, last author" citation system may help stem this rise in author numbers.     

An Analysis of Peer-Reviewed Scores and Impact Factors with Different Citation Time Windows: A Case Study of 28 Ophthalmologic Journals

Background

An important attribute of the traditional impact factor was the controversial 2-year citation window. So far, several scholars have proposed using different citation time windows for evaluating journals. However, there is no confirmation whether a longer citation time window would be better. How did the journal evaluation effects of 3IF, 4IF, and 6IF comparing with 2IF and 5IF? In order to understand these questions, we made a comparative study of impact factors with different citation time windows with the peer-reviewed scores of ophthalmologic journals indexed by Science Citation Index Expanded (SCIE) database.

Methods

The peer-reviewed scores of 28 ophthalmologic journals were obtained through a self-designed survey questionnaire. Impact factors with different citation time windows (including 2IF, 3IF, 4IF, 5IF, and 6IF) of 28 ophthalmologic journals were computed and compared in accordance with each impact factor’s definition and formula, using the citation analysis function of the Web of Science (WoS) database. An analysis of the correlation between impact factors with different citation time windows and peer-reviewed scores was carried out.

Results

Although impact factor values with different citation time windows were different, there was a high level of correlation between them when it came to evaluating journals. In the current study, for ophthalmologic journals’ impact factors with different time windows in 2013, 3IF and 4IF seemed the ideal ranges for comparison, when assessed in relation to peer-reviewed scores. In addition, the 3-year and 4-year windows were quite consistent with the cited peak age of documents published by ophthalmologic journals.

Research Limitations

Our study is based on ophthalmology journals and we only analyze the impact factors with different citation time window in 2013, so it has yet to be ascertained whether other disciplines (especially those with a later cited peak) or other years would follow the same or similar patterns.

Originality/ Value

We designed the survey questionnaire ourselves, specifically to assess the real influence of journals. We used peer-reviewed scores to judge the journal evaluation effect of impact factors with different citation time windows. The main purpose of this study was to help researchers better understand the role of impact factors with different citation time windows in journal evaluation.     

Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research

Background

Citation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research.

Methodology

A visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact.

Results

Visualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields: Cardiac & cardiovascular systems, Clinical neurology, and Surgery. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.

Conclusions

Popular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.     

An increasing citation black hole in ecology and evolution

Citations published in online supplementary material (OSM) are invisible to search engines used to calculate citation counts, potentially negatively impacting popular performance indices and journal rankings that rely on citation counts for quantification. To quantify the number of citations that are “lost” in OSM, we conducted a systematic survey of supplementary citation practices in four high‐ranking, society‐run journals from two geographical locations (Europe and North America). In 2012, 6% of all citations were only included in the OSM and were therefore not included in citation counts. We found a significant increase in the number of references invisible to citation counting services over the last two decades. A solution to this problem is urgently required and could include journal indexing of citations in OSM or the inclusion of all references in the main text.     

Scientometric sorting by importance for literatures on life cycle assessments and some related methodological discussions

Purpose

This paper aims to sort the literatures on life cycle assessments (LCA) by their respective importance through citation and co-citation analysis and to further discuss the strengths and weaknesses of these kinds of scientometric methods in the case of LCA research.

Methods

CiteSpace II was used to generate document co-citation networks based on 3,824 articles retrieved from the ISI Web of Science database on this topic.

Results

Table 1 provides the top 50 highest cited documents in the LCA field. Here, we use two indicators, i.e., citation frequency in citation analysis and betweenness centrality metric in co-citation analysis, to measure the importance of these LCA literatures.

Conclusions

Citation and co-citation analysis are useful for environmental scientists and engineers to get a better understanding of the inner structure of LCA research. However, like all other research methods, this kind of analysis has some limitations. On the one hand, Scientometric studies and related software are very dependent on ISI Web of Science database, but considering the ISI Web of Science only began to track the LCA field fairly recently, the Scopus database would probably give a fuller picture. On the other hand, since the essence of scientometrics analysis is outsiders commenting insiders, so with only citation and co-citation analysis, to our understanding of the past, present, and future of LCA field, is insufficient.     

《中国真菌学杂志》高被引论文分析

目的对中国真菌学杂志高被引论文进行文献计量分析,为杂志发展及中国真菌学研究提供参考。方法登陆中国知网CNKI数据库（http：／／www．cnki．net／）,选择刊名“中国真菌学杂志”,时间跨度限定为2006—2012年,检索数据,检索结果按照被引频次降序排列,选择排名排名前100位的文献进行数据分析和统计。结果2006—2012共刊发论文843篇,截止2013年6月9日,总被引频次2132次,篇均被引频次2．53次,单篇最高被引频次42次,被引频次〉=4次的文章173篇,占所有文献量的20．5％。高被引论文主要集中在2006、2007年两年,机构主要集中于上海、北京、四川等地;高被引论文研究内容主要集中于真菌感染的临床流行病学调查研究、真菌感染的临床分子诊断研究、抗真菌药物的开发研制及机制探讨以及真菌临床诊治专家共识等方面;中医药治疗真菌感染也是目前研究的热点。结论《中国真菌学杂志》刊发论文基本符合Pareto定律,但论文质量偏低;真菌学研究地区发展不平衡;临床流行病学研究、真菌感染分子诊断以及真菌感染的中医药治疗是研究的热点。     

Limitations on diversity in basic science departments

It has been over 30 years since the beginning of efforts to improve diversity in academia. We can identify four major stages: (1) early and continuing efforts to diversify the pipeline by increasing numbers of women and minorities getting advanced degrees, particularly in science, technology, engineering, and math (STEM); (2) requiring academic institutions to develop their own "affirmative action plans" for hiring and promotion; (3) introducing mentoring programs and coping strategies to help women and minorities deal with faculty practices from an earlier era; (4) asking academic institutions to rethink their practices and policies with an eye toward enabling more faculty diversity, a process known as institutional transformation. The thesis of this article is that research-intensive basic science departments of highly ranked U.S. medical schools are stuck at stage 3, resulting in a less diverse tenured and tenure-track faculty than seen in well-funded science departments of major universities. A review of Web-based records of research-intensive departments in universities with both medical school and nonmedical school departments indicates that the proportion of women and Black faculty in science departments of medical schools is lower than the proportion in similarly research-intensive university science departments. Expectations for faculty productivity in research-intensive medical school departments versus university-based departments may lead to these differences in faculty diversity.     

A Method for the Automated,Reliable Retrieval of Publication-Citation Records

Background

Publication records and citation indices often are used to evaluate academic performance. For this reason, obtaining or computing them accurately is important. This can be difficult, largely due to a lack of complete knowledge of an individual''s publication list and/or lack of time available to manually obtain or construct the publication-citation record. While online publication search engines have somewhat addressed these problems, using raw search results can yield inaccurate estimates of publication-citation records and citation indices.

Methodology

In this paper, we present a new, automated method that produces estimates of an individual''s publication-citation record from an individual''s name and a set of domain-specific vocabulary that may occur in the individual''s publication titles. Because this vocabulary can be harvested directly from a research web page or online (partial) publication list, our method delivers an easy way to obtain estimates of a publication-citation record and the relevant citation indices. Our method works by applying a series of stringent name and content filters to the raw publication search results returned by an online publication search engine. In this paper, our method is run using Google Scholar, but the underlying filters can be easily applied to any existing publication search engine. When compared against a manually constructed data set of individuals and their publication-citation records, our method provides significant improvements over raw search results. The estimated publication-citation records returned by our method have an average sensitivity of and specificity of (in contrast to raw search result specificity of less than 10%). When citation indices are computed using these records, the estimated indices are within of the true value, compared to raw search results which have overestimates of, on average, .

Conclusions

These results confirm that our method provides significantly improved estimates over raw search results, and these can either be used directly for large-scale (departmental or university) analysis or further refined manually to quickly give accurate publication-citation records.     

《环境昆虫学报》2001-2010年载文被引统计分析

利用"中国学术期刊文献评价统计分析系统",对《环境昆虫学报》载文在2001～2010年所刊载论文的被引用情况进行统计分析。结果表明,在2001～2010年间《环境昆虫学报》10年共载文504篇,有343篇论文,被引率为68.06%;被引用1904次,篇均被引次数为5.55,单篇被引最高次数为40次;被引用5次以下的论文占63.57%;引证期刊262种,引用3次以下的期刊占74.81%。同时,对被引论文作者进行了系统分析,特别阐述了高被引论文的详细信息,从而提出从统计得出的高被引频次论文作者及研究热点着手,加强组稿,突出栏目特色,提高刊物对高质量论文的吸纳能力。     

PageRank as a method to rank biomedical literature by importance

Background

Optimal ranking of literature importance is vital in overcoming article overload. Existing ranking methods are typically based on raw citation counts, giving a sum of ‘inbound’ links with no consideration of citation importance. PageRank, an algorithm originally developed for ranking webpages at the search engine, Google, could potentially be adapted to bibliometrics to quantify the relative importance weightings of a citation network. This article seeks to validate such an approach on the freely available, PubMed Central open access subset (PMC-OAS) of biomedical literature.

Results

On-demand cloud computing infrastructure was used to extract a citation network from over 600,000 full-text PMC-OAS articles. PageRanks and citation counts were calculated for each node in this network. PageRank is highly correlated with citation count (R?=?0.905, P?<?0.01) and we thus validate the former as a surrogate of literature importance. Furthermore, the algorithm can be run in trivial time on cheap, commodity cluster hardware, lowering the barrier of entry for resource-limited open access organisations.

Conclusions

PageRank can be trivially computed on commodity cluster hardware and is linearly correlated with citation count. Given its putative benefits in quantifying relative importance, we suggest it may enrich the citation network, thereby overcoming the existing inadequacy of citation counts alone. We thus suggest PageRank as a feasible supplement to, or replacement of, existing bibliometric ranking methods.

     

The Kardashian index: a measure of discrepant social media profile for scientists

In the era of social media there are now many different ways that a scientist can build their public profile; the publication of high-quality scientific papers being just one. While social media is a valuable tool for outreach and the sharing of ideas, there is a danger that this form of communication is gaining too high a value and that we are losing sight of key metrics of scientific value, such as citation indices. To help quantify this, I propose the ‘Kardashian Index’, a measure of discrepancy between a scientist’s social media profile and publication record based on the direct comparison of numbers of citations and Twitter followers.     

A Principal Component Analysis of 39 Scientific Impact Measures

Background

The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact.

Methodology

We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data.

Conclusions

Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.