Evaluating research and impact: a bibliometric analysis of research by the NIH/NIAID HIV/AIDS clinical trials networks

Evaluative bibliometrics uses advanced techniques to assess the impact of scholarly work in the context of other scientific work and usually compares the relative scientific contributions of research groups or institutions. Using publications from the National Institute of Allergy and Infectious Diseases (NIAID) HIV/AIDS extramural clinical trials networks, we assessed the presence, performance, and impact of papers published in 2006-2008. Through this approach, we sought to expand traditional bibliometric analyses beyond citation counts to include normative comparisons across journals and fields, visualization of co-authorship across the networks, and assess the inclusion of publications in reviews and syntheses. Specifically, we examined the research output of the networks in terms of the a) presence of papers in the scientific journal hierarchy ranked on the basis of journal influence measures, b) performance of publications on traditional bibliometric measures, and c) impact of publications in comparisons with similar publications worldwide, adjusted for journals and fields. We also examined collaboration and interdisciplinarity across the initiative, through network analysis and modeling of co-authorship patterns. Finally, we explored the uptake of network produced publications in research reviews and syntheses. Overall, the results suggest the networks are producing highly recognized work, engaging in extensive interdisciplinary collaborations, and having an impact across several areas of HIV-related science. The strengths and limitations of the approach for evaluation and monitoring research initiatives are discussed.     

Network Effects on Scientific Collaborations

Background

The analysis of co-authorship network aims at exploring the impact of network structure on the outcome of scientific collaborations and research publications. However, little is known about what network properties are associated with authors who have increased number of joint publications and are being cited highly.

Methodology/Principal Findings

Measures of social network analysis, for example network centrality and tie strength, have been utilized extensively in current co-authorship literature to explore different behavioural patterns of co-authorship networks. Using three SNA measures (i.e., degree centrality, closeness centrality and betweenness centrality), we explore scientific collaboration networks to understand factors influencing performance (i.e., citation count) and formation (tie strength between authors) of such networks. A citation count is the number of times an article is cited by other articles. We use co-authorship dataset of the research field of ‘steel structure’ for the year 2005 to 2009. To measure the strength of scientific collaboration between two authors, we consider the number of articles co-authored by them. In this study, we examine how citation count of a scientific publication is influenced by different centrality measures of its co-author(s) in a co-authorship network. We further analyze the impact of the network positions of authors on the strength of their scientific collaborations. We use both correlation and regression methods for data analysis leading to statistical validation. We identify that citation count of a research article is positively correlated with the degree centrality and betweenness centrality values of its co-author(s). Also, we reveal that degree centrality and betweenness centrality values of authors in a co-authorship network are positively correlated with the strength of their scientific collaborations.

Conclusions/Significance

Authors’ network positions in co-authorship networks influence the performance (i.e., citation count) and formation (i.e., tie strength) of scientific collaborations.     

Canada's Neglected Tropical Disease Research Network: Who's in the Core—Who's on the Periphery?

Background

This study designed and applied accessible yet systematic methods to generate baseline information about the patterns and structure of Canada''s neglected tropical disease (NTD) research network; a network that, until recently, was formed and functioned on the periphery of strategic Canadian research funding.

Methodology

Multiple methods were used to conduct this study, including: (1) a systematic bibliometric procedure to capture archival NTD publications and co-authorship data; (2) a country-level “core-periphery” network analysis to measure and map the structure of Canada''s NTD co-authorship network including its size, density, cliques, and centralization; and (3) a statistical analysis to test the correlation between the position of countries in Canada''s NTD network (“k-core measure”) and the quantity and quality of research produced.

Principal Findings

Over the past sixty years (1950–2010), Canadian researchers have contributed to 1,079 NTD publications, specializing in Leishmania, African sleeping sickness, and leprosy. Of this work, 70% of all first authors and co-authors (n = 4,145) have been Canadian. Since the 1990s, however, a network of international co-authorship activity has been emerging, with representation of researchers from 62 different countries; largely researchers from OECD countries (e.g. United States and United Kingdom) and some non-OECD countries (e.g. Brazil and Iran). Canada has a core-periphery NTD international research structure, with a densely connected group of OECD countries and some African nations, such as Uganda and Kenya. Sitting predominantly on the periphery of this research network is a cluster of 16 non-OECD nations that fall within the lowest GDP percentile of the network.

Conclusion/Significance

The publication specialties, composition, and position of NTD researchers within Canada''s NTD country network provide evidence that while Canadian researchers currently remain the overall gatekeepers of the NTD research they generate; there is opportunity to leverage existing research collaborations and help advance regions and NTD areas that are currently under-developed.     

The data‐index: An author‐level metric that values impactful data and incentivizes data sharing

Author‐level metrics are a widely used measure of scientific success. The h‐index and its variants measure publication output (number of publications) and research impact (number of citations). They are often used to influence decisions, such as allocating funding or jobs. Here, we argue that the emphasis on publication output and impact hinders scientific progress in the fields of ecology and evolution because it disincentivizes two fundamental practices: generating impactful (and therefore often long‐term) datasets and sharing data. We describe a new author‐level metric, the data‐index, which values both dataset output (number of datasets) and impact (number of data‐index citations), so promotes generating and sharing data as a result. We discuss how it could be implemented and provide user guidelines. The data‐index is designed to complement other metrics of scientific success, as scientific contributions are diverse and our value system should reflect that both for the benefit of scientific progress and to create a value system that is more equitable, diverse, and inclusive. Future work should focus on promoting other scientific contributions, such as communicating science, informing policy, mentoring other scientists, and providing open‐access code and tools.     

Exploring the evolution of sustainable fisheries development: Focusing on ecological,environmental and management issues

Living marine resources are crucial for guaranteeing food security, meeting nutritional needs, generating employment, and solving other human challenges worldwide. Nevertheless, resource overexploitation and environmental contamination pose serious challenges to the sustainable development of fisheries (SDF). Numerous studies have been conducted in various disciplines worldwide to address these challenges. In this study, we collected 4450 journal articles from the Web of Science Core Collection database to help explain the evolution process, current state of affairs, research hotspots, and trends of research on the SDF. Using bibliometric tools, VOSviewer, CiteSpace, and Scimago Graphica, a scientometric analysis was conducted to define the knowledge structure by visualizing the co-occurrence network, co-authorship network, co-citation network, and emergence analysis. The findings indicate that the number of publications in this field are expanding rapidly, and key events related to the SDF have influenced publication numbers. Additionally, performance analysis from the author, journal and national perspectives provides scientific information for researchers. The thematic content on the SDF has also changed to emphasize ecosystem structure and its services.     

Evolution of Cooperation Patterns in Psoriasis Research: Co-Authorship Network Analysis of Papers in Medline (1942–2013)

Background

Although researchers have worked in collaboration since the origins of modern science and the publication of the first scientific journals in the eighteenth century, this phenomenon has acquired exceptional importance in the last several decades. Since the mid-twentieth century, new knowledge has been generated from within an ever-growing network of investigators, working cooperatively in research groups across countries and institutions. Cooperation is a crucial determinant of academic success.

Objective

The aim of the present paper is to analyze the evolution of scientific collaboration at the micro level, with regard to the scientific production generated on psoriasis research.

Methods

A bibliographic search in the Medline database containing the MeSH terms “psoriasis” or “psoriatic arthritis” was carried out. The search results were limited to articles, reviews and letters. After identifying the co-authorships of documents on psoriasis indexed in the Medline database (1942–2013), various bibliometric indicators were obtained, including the average number of authors per document and degree of multi-authorship over time. In addition, we performed a network analysis to study the evolution of certain features of the co-authorship network as a whole: average degree, size of the largest component, clustering coefficient, density and average distance. We also analyzed the evolution of the giant component to characterize the changing research patterns in the field, and we calculated social network indicators for the nodes, namely betweenness and closeness.

Results

The main active research clusters in the area were identified, along with their authors of reference. Our analysis of 28,670 documents sheds light on different aspects related to the evolution of scientific collaboration in the field, including the progressive increase in the mean number of co-authors (which stood at 5.17 in the 2004–2013 decade), and the rise in multi-authored papers signed by many different authors (in the same decade, 25.77% of the documents had between 6 and 9 co-authors, and 10.28% had 10 or more). With regard to the network indicators, the average degree gradually increased up to 10.97 in the study period. The percentage of authors pertaining to the largest component also rose to 73.02% of the authors. The clustering coefficient, on the other hand, remained stable throughout the entire 70-year period, with values hovering around 0.9. Finally, the average distance peaked in the decades 1974–1983 (8.29) and 1984–2003 (8.12) then fell over the next two decades, down to 5.25 in 2004–2013. The construction of the co-authorship network (threshold of collaboration ≥ 10 co-authored works) revealed a giant component of 161 researchers, containing 6 highly cohesive sub-components.

Conclusions

Our study reveals the existence of a growing research community in which collaboration is increasingly important. We can highlight an essential feature associated with scientific collaboration: multi-authored papers, with growing numbers of collaborators contributing to them, are becoming more and more common, therefore the formation of research groups of increasing depth (specialization) and breadth (multidisciplinarity) is now a cornerstone of research success.     

On the Compliance of Women Engineers with a Gendered Scientific System

There has been considerable effort in the last decade to increase the participation of women in engineering through various policies. However, there has been little empirical research on gender disparities in engineering which help underpin the effective preparation, co-ordination, and implementation of the science and technology (S&T) policies. This article aims to present a comprehensive gendered analysis of engineering publications across different specialties and provide a cross-gender analysis of research output and scientific impact of engineering researchers in academic, governmental, and industrial sectors. For this purpose, 679,338 engineering articles published from 2008 to 2013 are extracted from the Web of Science database and 974,837 authorships are analyzed. The structures of co-authorship collaboration networks in different engineering disciplines are examined, highlighting the role of female scientists in the diffusion of knowledge. The findings reveal that men dominate 80% of all the scientific production in engineering. Women engineers publish their papers in journals with higher Impact Factors than their male peers, but their work receives lower recognition (fewer citations) from the scientific community. Engineers—regardless of their gender—contribute to the reproduction of the male-dominated scientific structures through forming and repeating their collaborations predominantly with men. The results of this study call for integration of data driven gender-related policies in existing S&T discourse.     

基于Web of Science的国际海草研究文献计量评价

海草及其与周围环境形成的海草场是近海海洋生态系统的重要组成部分,具有多种重要的生态服务功能。相较于红树林和珊瑚礁,国内外学者和公众对海草的关注度偏低,至今对国际海草研究的现状、趋势和热点等的认识还较为有限。检索了1983-2017年Web of Science的SCI-E数据库中收录的海草研究相关文献,借助文献计量信息可视化分析方法,从年度发文量、研究力量和研究热点与主题等方面进行了较为系统的文献计量评价,以期阐明国内外海草研究的态势与热点。结果表明:1983-2017年间国际海草研究的文献数量总体呈明显上升趋势,该领域发文量排前3位的国家是美国、澳大利亚和西班牙;全球发文量最多的机构是佛罗里达州立大学,其次是西班牙高等科学研究理事会与美国国家海洋和大气管理局;发文总量较多的作者是Duarte CM、Marba N和Fourqurean JW,近3年发文量最高的是学者Santos R;刊文量最多的期刊分别是《Marine Ecology Progress Series》、《Aquatic Botany》和《Estuarine, Coastal and Shelf Science》,刊文量前15期刊中影响因子最高的是《Limnology and Oceanography》(3.969)。在国家、机构与作者合作方面,西班牙、丹麦、荷兰、墨西哥4个国家形成了关系密切的合作网络,西班牙高等科学研究理事会和美国国家海洋和大气管理局是海草研究机构合作网络中的两个中心,Duarte CM、Marba N等较多学者间有紧密的学术合作关系。通过高频关键词的关联关系分析,表明该领域的3个热点研究主题,即环境(污染)胁迫对海草场的影响、海草的生长与生理生态和海草及其相关生态系统的结构与功能。中国科学院、中国海洋大学等机构为我国海草研究做出了重要贡献,但由于研究起步较晚,我国学者的发文量、篇均被引频次等与欧美国家学者存在一定差距,但近几年的发文量增长迅速,预计未来的发展趋势良好。     

Global Myeloma Research Clusters,Output, and Citations: A Bibliometric Mapping and Clustering Analysis

Background

International collaborative research is a mechanism for improving the development of disease-specific therapies and for improving health at the population level. However, limited data are available to assess the trends in research output related to orphan diseases.

Methods and Findings

We used bibliometric mapping and clustering methods to illustrate the level of fragmentation in myeloma research and the development of collaborative efforts. Publication data from Thomson Reuters Web of Science were retrieved for 2005–2009 and followed until 2013. We created a database of multiple myeloma publications, and we analysed impact and co-authorship density to identify scientific collaborations, developments, and international key players over time. The global annual publication volume for studies on multiple myeloma increased from 1,144 in 2005 to 1,628 in 2009, which represents a 43% increase. This increase is high compared to the 24% and 14% increases observed for lymphoma and leukaemia. The major proportion (>90% of publications) was from the US and EU over the study period. The output and impact in terms of citations, identified several successful groups with a large number of intra-cluster collaborations in the US and EU. The US-based myeloma clusters clearly stand out as the most productive and highly cited, and the European Myeloma Network members exhibited a doubling of collaborative publications from 2005 to 2009, still increasing up to 2013.

Conclusion and Perspective

Multiple myeloma research output has increased substantially in the past decade. The fragmented European myeloma research activities based on national or regional groups are progressing, but they require a broad range of targeted research investments to improve multiple myeloma health care.     

Measuring co-authorship and networking-adjusted scientific impact

Appraisal of the scientific impact of researchers, teams and institutions with productivity and citation metrics has major repercussions. Funding and promotion of individuals and survival of teams and institutions depend on publications and citations. In this competitive environment, the number of authors per paper is increasing and apparently some co-authors don't satisfy authorship criteria. Listing of individual contributions is still sporadic and also open to manipulation. Metrics are needed to measure the networking intensity for a single scientist or group of scientists accounting for patterns of co-authorship. Here, I define I(1) for a single scientist as the number of authors who appear in at least I(1) papers of the specific scientist. For a group of scientists or institution, I(n) is defined as the number of authors who appear in at least I(n) papers that bear the affiliation of the group or institution. I(1) depends on the number of papers authored N(p). The power exponent R of the relationship between I(1) and N(p) categorizes scientists as solitary (R>2.5), nuclear (R = 2.25-2.5), networked (R = 2-2.25), extensively networked (R = 1.75-2) or collaborators (R<1.75). R may be used to adjust for co-authorship networking the citation impact of a scientist. I(n) similarly provides a simple measure of the effective networking size to adjust the citation impact of groups or institutions. Empirical data are provided for single scientists and institutions for the proposed metrics. Cautious adoption of adjustments for co-authorship and networking in scientific appraisals may offer incentives for more accountable co-authorship behaviour in published articles.     

Predicting Co-Author Relationship in Medical Co-Authorship Networks

Research collaborations are encouraged because a synergistic effect yielding good results often appears. However, creating and organizing a strong research group is a difficult task. One of the greatest concerns of an individual researcher is locating potential collaborators whose expertise complement his best. In this paper, we propose a method that makes link predictions in co-authorship networks, where topological features between authors such as Adamic/Adar, Common Neighbors, Jaccard''s Coefficient, Preferential Attachment, Katz_β, and PropFlow may be good indicators of their future collaborations. Firstly, these topological features were systematically extracted from the network. Then, supervised models were used to learn the best weights associated with different topological features in deciding co-author relationships. Finally, we tested our models on the co-authorship networks in the research field of Coronary Artery Disease and obtained encouraging accuracy (the precision, recall, F1 score and AUC were, respectively, 0.696, 0.677, 0.671 and 0.742 for Logistic Regression, and respectively, 0.697, 0.678, 0.671 and 0.743 for SVM). This suggests that our models could be used to build and manage strong research groups.     

On the Relation between the Small World Structure and Scientific Activities

The modern science has become more complex and interdisciplinary in its nature which might encourage researchers to be more collaborative and get engaged in larger collaboration networks. Various aspects of collaboration networks have been examined so far to detect the most determinant factors in knowledge creation and scientific production. One of the network structures that recently attracted much theoretical attention is called small world. It has been suggested that small world can improve the information transmission among the network actors. In this paper, using the data on 12 periods of journal publications of Canadian researchers in natural sciences and engineering, the co-authorship networks of the researchers are created. Through measuring small world indicators, the small worldiness of the mentioned network and its relation with researchers’ productivity, quality of their publications, and scientific team size are assessed. Our results show that the examined co-authorship network strictly exhibits the small world properties. In addition, it is suggested that in a small world network researchers expand their team size through getting connected to other experts of the field. This team size expansion may result in higher productivity of the whole team as a result of getting access to new resources, benefitting from the internal referring, and exchanging ideas among the team members. Moreover, although small world network is positively correlated with the quality of the articles in terms of both citation count and journal impact factor, it is negatively related with the average productivity of researchers in terms of the number of their publications.     

Using Co-authorship Networks to Map and Analyse Global Neglected Tropical Disease Research with an Affiliation to Germany

Background

Research on Neglected Tropical Diseases (NTDs) has increased in recent decades, and significant need-gaps in diagnostic and treatment tools remain. Analysing bibliometric data from published research is a powerful method for revealing research efforts, partnerships and expertise. We aim to identify and map NTD research networks in Germany and their partners abroad to enable an informed and transparent evaluation of German contributions to NTD research.

Methodology/Principal Findings

A SCOPUS database search for articles with German author affiliations that were published between 2002 and 2012 was conducted for kinetoplastid and helminth diseases. Open-access tools were used for data cleaning and scientometrics (OpenRefine), geocoding (OpenStreetMaps) and to create (Table2Net), visualise and analyse co-authorship networks (Gephi). From 26,833 publications from around the world that addressed 11 diseases, we identified 1,187 (4.4%) with at least one German author affiliation, and we processed 972 publications for the five most published-about diseases. Of those, we extracted 4,007 individual authors and 863 research institutions to construct co-author networks. The majority of co-authors outside Germany were from high-income countries and Brazil. Collaborations with partners on the African continent remain scattered. NTD research within Germany was distributed among 220 research institutions. We identified strong performers on an individual level by using classic parameters (number of publications, h-index) and social network analysis parameters (betweenness centrality). The research network characteristics varied strongly between diseases.

Conclusions/Significance

The share of NTD publications with German affiliations is approximately half of its share in other fields of medical research. This finding underlines the need to identify barriers and expand Germany’s otherwise strong research activities towards NTDs. A geospatial analysis of research collaborations with partners abroad can support decisions to strengthen research capacity, particularly in low- and middle-income countries, which were less involved in collaborations than high-income countries. Identifying knowledge hubs within individual researcher networks complements traditional scientometric indicators that are used to identify opportunities for collaboration. Using free tools to analyse research processes and output could facilitate data-driven health policies. Our findings contribute to the prioritisation of efforts in German NTD research at a time of impending local and global policy decisions.     

Multiple Citation Indicators and Their Composite across Scientific Disciplines

Many fields face an increasing prevalence of multi-authorship, and this poses challenges in assessing citation metrics. Here, we explore multiple citation indicators that address total impact (number of citations, Hirsch H index [H]), co-authorship adjustment (Schreiber Hm index [Hm]), and author order (total citations to papers as single; single or first; or single, first, or last author). We demonstrate the correlation patterns between these indicators across 84,116 scientists (those among the top 30,000 for impact in a single year [2013] in at least one of these indicators) and separately across 12 scientific fields. Correlation patterns vary across these 12 fields. In physics, total citations are highly negatively correlated with indicators of co-authorship adjustment and of author order, while in other sciences the negative correlation is seen only for total citation impact and citations to papers as single author. We propose a composite score that sums standardized values of these six log-transformed indicators. Of the 1,000 top-ranked scientists with the composite score, only 322 are in the top 1,000 based on total citations. Many Nobel laureates and other extremely influential scientists rank among the top-1,000 with the composite indicator, but would rank much lower based on total citations. Conversely, many of the top 1,000 authors on total citations have had no single/first/last-authored cited paper. More Nobel laureates of 2011–2015 are among the top authors when authors are ranked by the composite score than by total citations, H index, or Hm index; 40/47 of these laureates are among the top 30,000 by at least one of the six indicators. We also explore the sensitivity of indicators to self-citation and alphabetic ordering of authors in papers across different scientific fields. Multiple indicators and their composite may give a more comprehensive picture of impact, although no citation indicator, single or composite, can be expected to select all the best scientists.     

The Citation Wake of Publications Detects Nobel Laureates' Papers

For several decades, a leading paradigm of how to quantitatively assess scientific research has been the analysis of the aggregated citation information in a set of scientific publications. Although the representation of this information as a citation network has already been coined in the 1960s, it needed the systematic indexing of scientific literature to allow for impact metrics that actually made use of this network as a whole, improving on the then prevailing metrics that were almost exclusively based on the number of direct citations. However, besides focusing on the assignment of credit, the paper citation network can also be studied in terms of the proliferation of scientific ideas. Here we introduce a simple measure based on the shortest-paths in the paper''s in-component or, simply speaking, on the shape and size of the wake of a paper within the citation network. Applied to a citation network containing Physical Review publications from more than a century, our approach is able to detect seminal articles which have introduced concepts of obvious importance to the further development of physics. We observe a large fraction of papers co-authored by Nobel Prize laureates in physics among the top-ranked publications.     

Jane goodall and the chimpanzees of gombe: An analysis of publications and their impact on teaching science

Five types of introductory university textbooks (N=37) were analyzed for references and citations pertaining to research on wild chimpanzees. Jane Goodall's publications were cited about three times as often as the publications from field sites other than Gombe and approximately five times more often than other Gombe researchers. Biological anthropology textbooks cited Goodall's work most often, followed by textbooks in general anthropology and cultural anthropology. Psychology and biology textbooks cited Goodall least often. Goodall's most comprehensive work, The Chimpanzees of Gombe: Patterns of Behavior (1986), was the most often cited publication about Gombe's apes, and tool-use was the most cited topic. The number of citations to wild chimpanzees tripled from publications in the 1960s to those in the 1980s, suggesting a growing recognition of primatology in the teaching of science.     

A view of clinical engineering from scientific publications

The aim of this paper was to characterize clinical engineering from the perspective of scientific publications. First of all, the most significant factors that influence the change in health systems are briefly exposed. Then, clinical engineering is defined and its main functions and evolution within the development of the health system is explained. Finally, the author describes the state of the field from the viewpoint of the scientific publications; an apparent lack of interest in the engineering community to publish scientific research was observed. This behavior can be seen in the clear declining tendency in the number of citations and the number of publications (in volume or quantity) in major scientific journals in the field. Finally, current challenges and future developments must be addressed to accomplish a better positioning of the specialty in the publishing world.     

Universality of Citation Distributions for Academic Institutions and Journals

Citations measure the importance of a publication, and may serve as a proxy for its popularity and quality of its contents. Here we study the distributions of citations to publications from individual academic institutions for a single year. The average number of citations have large variations between different institutions across the world, but the probability distributions of citations for individual institutions can be rescaled to a common form by scaling the citations by the average number of citations for that institution. We find this feature seems to be universal for a broad selection of institutions irrespective of the average number of citations per article. A similar analysis for citations to publications in a particular journal in a single year reveals similar results. We find high absolute inequality for both these sets, Gini coefficients being around 0.66 and 0.58 for institutions and journals respectively. We also find that the top 25% of the articles hold about 75% of the total citations for institutions and the top 29% of the articles hold about 71% of the total citations for journals.     

Genealogical Trees of Scientific Papers

Many results have been obtained when studying scientific papers citations databases in a network perspective. Articles can be ranked according to their current in-degree and their future popularity or citation counts can even be predicted. The dynamical properties of such networks and the observation of the time evolution of their nodes started more recently. This work adopts an evolutionary perspective and proposes an original algorithm for the construction of genealogical trees of scientific papers on the basis of their citation count evolution in time. The fitness of a paper now amounts to its in-degree growing trend and a “dying” paper will suddenly see this trend declining in time. It will give birth and be taken over by some of its most prevalent citing “offspring”. Practically, this might be used to trace the successive published milestones of a research field.     

Predicting Scholars' Scientific Impact

We tested the underlying assumption that citation counts are reliable predictors of future success, analyzing complete citation data on the careers of scientists. Our results show that i) among all citation indicators, the annual citations at the time of prediction is the best predictor of future citations, ii) future citations of a scientist''s published papers can be predicted accurately ( for a 1-year prediction, ) but iii) future citations of future work are hardly predictable.