Assessment of transparency indicators across the biomedical literature: How open is open?

Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.

This study uses novel open source automated tools to monitor transparency across all 2.75 million open access articles on PubMed Central, discovering that different disciplines, journals and publishers abide by principles of transparency to varying degrees over time.     

Empirical Study of Data Sharing by Authors Publishing in PLoS Journals

Background

Many journals now require authors share their data with other investigators, either by depositing the data in a public repository or making it freely available upon request. These policies are explicit, but remain largely untested. We sought to determine how well authors comply with such policies by requesting data from authors who had published in one of two journals with clear data sharing policies.

Methods and Findings

We requested data from ten investigators who had published in either PLoS Medicine or PLoS Clinical Trials. All responses were carefully documented. In the event that we were refused data, we reminded authors of the journal''s data sharing guidelines. If we did not receive a response to our initial request, a second request was made. Following the ten requests for raw data, three investigators did not respond, four authors responded and refused to share their data, two email addresses were no longer valid, and one author requested further details. A reminder of PLoS''s explicit requirement that authors share data did not change the reply from the four authors who initially refused. Only one author sent an original data set.

Conclusions

We received only one of ten raw data sets requested. This suggests that journal policies requiring data sharing do not lead to authors making their data sets available to independent investigators.     

The Oligopoly of Academic Publishers in the Digital Era

The consolidation of the scientific publishing industry has been the topic of much debate within and outside the scientific community, especially in relation to major publishers’ high profit margins. However, the share of scientific output published in the journals of these major publishers, as well as its evolution over time and across various disciplines, has not yet been analyzed. This paper provides such analysis, based on 45 million documents indexed in the Web of Science over the period 1973-2013. It shows that in both natural and medical sciences (NMS) and social sciences and humanities (SSH), Reed-Elsevier, Wiley-Blackwell, Springer, and Taylor & Francis increased their share of the published output, especially since the advent of the digital era (mid-1990s). Combined, the top five most prolific publishers account for more than 50% of all papers published in 2013. Disciplines of the social sciences have the highest level of concentration (70% of papers from the top five publishers), while the humanities have remained relatively independent (20% from top five publishers). NMS disciplines are in between, mainly because of the strength of their scientific societies, such as the ACS in chemistry or APS in physics. The paper also examines the migration of journals between small and big publishing houses and explores the effect of publisher change on citation impact. It concludes with a discussion on the economics of scholarly publishing.     

Reproducible Research Practices and Transparency across the Biomedical Literature

There is a growing movement to encourage reproducibility and transparency practices in the scientific community, including public access to raw data and protocols, the conduct of replication studies, systematic integration of evidence in systematic reviews, and the documentation of funding and potential conflicts of interest. In this survey, we assessed the current status of reproducibility and transparency addressing these indicators in a random sample of 441 biomedical journal articles published in 2000–2014. Only one study provided a full protocol and none made all raw data directly available. Replication studies were rare (n = 4), and only 16 studies had their data included in a subsequent systematic review or meta-analysis. The majority of studies did not mention anything about funding or conflicts of interest. The percentage of articles with no statement of conflict decreased substantially between 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles reporting statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0% in 2014) increased. Articles published in journals in the clinical medicine category versus other fields were almost twice as likely to not include any information on funding and to have private funding. This study provides baseline data to compare future progress in improving these indicators in the scientific literature.     

Changes in Data Sharing and Data Reuse Practices and Perceptions among Scientists Worldwide

The incorporation of data sharing into the research lifecycle is an important part of modern scholarly debate. In this study, the DataONE Usability and Assessment working group addresses two primary goals: To examine the current state of data sharing and reuse perceptions and practices among research scientists as they compare to the 2009/2010 baseline study, and to examine differences in practices and perceptions across age groups, geographic regions, and subject disciplines. We distributed surveys to a multinational sample of scientific researchers at two different time periods (October 2009 to July 2010 and October 2013 to March 2014) to observe current states of data sharing and to see what, if any, changes have occurred in the past 3–4 years. We also looked at differences across age, geographic, and discipline-based groups as they currently exist in the 2013/2014 survey. Results point to increased acceptance of and willingness to engage in data sharing, as well as an increase in actual data sharing behaviors. However, there is also increased perceived risk associated with data sharing, and specific barriers to data sharing persist. There are also differences across age groups, with younger respondents feeling more favorably toward data sharing and reuse, yet making less of their data available than older respondents. Geographic differences exist as well, which can in part be understood in terms of collectivist and individualist cultural differences. An examination of subject disciplines shows that the constraints and enablers of data sharing and reuse manifest differently across disciplines. Implications of these findings include the continued need to build infrastructure that promotes data sharing while recognizing the needs of different research communities. Moving into the future, organizations such as DataONE will continue to assess, monitor, educate, and provide the infrastructure necessary to support such complex grand science challenges.     

Who Shares? Who Doesn't? Factors Associated with Openly Archiving Raw Research Data

Many initiatives encourage investigators to share their raw datasets in hopes of increasing research efficiency and quality. Despite these investments of time and money, we do not have a firm grasp of who openly shares raw research data, who doesn''t, and which initiatives are correlated with high rates of data sharing. In this analysis I use bibliometric methods to identify patterns in the frequency with which investigators openly archive their raw gene expression microarray datasets after study publication.Automated methods identified 11,603 articles published between 2000 and 2009 that describe the creation of gene expression microarray data. Associated datasets in best-practice repositories were found for 25% of these articles, increasing from less than 5% in 2001 to 30%–35% in 2007–2009. Accounting for sensitivity of the automated methods, approximately 45% of recent gene expression studies made their data publicly available.First-order factor analysis on 124 diverse bibliometric attributes of the data creation articles revealed 15 factors describing authorship, funding, institution, publication, and domain environments. In multivariate regression, authors were most likely to share data if they had prior experience sharing or reusing data, if their study was published in an open access journal or a journal with a relatively strong data sharing policy, or if the study was funded by a large number of NIH grants. Authors of studies on cancer and human subjects were least likely to make their datasets available.These results suggest research data sharing levels are still low and increasing only slowly, and data is least available in areas where it could make the biggest impact. Let''s learn from those with high rates of sharing to embrace the full potential of our research output.     

Elementary and middle level biology topics: a content analysis of Science and Children and Science Scope from 1990 to 2014

Science and Children and Science Scope are peer-reviewed science practitioner journals that publish articles for science educators who teach children of ages 5–10 years and 10–13 years, respectively. As such, these articles are a reflection of the science concepts that are being communicated to science educators. This comprehensive literature review was completed to determine the extent to which the articles included in these journals from 1990 to 2014 focused on biology topics, incorporated other non-science subjects and science disciplines, and encouraged inquiry-based learning. The results indicate that out of the 2701 total articles, 557 (21%) focused on biology. The biology topic covered most often by both journals was animals. The biology topics covered least often were fungi, photosynthesis, respiration and viruses. The findings are taken into consideration and the educational implications for the journals and educators are discussed.     

Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science

Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.     

生物多样性数据论文发表趋势分析

近年来有关科学数据共享的呼声越来越高, 基于同行评审的生物多样性数据论文也受到越来越多的关注, 并出现了一些专门发表数据论文的数据期刊。本文总结了近年来生物多样性数据发表方面的进展, 选择两本代表性数据期刊(Biodiversity Data Journal和Scientific Data), 分析了它们自创刊以来的发文数量、涉及生物类群、文章浏览量和被引次数等指标。结果显示两本数据期刊的发文量都呈稳步增长趋势, 其生物多样性数据论文覆盖了包括动物界、植物界、真菌界在内的众多生物类群, 文章浏览量和被引次数方面也有可喜的表现, 说明数据论文正在被越来越多的研究者所接受。对文章作者国别的分析则显示了不同地区的研究者在发表生物多样性数据论文或数据共享方面的不均衡。建议相关领域的中国研究者和期刊关注生物多样性数据论文和数据共享政策, 更多地践行数据共享。     

Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study

Objective To determine if citation counts at two years could be predicted for clinical articles that pass basic criteria for critical appraisal using data within three weeks of publication from external sources and an online article rating service.Design Retrospective cohort study.Setting Online rating service, Canada.Participants 1274 articles from 105 journals published from January to June 2005, randomly divided into a 60:40 split to provide derivation and validation datasets.Main outcome measures 20 article and journal features, including ratings of clinical relevance and newsworthiness, routinely collected by the McMaster online rating of evidence system, compared with citation counts at two years.Results The derivation analysis showed that the regression equation accounted for 60% of the variation (R²=0.60, 95% confidence interval 0.538 to 0.629). This model applied to the validation dataset gave a similar prediction (R²=0.56, 0.476 to 0.596, shrinkage 0.04; shrinkage measures how well the derived equation matches data from the validation dataset). Cited articles in the top half and top third were predicted with 83% and 61% sensitivity and 72% and 82% specificity. Higher citations were predicted by indexing in numerous databases; number of authors; abstraction in synoptic journals; clinical relevance scores; number of cited references; and original, multicentred, and therapy articles from journals with a greater proportion of articles abstracted.Conclusion Citation counts can be reliably predicted at two years using data within three weeks of publication.     

Case studies in reproducibility

Reproducible research is a concept of providing access to data and software along with published scientific findings. By means of some case studies from different disciplines, we will illustrate reasons why readers should be given the possibility to look at the data and software independently from the authors of the original publication. We report results of a survey comprising 100 papers recently published in Bioinformatics. The main finding is that authors of this journal share a culture of making data available. However, the number of papers where source code for simulation studies or analyzes is available is still rather limited.     

Representation of Women and Pregnant Women in HIV Research: A Limited Systematic Review

Background

HIV-related outcomes may be affected by biological sex and by pregnancy. Including women in general and pregnant women in particular in HIV-related research is important for generalizability of findings.

Objective

To characterize representation of pregnant and non-pregnant women in HIV-related research conducted in general populations.

Data Sources

All HIV-related articles published in fifteen journals from January to March of 2011. We selected the top five journals by 2010 impact factor, in internal medicine, infectious diseases, and HIV/AIDS.

Study Eligibility Criteria

HIV-related studies reporting original research on questions applicable to both men and women of reproductive age were considered; studies were excluded if they did not include individual-level patient data.

Study appraisal and synthesis methods.

Articles were doubly reviewed and abstracted; discrepancies were resolved through consensus. We recorded proportion of female study participants, whether pregnant women were included or excluded, and other key factors.

Results

In total, 2014 articles were published during this period. After screening, 259 articles were included as original HIV-related research reporting individual-level data; of these, 226 were determined to be articles relevant to both men and women of reproductive age. In these articles, women were adequately represented within geographic region. The vast majority of published articles, 183/226 (81%), did not mention pregnancy (or related issues); still fewer included pregnant women (n=33), reported numbers of pregnant women (n=19), or analyzed using pregnancy status (n=9).

Limitations

Data were missing for some key variables, including pregnancy. The time period over which published works were evaluated was relatively short.

Conclusions and implications of key findings.

The under-reporting and inattention to pregnancy in the HIV literature may reduce policy-makers’ ability to set evidence-based policy around HIV/AIDS care for pregnant women and women of child-bearing age.     

Three options for citation tracking: Google Scholar,Scopus and Web of Science

Background

Researchers turn to citation tracking to find the most influential articles for a particular topic and to see how often their own published papers are cited. For years researchers looking for this type of information had only one resource to consult: the Web of Science from Thomson Scientific. In 2004 two competitors emerged – Scopus from Elsevier and Google Scholar from Google. The research reported here uses citation analysis in an observational study examining these three databases; comparing citation counts for articles from two disciplines (oncology and condensed matter physics) and two years (1993 and 2003) to test the hypothesis that the different scholarly publication coverage provided by the three search tools will lead to different citation counts from each.

Methods

Eleven journal titles with varying impact factors were selected from each discipline (oncology and condensed matter physics) using the Journal Citation Reports (JCR). All articles published in the selected titles were retrieved for the years 1993 and 2003, and a stratified random sample of articles was chosen, resulting in four sets of articles. During the week of November 7–12, 2005, the citation counts for each research article were extracted from the three sources. The actual citing references for a subset of the articles published in 2003 were also gathered from each of the three sources.

Results

For oncology 1993 Web of Science returned the highest average number of citations, 45.3. Scopus returned the highest average number of citations (8.9) for oncology 2003. Web of Science returned the highest number of citations for condensed matter physics 1993 and 2003 (22.5 and 3.9 respectively). The data showed a significant difference in the mean citation rates between all pairs of resources except between Google Scholar and Scopus for condensed matter physics 2003. For articles published in 2003 Google Scholar returned the largest amount of unique citing material for oncology and Web of Science returned the most for condensed matter physics.

Conclusion

This study did not identify any one of these three resources as the answer to all citation tracking needs. Scopus showed strength in providing citing literature for current (2003) oncology articles, while Web of Science produced more citing material for 2003 and 1993 condensed matter physics, and 1993 oncology articles. All three tools returned some unique material. Our data indicate that the question of which tool provides the most complete set of citing literature may depend on the subject and publication year of a given article.

     

Water,Water, Everywhere: Defining and Assessing Data Sharing in Academia

Sharing of research data has begun to gain traction in many areas of the sciences in the past few years because of changing expectations from the scientific community, funding agencies, and academic journals. National Science Foundation (NSF) requirements for a data management plan (DMP) went into effect in 2011, with the intent of facilitating the dissemination and sharing of research results. Many projects that were funded during 2011 and 2012 should now have implemented the elements of the data management plans required for their grant proposals. In this paper we define ‘data sharing’ and present a protocol for assessing whether data have been shared and how effective the sharing was. We then evaluate the data sharing practices of researchers funded by the NSF at Oregon State University in two ways: by attempting to discover project-level research data using the associated DMP as a starting point, and by examining data sharing associated with journal articles that acknowledge NSF support. Sharing at both the project level and the journal article level was not carried out in the majority of cases, and when sharing was accomplished, the shared data were often of questionable usability due to access, documentation, and formatting issues. We close the article by offering recommendations for how data producers, journal publishers, data repositories, and funding agencies can facilitate the process of sharing data in a meaningful way.     

Comparative study of scientific publications in urology and nephrology journals originating from USA, china and Japan (2001-2010)

Background

In the past decade, scientific research has developed rapidly in China, but the growth seems to vary widely between different disciplines. In this study, we aimed to compare the quantity and quality of publications in urology and nephrology journals from USA, China and Japan.

Methods

Journals listed in the “Urology and Nephrology” category of Science Citation Index Expanded subject categories were included. Scientific papers in these journals written by researchers from USA, Japan and China were retrieved from the “PubMed” and “Web of Knowledge” online databases.

Results

The annual number of total scientific articles increased significantly from 2001 to 2010 in China, and has ranked second in the world since 2006. In the field of urology and nephrology, the annual number increased significantly from 2001 to 2010 in USA and China; but not in Japan. The share of articles increased significantly over time in China, decreased significantly in Japan, and remained unchanged in USA. In 2010, USA contributed 32.17% of the total world output in urology and nephrology field and ranked 1^st; Japan contributed 5.19% and ranked 5^th; China contributed 3.83% and ranked 9^th. Publications from USA had the highest accumulated IFs and the highest total citations of articles (USA>Japan>China, p<0.001). No significant difference was found in average IF among the three countries. USA published the most articles in the top 10 urology and nephrology journals (USA(35165)>Japan(6704)>China(2233), p<0.001). Researchers from USA published more clinical trials and randomized controlled trials than Japan and China (USA>Japan>China, p<0.001).

Conclusion

Although China has undergone significant increase in annual number and percentage of scientific publication in urology and nephrology journals in the past decade, it still lags far behind USA and Japan in the field of urology and nephrology in terms of quantity and quality.     

Impact of Article Language in Multi-Language Medical Journals - a Bibliometric Analysis of Self-Citations and Impact Factor

Background

In times of globalization there is an increasing use of English in the medical literature. The aim of this study was to analyze the influence of English-language articles in multi-language medical journals on their international recognition – as measured by a lower rate of self-citations and higher impact factor (IF).

Methods and Findings

We analyzed publications in multi-language journals in 2008 and 2009 using the Web of Science (WoS) of Thomson Reuters (former Institute of Scientific Information) and PubMed as sources of information. The proportion of English-language articles during the period was compared with both the share of self-citations in the year 2010 and the IF with and without self-citations. Multivariable linear regression analysis was performed to analyze these factors as well as the influence of the journals‘ countries of origin and of the other language(s) used in publications besides English.We identified 168 multi-language journals that were listed in WoS as well as in PubMed and met our criteria. We found a significant positive correlation of the share of English articles in 2008 and 2009 with the IF calculated without self-citations (Pearson r=0.56, p = <0.0001), a correlation with the overall IF (Pearson r = 0.47, p = <0.0001) and with the cites to years of IF calculation (Pearson r = 0.34, p = <0.0001), and a weak negative correlation with the share of self-citations (Pearson r = -0.2, p = 0.009). The IF without self-citations also correlated with the journal‘s country of origin – North American journals had a higher IF compared to Middle and South American or European journals.

Conclusion

Our findings suggest that a larger share of English articles in multi-language medical journals is associated with greater international recognition. Fewer self-citations were found in multi-language journals with a greater share of original articles in English.     

Graph use in psychology and other sciences

Since the early 19th century, graphs have been recognised as an effective method of analysing and representing scientific data. However, levels of graph use have varied widely since then, partly due to increasing reliance on inferential statistics in some fields. Recent studies indicate that graph use is closely related to the 'hardness' of scientific disciplines, and that this finding holds for journal articles and textbooks across the subfields of psychology. In the area of animal behaviour, journals devote about one-sixth of their page space to graphs, a level of graph use approximating that of biology and physics. Implications for the training of scientists in the use of visual displays are considered.