Is Qualitative Research Second Class Science? A Quantitative Longitudinal Examination of Qualitative Research in Medical Journals

Background

Qualitative research appears to be gaining acceptability in medical journals. Yet, little is actually known about the proportion of qualitative research and factors affecting its publication. This study describes the proportion of qualitative research over a 10 year period and correlates associated with its publication.

Design

A quantitative longitudinal examination of the proportion of original qualitative research in 67 journals of general medicine during a 10 year period (1998–2007). The proportion of qualitative research was determined by dividing original qualitative studies published (numerator) by all original research articles published (denominator). We used a generalized estimating equations approach to assess the longitudinal association between the proportion of qualitative studies and independent variables (i.e. journals'' country of publication and impact factor; editorial/methodological papers discussing qualitative research; and specific journal guidelines pertaining to qualitative research).

Findings

A 2.9% absolute increase and 3.4-fold relative increase in qualitative research publications occurred over a 10 year period (1.2% in 1998 vs. 4.1% in 2007). The proportion of original qualitative research was independently and significantly associated with the publication of editorial/methodological papers in the journal (b = 3.688, P = 0.012); and with qualitative research specifically mentioned in guidelines for authors (b = 6.847, P<0.001). Additionally, a higher proportion of qualitative research was associated only with journals published in the UK in comparison to other countries, yet with borderline statistical significance (b = 1.776, P = 0.075). The journals'' impact factor was not associated with the publication of qualitative research.

Conclusions

Despite an increase in the proportion of qualitative research in medical journals over a 10 year period, the proportion remains low. Journals'' policies pertaining to qualitative research, as expressed by the appearance of specific guidelines and editorials/methodological papers on the subject, are independently associated with the publication of original qualitative research; irrespective of the journals'' impact factor.     

Reporting science and conflicts of interest in the lay press

Background

Forthright reporting of financial ties and conflicts of interest of researchers is associated with public trust in and esteem for the scientific enterprise.

Methods/Principal Findings

We searched Lexis/Nexis Academic News for the top news stories in science published in 2004 and 2005. We conducted a content analysis of 1152 newspaper stories. Funders of the research were identified in 38% of stories, financial ties of the researchers were reported in 11% of stories, and 5% reported financial ties of sources quoted. Of 73 stories not reporting on financial ties, 27% had financial ties publicly disclosed in scholarly journals.

Conclusions/Significance

Because science journalists often did not report conflict of interest information, adherence to gold-standard recommendations for science journalism was low. Journalists work under many different constraints, but nonetheless news reports of scientific research were incomplete, potentially eroding public trust in science.     

Consistency of financial interest disclosures in the biomedical literature: the case of coronary stents

Background

Disclosure of authors'' financial interests has been proposed as a strategy for protecting the integrity of the biomedical literature. We examined whether authors'' financial interests were disclosed consistently in articles on coronary stents published in 2006.

Methodology/Principal Findings

We searched PubMed for English-language articles published in 2006 that provided evidence or guidance regarding the use of coronary artery stents. We recorded article characteristics, including information about authors'' financial disclosures. The main outcome measures were the prevalence, nature, and consistency of financial disclosures. There were 746 articles, 2985 authors, and 135 journals in the database. Eighty-three percent of the articles did not contain disclosure statements for any author (including declarations of no interests). Only 6% of authors had an article with a disclosure statement. In comparisons between articles by the same author, the types of disagreement were as follows: no disclosure statements vs declarations of no interests (64%); specific disclosures vs no disclosure statements (34%); and specific disclosures vs declarations of no interests (2%). Among the 75 authors who disclosed at least 1 relationship with an organization, there were 2 cases (3%) in which the organization was disclosed in every article the author wrote.

Conclusions/Significance

In the rare instances when financial interests were disclosed, they were not disclosed consistently, suggesting that there are problems with transparency in an area of the literature that has important implications for patient care. Our findings suggest that the inconsistencies we observed are due to both the policies of journals and the behavior of some authors.     

Empirical Study of Data Sharing by Authors Publishing in PLoS Journals

Background

Many journals now require authors share their data with other investigators, either by depositing the data in a public repository or making it freely available upon request. These policies are explicit, but remain largely untested. We sought to determine how well authors comply with such policies by requesting data from authors who had published in one of two journals with clear data sharing policies.

Methods and Findings

We requested data from ten investigators who had published in either PLoS Medicine or PLoS Clinical Trials. All responses were carefully documented. In the event that we were refused data, we reminded authors of the journal''s data sharing guidelines. If we did not receive a response to our initial request, a second request was made. Following the ten requests for raw data, three investigators did not respond, four authors responded and refused to share their data, two email addresses were no longer valid, and one author requested further details. A reminder of PLoS''s explicit requirement that authors share data did not change the reply from the four authors who initially refused. Only one author sent an original data set.

Conclusions

We received only one of ten raw data sets requested. This suggests that journal policies requiring data sharing do not lead to authors making their data sets available to independent investigators.     

Temporal trends in the impact factor of European versus USA biomedical journals

Background

The impact factors of biomedical journals tend to rise over time. We sought to assess the trend in the impact factor, during the past decade, of journals published on behalf of United States (US) and European scientific societies, in four select biomedical subject categories (Biology, Cell Biology, Critical Care Medicine, and Infectious Diseases).

Methods

We identified all journals included in the above-mentioned subject categories of Thomson Reuters Journal Citation Reports® for the years 1999, 2002, 2005, and 2008. We selected those that were published on behalf of US or European scientific societies, as documented in journal websites.

Results

We included 167 journals (35 in the subject category of Biology, 79 in Cell Biology, 27 in Critical Care Medicine, and 26 in Infectious Diseases). Between 1999 and 2008, the percentage increase in the impact factor of the European journals was higher than for the US journals (73.7±110.0% compared with 39.7±70.0%, p = 0.049). Regarding specific subject categories, the percentage change in the factor of the European journals tended to be higher than the respective US journals for Cell Biology (61.7% versus 16.3%), Critical Care Medicine (212.4% versus 65.4%), Infectious Diseases (88.3% versus 48.7%), whereas the opposite was observed for journals in Biology (41.0% versus 62.5%).

Conclusion

Journals published on behalf of European scientific societies, in select biomedical fields, may tend to close the “gap” in impact factor compared with those of US societies.

What''s Already Known About This Topic?

The impact factors of biomedical journals tend to rise through years. The leading positions in productivity in biomedical research are held by developed countries, including those from North America and Western Europe.

What Does This Article Add?

The journals from European biomedical scientific societies tended, over the past decade, to increase their impact factor more than the respective US journals.     

Academic impact of a public electronic health database: bibliometric analysis of studies using the general practice research database

Background

Studies that use electronic health databases as research material are getting popular but the influence of a single electronic health database had not been well investigated yet. The United Kingdom''s General Practice Research Database (GPRD) is one of the few electronic health databases publicly available to academic researchers. This study analyzed studies that used GPRD to demonstrate the scientific production and academic impact by a single public health database.

Methodology and Findings

A total of 749 studies published between 1995 and 2009 with ‘General Practice Research Database’ as their topics, defined as GPRD studies, were extracted from Web of Science. By the end of 2009, the GPRD had attracted 1251 authors from 22 countries and been used extensively in 749 studies published in 193 journals across 58 study fields. Each GPRD study was cited 2.7 times by successive studies. Moreover, the total number of GPRD studies increased rapidly, and it is expected to reach 1500 by 2015, twice the number accumulated till the end of 2009. Since 17 of the most prolific authors (1.4% of all authors) contributed nearly half (47.9%) of GPRD studies, success in conducting GPRD studies may accumulate. The GPRD was used mainly in, but not limited to, the three study fields of “Pharmacology and Pharmacy”, “General and Internal Medicine”, and “Public, Environmental and Occupational Health”. The UK and United States were the two most active regions of GPRD studies. One-third of GRPD studies were internationally co-authored.

Conclusions

A public electronic health database such as the GPRD will promote scientific production in many ways. Data owners of electronic health databases at a national level should consider how to reduce access barriers and to make data more available for research.     

Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices

Background

Research needs to be reported transparently so readers can critically assess the strengths and weaknesses of the design, conduct, and analysis of studies. Reporting guidelines have been developed to inform reporting for a variety of study designs. The objective of this study was to identify whether there is a need to develop a reporting guideline for survey research.

Methods and Findings

We conducted a three-part project: (1) a systematic review of the literature (including “Instructions to Authors” from the top five journals of 33 medical specialties and top 15 general and internal medicine journals) to identify guidance for reporting survey research; (2) a systematic review of evidence on the quality of reporting of surveys; and (3) a review of reporting of key quality criteria for survey research in 117 recently published reports of self-administered surveys. Fewer than 7% of medical journals (n = 165) provided guidance to authors on survey research despite a majority having published survey-based studies in recent years. We identified four published checklists for conducting or reporting survey research, none of which were validated. We identified eight previous reviews of survey reporting quality, which focused on issues of non-response and accessibility of questionnaires. Our own review of 117 published survey studies revealed that many items were poorly reported: few studies provided the survey or core questions (35%), reported the validity or reliability of the instrument (19%), defined the response rate (25%), discussed the representativeness of the sample (11%), or identified how missing data were handled (11%).

Conclusions

There is limited guidance and no consensus regarding the optimal reporting of survey research. The majority of key reporting criteria are poorly reported in peer-reviewed survey research articles. Our findings highlight the need for clear and consistent reporting guidelines specific to survey research. Please see later in the article for the Editors'' Summary     

Sharing data for public health research by members of an international online diabetes social network

Background

Surveillance and response to diabetes may be accelerated through engaging online diabetes social networks (SNs) in consented research. We tested the willingness of an online diabetes community to share data for public health research by providing members with a privacy-preserving social networking software application for rapid temporal-geographic surveillance of glycemic control.

Methods and Findings

SN-mediated collection of cross-sectional, member-reported data from an international online diabetes SN entered into a software applicaction we made available in a “Facebook-like” environment to enable reporting, charting and optional sharing of recent hemoglobin A1c values through a geographic display. Self-enrollment by 17% (n = 1,136) of n = 6,500 active members representing 32 countries and 50 US states. Data were current with 83.1% of most recent A1c values reported obtained within the past 90 days. Sharing was high with 81.4% of users permitting data donation to the community display. 34.1% of users also displayed their A1cs on their SN profile page. Users selecting the most permissive sharing options had a lower average A1c (6.8%) than users not sharing with the community (7.1%, p = .038). 95% of users permitted re-contact. Unadjusted aggregate A1c reported by US users closely resembled aggregate 2007–2008 NHANES estimates (respectively, 6.9% and 6.9%, p = 0.85).

Conclusions

Success within an early adopter community demonstrates that online SNs may comprise efficient platforms for bidirectional communication with and data acquisition from disease populations. Advancing this model for cohort and translational science and for use as a complementary surveillance approach will require understanding of inherent selection and publication (sharing) biases in the data and a technology model that supports autonomy, anonymity and privacy.     

Epidemiology, quality and reporting characteristics of systematic reviews of traditional Chinese medicine interventions published in Chinese journals

Background

Systematic reviews (SRs) of TCM have become increasingly popular in China and have been published in large numbers. This review provides the first examination of epidemiological characteristics of these SRs as well as compliance with the PRISMA and AMSTAR guidelines.

Objectives

To examine epidemiological and reporting characteristics as well as methodological quality of SRs of TCM published in Chinese journals.

Methods

Four Chinese databases were searched (CBM, CSJD, CJFD and Wanfang Database) for SRs of TCM, from inception through Dec 2009. Data were extracted into Excel spreadsheets. The PRISMA and AMSTAR checklists were used to assess reporting characteristics and methodological quality, respectively.

Results

A total of 369 SRs were identified, most (97.6%) of which used the terms systematic review or meta-analysis in the title. None of the reviews had been updated. Half (49.8%) were written by clinicians and nearly half (47.7%) were reported in specialty journals. The impact factors of 45.8% of the journals published in were zero. The most commonly treated conditions were diseases of the circulatory and digestive disease. Funding sources were not reported for any reviews. Most (68.8%) reported information about quality assessment, while less than half (43.6%) reported assessing for publication bias. Statistical mistakes appeared in one-third (29.3%) of reviews and most (91.9%) did not report on conflict of interest.

Conclusions

While many SRs of TCM interventions have been published in Chinese journals, the quality of these reviews is troubling. As a potential key source of information for clinicians and researchers, not only were many of these reviews incomplete, some contained mistakes or were misleading. Focusing on improving the quality of SRs of TCM, rather than continuing to publish them in great quantity, is urgently needed in order to increase the value of these studies.     

Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy

Objective

We assessed the adequacy of randomized controlled trial (RCT) registration, changes to registration data and reporting completeness for articles in ICMJE journals during 2.5 years after registration requirement policy.

Methods

For a set of 149 reports of 152 RCTs with ClinicalTrials.gov registration number, published from September 2005 to April 2008, we evaluated the completeness of 9 items from WHO 20-item Minimum Data Set relevant for assessing trial quality. We also assessed changes to the registration elements at the Archive site of ClinicalTrials.gov and compared published and registry data.

Results

RCTs were mostly registered before 13 September 2005 deadline (n = 101, 66.4%); 118 (77.6%) started recruitment before and 31 (20.4%) after registration. At the time of registration, 152 RCTs had a total of 224 missing registry fields, most commonly ‘Key secondary outcomes’ (44.1% RCTs) and ‘Primary outcome’ (38.8%). More RCTs with post-registration recruitment had missing Minimum Data Set items than RCTs with pre-registration recruitment: 57/118 (48.3%) vs. 24/31 (77.4%) (χ² ₁ = 7.255, P = 0.007). Major changes in the data entries were found for 31 (25.2%) RCTs. The number of RCTs with differences between registered and published data ranged from 21 (13.8%) for Study type to 118 (77.6%) for Target sample size.

Conclusions

ICMJE journals published RCTs with proper registration but the registration data were often not adequate, underwent substantial changes in the registry over time and differed in registered and published data. Editors need to establish quality control procedures in the journals so that they continue to contribute to the increased transparency of clinical trials.     

Quality of pharmaceutical industry press releases based on original research

Background

Press releases are a popular vehicle to disseminate health information to the lay media. While the quality of press releases issued by scientific conferences and medical journals has been questioned, no efforts to assess pharmaceutical industry press releases have been made. Therefore, we sought to systematically examine pharmaceutical company press releases about original research for measures of quality.

Methodolgy/Principal Findings

Press releases issued by the ten top selling, international pharmaceutical companies in the year 2005 were selected for evaluation. A total of 1028 electronic press releases were issued and 235 were based on original research. More than half (59%) reported results presented at a scientific meeting. Twenty-one percent of releases were not explicit about the source of original data. While harms or adverse events were commonly cited (76%), study limitations were rarely noted (6%). Almost one-third (29%) of releases did not quantify study results. Studies presented in abstract form were subsequently published within at least 20 months in 53% of cases.

Conclusions

Pharmaceutical company press releases frequently report basic study details. However, readers should be cautioned by the preliminary nature of the data and lack of identified limitations. Methods to improve the reporting and interpretation of drug company press releases are desirable to prevent misleading media coverage.     

Relation between the global burden of disease and randomized clinical trials conducted in Latin America published in the five leading medical journals

Background

Since 1990 non communicable diseases and injuries account for the majority of death and disability-adjusted life years in Latin America. We analyzed the relationship between the global burden of disease and Randomized Clinical Trials (RCTs) conducted in Latin America that were published in the five leading medical journals.

Methodology/Principal Findings

We included all RCTs in humans, exclusively conducted in Latin American countries, and published in any of the following journals: Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, Lancet, and New England Journal of Medicine. We described the trials and reported the number of RCTs according to the main categories of the global burden of disease. Sixty-six RCTs were identified. Communicable diseases accounted for 38 (57%) reports. Maternal, perinatal, and nutritional conditions accounted for 19 (29%) trials. Non-communicable diseases represent 48% of the global burden of disease but only 14% of reported trials. No trial addressed injuries despite its 18% contribution to the burden of disease in 2000.

Conclusions/Significance

A poor correlation between the burden of disease and RCTs publications was found. Non communicable diseases and injuries account for up to two thirds of the burden of disease in Latin America but these topics are seldom addressed in published RCTs in the selected sample of journals. Funding bodies of health research and editors should be aware of the increasing burden of non communicable diseases and injuries occurring in Latin America to ensure that this growing epidemic is not neglected in the research agenda and not affected by publication bias.     

Commercial serological tests for the diagnosis of active pulmonary and extrapulmonary tuberculosis: an updated systematic review and meta-analysis

Background

Serological (antibody detection) tests for tuberculosis (TB) are widely used in developing countries. As part of a World Health Organization policy process, we performed an updated systematic review to assess the diagnostic accuracy of commercial serological tests for pulmonary and extrapulmonary TB with a focus on the relevance of these tests in low- and middle-income countries.

Methods and Findings

We used methods recommended by the Cochrane Collaboration and GRADE approach for rating quality of evidence. In a previous review, we searched multiple databases for papers published from 1 January 1990 to 30 May 2006, and in this update, we add additional papers published from that period until 29 June 2010. We prespecified subgroups to address heterogeneity and summarized test performance using bivariate random effects meta-analysis. For pulmonary TB, we included 67 studies (48% from low- and middle-income countries) with 5,147 participants. For all tests, estimates were variable for sensitivity (0% to 100%) and specificity (31% to 100%). For anda-TB IgG, the only test with enough studies for meta-analysis, pooled sensitivity was 76% (95% CI 63%–87%) in smear-positive (seven studies) and 59% (95% CI 10%–96%) in smear-negative (four studies) patients; pooled specificities were 92% (95% CI 74%–98%) and 91% (95% CI 79%–96%), respectively. Compared with ELISA (pooled sensitivity 60% [95% CI 6%–65%]; pooled specificity 98% [95% CI 96%–99%]), immunochromatographic tests yielded lower pooled sensitivity (53%, 95% CI 42%–64%) and comparable pooled specificity (98%, 95% CI 94%–99%). For extrapulmonary TB, we included 25 studies (40% from low- and middle-income countries) with 1,809 participants. For all tests, estimates were variable for sensitivity (0% to 100%) and specificity (59% to 100%). Overall, quality of evidence was graded very low for studies of pulmonary and extrapulmonary TB.

Conclusions

Despite expansion of the literature since 2006, commercial serological tests continue to produce inconsistent and imprecise estimates of sensitivity and specificity. Quality of evidence remains very low. These data informed a recently published World Health Organization policy statement against serological tests. Please see later in the article for the Editors'' Summary