Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France

Background

Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.

Methodology and Principal Findings

We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.

Conclusions

Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.     

The validity of peer review in a general medicine journal

All the opinions in this article are those of the authors and should not be construed to reflect, in any way, those of the Department of Veterans Affairs.

Background

Our study purpose was to assess the predictive validity of reviewer quality ratings and editorial decisions in a general medicine journal.

Methods

Submissions to the Journal of General Internal Medicine (JGIM) between July 2004 and June 2005 were included. We abstracted JGIM peer review quality ratings, verified the publication status of all articles and calculated an impact factor for published articles (Rw) by dividing the 3-year citation rate by the average for this group of papers; an Rw>1 indicates a greater than average impact.

Results

Of 507 submissions, 128 (25%) were published in JGIM, 331 rejected (128 with review) and 48 were either not resubmitted after revision was requested or were withdrawn by the author. Of 331 rejections, 243 were published elsewhere. Articles published in JGIM had a higher citation rate than those published elsewhere (Rw: 1.6 vs. 1.1, p = 0.002). Reviewer quality ratings of article quality had good internal consistency and reviewer recommendations markedly influenced publication decisions. There was no quality rating cutpoint that accurately distinguished high from low impact articles. There was a stepwise increase in Rw for articles rejected without review, rejected after review or accepted by JGIM (Rw 0.60 vs. 0.87 vs. 1.56, p<0.0005). However, there was low agreement between reviewers for quality ratings and publication recommendations. The editorial publication decision accurately discriminated high and low impact articles in 68% of submissions. We found evidence of better accuracy with a greater number of reviewers.

Conclusions

The peer review process largely succeeds in selecting high impact articles and dispatching lower impact ones, but the process is far from perfect. While the inter-rater reliability between individual reviewers is low, the accuracy of sorting is improved with a greater number of reviewers.     

Acute Myocardial Infarction in Sub-Saharan Africa: The Need for Data

Background

Trends in the prevalence of acute myocardial infarction in sub-Saharan Africa have not been well described, despite growing recognition of the increasing burden of cardiovascular disease in low- and middle-income countries. The aim of this systematic review was to describe the prevalence of acute myocardial infarction in sub-Saharan Africa.

Methods

We searched PubMed, EMBASE, Global Health Archive, CINAHL, and Web of Science, and conducted reference and citation analyses. Inclusion criteria were: observational studies, studies that reported incidence or prevalence of acute myocardial infarction, studies conducted in sub-Saharan Africa, and studies that defined acute myocardial infarction by EKG changes or elevation of cardiac biomarkers. Studies conducted prior to 1992 were excluded. Two independent reviewers analyzed titles and abstracts, full-texts, and references and citations. These reviewers also performed quality assessment and data extraction. Quality assessment was conducted with a validated scale for observational studies.

Findings

Of 2292 records retrieved, seven studies met all inclusion criteria. These studies included a total of 92,378 participants from highly heterogeneous study populations in five different countries. Methodological quality assessment demonstrated scores ranging from 3 to 7 points (on an 8-point scale). Prevalence of acute myocardial infarction ranged from 0.1 to 10.4% among the included studies.

Interpretation

There is insufficient population-based data describing the prevalence of acute myocardial infarction in sub-Saharan Africa. Well-designed registries and surveillance studies that capture the broad and diverse population with acute myocardial infarction in sub-Saharan Africa using common diagnostic criteria are critical in order to guide prevention and treatment strategies.

Registration

Registered in International Prospective Register of Systematic Reviews (PROSPERO) Database #CRD42012003161.     

Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?

Background

The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement.

Objective

We analyze the relationship between peers’ ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields.

Methods and Materials

We analyze peers’ ratings for 2333 applications. We also gathered principal investigators’ research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on the 2002-2006 period. First, we analyzed the distribution of granted and rejected proposals considering a given set of bibliometric indicators to test if there are significant differences. Then, we applied a multiple logistic regression analysis to determine if bibliometric indicators can explain by themselves the concession of grant proposals.

Results

63.4% of the applications were funded. Bibliometric indicators for accepted proposals showed a better previous performance than for those rejected; however the correlation between peer review and bibliometric indicators is very heterogeneous among most areas. The logistic regression analysis showed that the main bibliometric indicators that explain the granting of research proposals in most cases are the output (number of published articles) and the number of papers published in journals that belong to the first quartile ranking of the Journal Citations Report.

Discussion

Bibliometric indicators predict the concession of grant proposals at least as well as peer ratings. Social Sciences and Education are the only areas where no relation was found, although this may be due to the limitations of the Web of Science’s coverage. These findings encourage the use of bibliometric indicators as a complement to peer review in most of the analyzed areas.     

Reliability of the peer-review process for adverse event rating

Background

Adverse events are poor patient outcomes caused by medical care. Their identification requires the peer-review of poor outcomes, which may be unreliable. Combining physician ratings might improve the accuracy of adverse event classification.

Objective

To evaluate the variation in peer-reviewer ratings of adverse outcomes; determine the impact of this variation on estimates of reviewer accuracy; and determine the number of reviewers who judge an adverse event occurred that is required to ensure that the true probability of an adverse event exceeded 50%, 75% or 95%.

Methods

Thirty physicians rated 319 case reports giving details of poor patient outcomes following hospital discharge. They rated whether medical management caused the outcome using a six-point ordinal scale. We conducted latent class analyses to estimate the prevalence of adverse events as well as the sensitivity and specificity of each reviewer. We used this model and Bayesian calculations to determine the probability that an adverse event truly occurred to each patient as function of their number of positive ratings.

Results

The overall median score on the 6-point ordinal scale was 3 (IQR 2,4) but the individual rater median score ranged from a minimum of 1 (in four reviewers) to a maximum median score of 5. The overall percentage of cases rated as an adverse event was 39.7% (3798/9570). The median kappa for all pair-wise combinations of the 30 reviewers was 0.26 (IQR 0.16, 0.42; Min = −0.07, Max = 0.62). Reviewer sensitivity and specificity for adverse event classification ranged from 0.06 to 0.93 and 0.50 to 0.98, respectively. The estimated prevalence of adverse events using a latent class model with a common sensitivity and specificity for all reviewers (0.64 and 0.83 respectively) was 47.6%. For patients to have a 95% chance of truly having an adverse event, at least 3 of 3 reviewers are required to deem the outcome an adverse event.

Conclusion

Adverse event classification is unreliable. To be certain that a case truly represents an adverse event, there needs to be agreement among multiple reviewers.     

Computer-Assisted Interpretation of the EEG Background Pattern: A Clinical Evaluation

Objective

Interpretation of the EEG background pattern in routine recordings is an important part of clinical reviews. We evaluated the feasibility of an automated analysis system to assist reviewers with evaluation of the general properties in the EEG background pattern.

Methods

Quantitative EEG methods were used to describe the following five background properties: posterior dominant rhythm frequency and reactivity, anterior-posterior gradients, presence of diffuse slow-wave activity and asymmetry. Software running the quantitative methods were given to ten experienced electroencephalographers together with 45 routine EEG recordings and computer-generated reports. Participants were asked to review the EEGs by visual analysis first, and afterwards to compare their findings with the generated reports and correct mistakes made by the system. Corrected reports were returned for comparison.

Results

Using a gold-standard derived from the consensus of reviewers, inter-rater agreement was calculated for all reviewers and for automated interpretation. Automated interpretation together with most participants showed high (kappa > 0.6) agreement with the gold standard. In some cases, automated analysis showed higher agreement with the gold standard than participants. When asked in a questionnaire after the study, all participants considered computer-assisted interpretation to be useful for every day use in routine reviews.

Conclusions

Automated interpretation methods proved to be accurate and were considered to be useful by all participants.

Significance

Computer-assisted interpretation of the EEG background pattern can bring consistency to reviewing and improve efficiency and inter-rater agreement.     

Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

Objectives

To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers.

Methods

Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ).

Results

In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ  =  0.02, 95%CI: −0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ  = 0.19), ranging from κ  =  −0.04 (“Other bias”) to κ  =  0.62 (“Sequence generation”). There was also no agreement (κ  =  −0.29, 95%CI: −0.81, 0.35]) in the overall RoB assessment at the meta-analysis level.

Conclusions

Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.     

Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs

Objective

To compare expert assessment with bibliometric indicators as tools to assess the quality and importance of scientific research papers.

Methods and Materials

Shortly after their publication in 2005, the quality and importance of a cohort of nearly 700 Wellcome Trust (WT) associated research papers were assessed by expert reviewers; each paper was reviewed by two WT expert reviewers. After 3 years, we compared this initial assessment with other measures of paper impact.

Results

Shortly after publication, 62 (9%) of the 687 research papers were determined to describe at least a ‘major addition to knowledge’ –6 were thought to be ‘landmark’ papers. At an aggregate level, after 3 years, there was a strong positive association between expert assessment and impact as measured by number of citations and F1000 rating. However, there were some important exceptions indicating that bibliometric measures may not be sufficient in isolation as measures of research quality and importance, and especially not for assessing single papers or small groups of research publications.

Conclusion

When attempting to assess the quality and importance of research papers, we found that sole reliance on bibliometric indicators would have led us to miss papers containing important results as judged by expert review. In particular, some papers that were highly rated by experts were not highly cited during the first three years after publication. Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation.     

Risk Factors for Repetition of Self-Harm: A Systematic Review of Prospective Hospital-Based Studies

Background

Self-harm entails high costs to individuals and society in terms of suicide risk, morbidity and healthcare expenditure. Repetition of self-harm confers yet higher risk of suicide and risk assessment of self-harm patients forms a key component of the health care management of self-harm patients. To date, there has been no systematic review published which synthesises the extensive evidence on risk factors for repetition.

Objective

This review is intended to identify risk factors for prospective repetition of self-harm after an index self-harm presentation, irrespective of suicidal intent.

Data sources

PubMed, PsychInfo and Scirus were used to search for relevant publications. We included cohort studies which examining factors associated with prospective repetition among those presenting with self-harm to emergency departments. Journal articles, abstracts, letters and theses in any language published up to June 2012 were considered. Studies were quality-assessed and synthesised in narrative form.

Results

A total of 129 studies, including 329,001 participants, met our inclusion criteria. Some factors were studied extensively and were found to have a consistent association with repetition. These included previous self-harm, personality disorder, hopelessness, history of psychiatric treatment, schizophrenia, alcohol abuse/dependence, drug abuse/dependence, and living alone. However, the sensitivity values of these measures varied greatly across studies. Psychological risk factors and protective factors have been relatively under-researched but show emerging associations with repetition. Composite risk scales tended to have high sensitivity but poor specificity.

Conclusions

Many risk factors for repetition of self-harm match risk factors for initiation of self-harm, but the most consistent evidence for increased risk of repetition comes from long-standing psychosocial vulnerabilities, rather than characteristics of an index episode. The current review will enhance prediction of self-harm and assist in the efficient allocation of intervention resources.     

Addressing Dichotomous Data for Participants Excluded from Trial Analysis: A Guide for Systematic Reviewers

Introduction

Systematic reviewer authors intending to include all randomized participants in their meta-analyses need to make assumptions about the outcomes of participants with missing data.

Objective

The objective of this paper is to provide systematic reviewer authors with a relatively simple guidance for addressing dichotomous data for participants excluded from analyses of randomized trials.

Methods

This guide is based on a review of the Cochrane handbook and published methodological research. The guide deals with participants excluded from the analysis who were considered ‘non-adherent to the protocol’ but for whom data are available, and participants with missing data.

Results

Systematic reviewer authors should include data from ‘non-adherent’ participants excluded from the primary study authors'' analysis but for whom data are available. For missing, unavailable participant data, authors may conduct a complete case analysis (excluding those with missing data) as the primary analysis. Alternatively, they may conduct a primary analysis that makes plausible assumptions about the outcomes of participants with missing data. When the primary analysis suggests important benefit, sensitivity meta-analyses using relatively extreme assumptions that may vary in plausibility can inform the extent to which risk of bias impacts the confidence in the results of the primary analysis. The more plausible assumptions draw on the outcome event rates within the trial or in all trials included in the meta-analysis. The proposed guide does not take into account the uncertainty associated with assumed events.

Conclusions

This guide proposes methods for handling participants excluded from analyses of randomized trials. These methods can help in establishing the extent to which risk of bias impacts meta-analysis results.     

Fluorescence-Based Methods for Detecting Caries Lesions: Systematic Review,Meta-Analysis and Sources of Heterogeneity

Background

Fluorescence-based methods have been proposed to aid caries lesion detection. Summarizing and analysing findings of studies about fluorescence-based methods could clarify their real benefits.

Objective

We aimed to perform a comprehensive systematic review and meta-analysis to evaluate the accuracy of fluorescence-based methods in detecting caries lesions.

Data Source

Two independent reviewers searched PubMed, Embase and Scopus through June 2012 to identify papers/articles published. Other sources were checked to identify non-published literature.

Study Eligibility Criteria, Participants and Diagnostic Methods

The eligibility criteria were studies that: (1) have assessed the accuracy of fluorescence-based methods of detecting caries lesions on occlusal, approximal or smooth surfaces, in both primary or permanent human teeth, in the laboratory or clinical setting; (2) have used a reference standard; and (3) have reported sufficient data relating to the sample size and the accuracy of methods.

Study Appraisal and Synthesis Methods

A diagnostic 2×2 table was extracted from included studies to calculate the pooled sensitivity, specificity and overall accuracy parameters (Diagnostic Odds Ratio and Summary Receiver-Operating curve). The analyses were performed separately for each method and different characteristics of the studies. The quality of the studies and heterogeneity were also evaluated.

Results

Seventy five studies met the inclusion criteria from the 434 articles initially identified. The search of the grey or non-published literature did not identify any further studies. In general, the analysis demonstrated that the fluorescence-based method tend to have similar accuracy for all types of teeth, dental surfaces or settings. There was a trend of better performance of fluorescence methods in detecting more advanced caries lesions. We also observed moderate to high heterogeneity and evidenced publication bias.

Conclusions

Fluorescence-based devices have similar overall performance; however, better accuracy in detecting more advanced caries lesions has been observed.     

Use of Kampo Diagnosis in Randomized Controlled Trials of Kampo Products in Japan: A Systematic Review

Background

The Committee for Evidence-based Medicine (EBM) of the Japan Society for Oriental Medicine started compiling Evidence Reports of Kampo Treatment (EKAT) in 2007. EKAT is a compilation of structured abstracts of randomized controlled trials (RCTs), along with comments by a third party reviewer. As of 31 December, 2012, there were 378 RCTs of Kampo medicines in Japan. The primary research question of this study is “How frequently is Kampo diagnosis used in RCTs of Kampo medicines?” The secondary research question is “When is Kampo diagnosis used in RCTs?”

Materials and Methods

The structured abstract (SA) of each RCT article was reviewed to examine how Kampo diagnosis was used in RCTs, especially how Kampo diagnosis was used in the randomization process.

Results

Kampo diagnosis was used before randomization in 27 RCTs (7.1%), after randomization in 31 RCTs (8.2%), and not used in 320 RCTs (84.7%). Before randomization, Kampo diagnosis was used as a criterion for inclusion in 10 RCTs, criterion for exclusion in 9 RCTs, and criteria for both inclusion and exclusion in 2 RCTs. Kampo formulas were determined according to Kampo diagnosis in 7 RCTs. After randomization, subgroup analyses according to Kampo diagnosis were done in 27 RCTs, and grade of disease severity at Kampo diagnosis was used for analysis as an endpoint in 4 RCTs.

Conclusions

Kampo diagnosis was used before randomization only in approximately 15% of RCTs, and the number of RCT articles using Kampo diagnosis after randomization was almost the same as that before randomization. Further studies to determine the good RCTs conforming to CONSORT requirements and good systematic reviews conforming to PRISMA requirements are needed to clarify the significance of Kampo diagnosis.     

Validation of ACR/EULAR definition of remission in rheumatoid arthritis from RA practice: the ESPOIR cohort

Introduction

In development of the American College of Rheumatology (ACR)/European League Against Rheumatism (EULAR) rheumatoid arthritis (RA) remission definitions using clinical trials data, one criterion used to compare different remission definitions was whether, compared with those not in remission, those in remission had evidence of later disease stability defined by x-ray and functional status. Validation of the RA remission criteria using observational study data is necessary before recommending their use in practice.

Methods

Using data from those who met RA criteria in the ESPOIR cohort, we matched each person in remission with a person not in remission and then carried out analyses comparing later stability of x-ray and health assessment questionnaire (HAQ) between the two groups. We compared the predictive validity of the same candidate definitions of remission evaluated in the ACR/EULAR process. To minimize potential bias and produce more stable results, we used a bootstrap resampling approach to select those not in remission, repeating the sample matching analysis process 500 times.

Results

Results were similar to those of clinical trials analyzed for the ACR/EULAR remission criteria. Specifically, the ACR/EULAR remission definitions using either an simple disease activity index (SDAI) ≤ 3.3, clinical disease activity index (CDAI) ≤ 2.8 or a definition of remission requiring tender joint count, swollen joint count, patient global assessment all ≤ 1 performed as well or better than other candidate definitions of remission in terms of predicting later x-ray and function stability.

Conclusions

ACR/EULAR definitions of remission developed for trials are similarly valid in observational studies in RA and could be used in practice.     

Patient and Healthcare Provider Barriers to Hypertension Awareness,Treatment and Follow Up: A Systematic Review and Meta-Analysis of Qualitative and Quantitative Studies

Background

Although the importance of detecting, treating, and controlling hypertension has been recognized for decades, the majority of patients with hypertension remain uncontrolled. The path from evidence to practice contains many potential barriers, but their role has not been reviewed systematically. This review aimed to synthesize and identify important barriers to hypertension control as reported by patients and healthcare providers.

Methods

Electronic databases MEDLINE, EMBASE and Global Health were searched systematically up to February 2013. Two reviewers independently selected eligible studies. Two reviewers categorized barriers based on a theoretical framework of behavior change. The theoretical framework suggests that a change in behavior requires a strong commitment to change [intention], the necessary skills and abilities to adopt the behavior [capability], and an absence of health system and support constraints.

Findings

Twenty-five qualitative studies and 44 quantitative studies met the inclusion criteria. In qualitative studies, health system barriers were most commonly discussed in studies of patients and health care providers. Quantitative studies identified disagreement with clinical recommendations as the most common barrier among health care providers. Quantitative studies of patients yielded different results: lack of knowledge was the most common barrier to hypertension awareness. Stress, anxiety and depression were most commonly reported as barriers that hindered or delayed adoption of a healthier lifestyle. In terms of hypertension treatment adherence, patients mostly reported forgetting to take their medication. Finally, priority setting barriers were most commonly reported by patients in terms of following up with their health care providers.

Conclusions

This review identified a wide range of barriers facing patients and health care providers pursuing hypertension control, indicating the need for targeted multi-faceted interventions. More methodologically rigorous studies that encompass the range of barriers and that include low- and middle-income countries are required in order to inform policies to improve hypertension control.