共查询到20条相似文献,搜索用时 15 毫秒
1.
2.
Background
Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.Methodology and Principal Findings
We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.Conclusions
Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable. 相似文献3.
4.
All the opinions in this article are those of the authors and should not be construed to reflect, in any way, those of the Department of Veterans Affairs.
Background
Our study purpose was to assess the predictive validity of reviewer quality ratings and editorial decisions in a general medicine journal.Methods
Submissions to the Journal of General Internal Medicine (JGIM) between July 2004 and June 2005 were included. We abstracted JGIM peer review quality ratings, verified the publication status of all articles and calculated an impact factor for published articles (Rw) by dividing the 3-year citation rate by the average for this group of papers; an Rw>1 indicates a greater than average impact.Results
Of 507 submissions, 128 (25%) were published in JGIM, 331 rejected (128 with review) and 48 were either not resubmitted after revision was requested or were withdrawn by the author. Of 331 rejections, 243 were published elsewhere. Articles published in JGIM had a higher citation rate than those published elsewhere (Rw: 1.6 vs. 1.1, p = 0.002). Reviewer quality ratings of article quality had good internal consistency and reviewer recommendations markedly influenced publication decisions. There was no quality rating cutpoint that accurately distinguished high from low impact articles. There was a stepwise increase in Rw for articles rejected without review, rejected after review or accepted by JGIM (Rw 0.60 vs. 0.87 vs. 1.56, p<0.0005). However, there was low agreement between reviewers for quality ratings and publication recommendations. The editorial publication decision accurately discriminated high and low impact articles in 68% of submissions. We found evidence of better accuracy with a greater number of reviewers.Conclusions
The peer review process largely succeeds in selecting high impact articles and dispatching lower impact ones, but the process is far from perfect. While the inter-rater reliability between individual reviewers is low, the accuracy of sorting is improved with a greater number of reviewers. 相似文献5.
Julian T. Hertz Joseph M. Reardon Clarissa G. Rodrigues Luciano de Andrade Alexander T. Limkakeng Gerald S. Bloomfield Catherine A. Lynch 《PloS one》2014,9(5)
Background
Trends in the prevalence of acute myocardial infarction in sub-Saharan Africa have not been well described, despite growing recognition of the increasing burden of cardiovascular disease in low- and middle-income countries. The aim of this systematic review was to describe the prevalence of acute myocardial infarction in sub-Saharan Africa.Methods
We searched PubMed, EMBASE, Global Health Archive, CINAHL, and Web of Science, and conducted reference and citation analyses. Inclusion criteria were: observational studies, studies that reported incidence or prevalence of acute myocardial infarction, studies conducted in sub-Saharan Africa, and studies that defined acute myocardial infarction by EKG changes or elevation of cardiac biomarkers. Studies conducted prior to 1992 were excluded. Two independent reviewers analyzed titles and abstracts, full-texts, and references and citations. These reviewers also performed quality assessment and data extraction. Quality assessment was conducted with a validated scale for observational studies.Findings
Of 2292 records retrieved, seven studies met all inclusion criteria. These studies included a total of 92,378 participants from highly heterogeneous study populations in five different countries. Methodological quality assessment demonstrated scores ranging from 3 to 7 points (on an 8-point scale). Prevalence of acute myocardial infarction ranged from 0.1 to 10.4% among the included studies.Interpretation
There is insufficient population-based data describing the prevalence of acute myocardial infarction in sub-Saharan Africa. Well-designed registries and surveillance studies that capture the broad and diverse population with acute myocardial infarction in sub-Saharan Africa using common diagnostic criteria are critical in order to guide prevention and treatment strategies.Registration
Registered in International Prospective Register of Systematic Reviews (PROSPERO) Database #CRD42012003161. 相似文献6.
Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?
álvaro Cabezas-Clavijo Nicolás Robinson-García Manuel Escabias Evaristo Jiménez-Contreras 《PloS one》2013,8(6)
Background
The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement.Objective
We analyze the relationship between peers’ ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields.Methods and Materials
We analyze peers’ ratings for 2333 applications. We also gathered principal investigators’ research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on the 2002-2006 period. First, we analyzed the distribution of granted and rejected proposals considering a given set of bibliometric indicators to test if there are significant differences. Then, we applied a multiple logistic regression analysis to determine if bibliometric indicators can explain by themselves the concession of grant proposals.Results
63.4% of the applications were funded. Bibliometric indicators for accepted proposals showed a better previous performance than for those rejected; however the correlation between peer review and bibliometric indicators is very heterogeneous among most areas. The logistic regression analysis showed that the main bibliometric indicators that explain the granting of research proposals in most cases are the output (number of published articles) and the number of papers published in journals that belong to the first quartile ranking of the Journal Citations Report.Discussion
Bibliometric indicators predict the concession of grant proposals at least as well as peer ratings. Social Sciences and Education are the only areas where no relation was found, although this may be due to the limitations of the Web of Science’s coverage. These findings encourage the use of bibliometric indicators as a complement to peer review in most of the analyzed areas. 相似文献7.
8.
9.
Background
Adverse events are poor patient outcomes caused by medical care. Their identification requires the peer-review of poor outcomes, which may be unreliable. Combining physician ratings might improve the accuracy of adverse event classification.Objective
To evaluate the variation in peer-reviewer ratings of adverse outcomes; determine the impact of this variation on estimates of reviewer accuracy; and determine the number of reviewers who judge an adverse event occurred that is required to ensure that the true probability of an adverse event exceeded 50%, 75% or 95%.Methods
Thirty physicians rated 319 case reports giving details of poor patient outcomes following hospital discharge. They rated whether medical management caused the outcome using a six-point ordinal scale. We conducted latent class analyses to estimate the prevalence of adverse events as well as the sensitivity and specificity of each reviewer. We used this model and Bayesian calculations to determine the probability that an adverse event truly occurred to each patient as function of their number of positive ratings.Results
The overall median score on the 6-point ordinal scale was 3 (IQR 2,4) but the individual rater median score ranged from a minimum of 1 (in four reviewers) to a maximum median score of 5. The overall percentage of cases rated as an adverse event was 39.7% (3798/9570). The median kappa for all pair-wise combinations of the 30 reviewers was 0.26 (IQR 0.16, 0.42; Min = −0.07, Max = 0.62). Reviewer sensitivity and specificity for adverse event classification ranged from 0.06 to 0.93 and 0.50 to 0.98, respectively. The estimated prevalence of adverse events using a latent class model with a common sensitivity and specificity for all reviewers (0.64 and 0.83 respectively) was 47.6%. For patients to have a 95% chance of truly having an adverse event, at least 3 of 3 reviewers are required to deem the outcome an adverse event.Conclusion
Adverse event classification is unreliable. To be certain that a case truly represents an adverse event, there needs to be agreement among multiple reviewers. 相似文献10.
Objective
Interpretation of the EEG background pattern in routine recordings is an important part of clinical reviews. We evaluated the feasibility of an automated analysis system to assist reviewers with evaluation of the general properties in the EEG background pattern.Methods
Quantitative EEG methods were used to describe the following five background properties: posterior dominant rhythm frequency and reactivity, anterior-posterior gradients, presence of diffuse slow-wave activity and asymmetry. Software running the quantitative methods were given to ten experienced electroencephalographers together with 45 routine EEG recordings and computer-generated reports. Participants were asked to review the EEGs by visual analysis first, and afterwards to compare their findings with the generated reports and correct mistakes made by the system. Corrected reports were returned for comparison.Results
Using a gold-standard derived from the consensus of reviewers, inter-rater agreement was calculated for all reviewers and for automated interpretation. Automated interpretation together with most participants showed high (kappa > 0.6) agreement with the gold standard. In some cases, automated analysis showed higher agreement with the gold standard than participants. When asked in a questionnaire after the study, all participants considered computer-assisted interpretation to be useful for every day use in routine reviews.Conclusions
Automated interpretation methods proved to be accurate and were considered to be useful by all participants.Significance
Computer-assisted interpretation of the EEG background pattern can bring consistency to reviewing and improve efficiency and inter-rater agreement. 相似文献11.
Susan Armijo-Olivo Maria Ospina Bruno R. da Costa Matthias Egger Humam Saltaji Jorge Fuentes Christine Ha Greta G. Cummings 《PloS one》2014,9(5)
Objectives
To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers.Methods
Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ).Results
In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: −0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = −0.04 (“Other bias”) to κ = 0.62 (“Sequence generation”). There was also no agreement (κ = −0.29, 95%CI: −0.81, 0.35]) in the overall RoB assessment at the meta-analysis level.Conclusions
Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed. 相似文献12.
Objective
To compare expert assessment with bibliometric indicators as tools to assess the quality and importance of scientific research papers.Methods and Materials
Shortly after their publication in 2005, the quality and importance of a cohort of nearly 700 Wellcome Trust (WT) associated research papers were assessed by expert reviewers; each paper was reviewed by two WT expert reviewers. After 3 years, we compared this initial assessment with other measures of paper impact.Results
Shortly after publication, 62 (9%) of the 687 research papers were determined to describe at least a ‘major addition to knowledge’ –6 were thought to be ‘landmark’ papers. At an aggregate level, after 3 years, there was a strong positive association between expert assessment and impact as measured by number of citations and F1000 rating. However, there were some important exceptions indicating that bibliometric measures may not be sufficient in isolation as measures of research quality and importance, and especially not for assessing single papers or small groups of research publications.Conclusion
When attempting to assess the quality and importance of research papers, we found that sole reliance on bibliometric indicators would have led us to miss papers containing important results as judged by expert review. In particular, some papers that were highly rated by experts were not highly cited during the first three years after publication. Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation. 相似文献13.
Background
Self-harm entails high costs to individuals and society in terms of suicide risk, morbidity and healthcare expenditure. Repetition of self-harm confers yet higher risk of suicide and risk assessment of self-harm patients forms a key component of the health care management of self-harm patients. To date, there has been no systematic review published which synthesises the extensive evidence on risk factors for repetition.Objective
This review is intended to identify risk factors for prospective repetition of self-harm after an index self-harm presentation, irrespective of suicidal intent.Data sources
PubMed, PsychInfo and Scirus were used to search for relevant publications. We included cohort studies which examining factors associated with prospective repetition among those presenting with self-harm to emergency departments. Journal articles, abstracts, letters and theses in any language published up to June 2012 were considered. Studies were quality-assessed and synthesised in narrative form.Results
A total of 129 studies, including 329,001 participants, met our inclusion criteria. Some factors were studied extensively and were found to have a consistent association with repetition. These included previous self-harm, personality disorder, hopelessness, history of psychiatric treatment, schizophrenia, alcohol abuse/dependence, drug abuse/dependence, and living alone. However, the sensitivity values of these measures varied greatly across studies. Psychological risk factors and protective factors have been relatively under-researched but show emerging associations with repetition. Composite risk scales tended to have high sensitivity but poor specificity.Conclusions
Many risk factors for repetition of self-harm match risk factors for initiation of self-harm, but the most consistent evidence for increased risk of repetition comes from long-standing psychosocial vulnerabilities, rather than characteristics of an index episode. The current review will enhance prediction of self-harm and assist in the efficient allocation of intervention resources. 相似文献14.
Elie A. Akl Bradley C. Johnston Pablo Alonso-Coello Ignacio Neumann Shanil Ebrahim Matthias Briel Deborah J. Cook Gordon H. Guyatt 《PloS one》2013,8(2)
Introduction
Systematic reviewer authors intending to include all randomized participants in their meta-analyses need to make assumptions about the outcomes of participants with missing data.Objective
The objective of this paper is to provide systematic reviewer authors with a relatively simple guidance for addressing dichotomous data for participants excluded from analyses of randomized trials.Methods
This guide is based on a review of the Cochrane handbook and published methodological research. The guide deals with participants excluded from the analysis who were considered ‘non-adherent to the protocol’ but for whom data are available, and participants with missing data.Results
Systematic reviewer authors should include data from ‘non-adherent’ participants excluded from the primary study authors'' analysis but for whom data are available. For missing, unavailable participant data, authors may conduct a complete case analysis (excluding those with missing data) as the primary analysis. Alternatively, they may conduct a primary analysis that makes plausible assumptions about the outcomes of participants with missing data. When the primary analysis suggests important benefit, sensitivity meta-analyses using relatively extreme assumptions that may vary in plausibility can inform the extent to which risk of bias impacts the confidence in the results of the primary analysis. The more plausible assumptions draw on the outcome event rates within the trial or in all trials included in the meta-analysis. The proposed guide does not take into account the uncertainty associated with assumed events.Conclusions
This guide proposes methods for handling participants excluded from analyses of randomized trials. These methods can help in establishing the extent to which risk of bias impacts meta-analysis results. 相似文献15.
16.
Thais Gimenez Mariana Minatel Braga Daniela Procida Raggio Chris Deery David N. Ricketts Fausto Medeiros Mendes 《PloS one》2013,8(4)
Background
Fluorescence-based methods have been proposed to aid caries lesion detection. Summarizing and analysing findings of studies about fluorescence-based methods could clarify their real benefits.Objective
We aimed to perform a comprehensive systematic review and meta-analysis to evaluate the accuracy of fluorescence-based methods in detecting caries lesions.Data Source
Two independent reviewers searched PubMed, Embase and Scopus through June 2012 to identify papers/articles published. Other sources were checked to identify non-published literature.Study Eligibility Criteria, Participants and Diagnostic Methods
The eligibility criteria were studies that: (1) have assessed the accuracy of fluorescence-based methods of detecting caries lesions on occlusal, approximal or smooth surfaces, in both primary or permanent human teeth, in the laboratory or clinical setting; (2) have used a reference standard; and (3) have reported sufficient data relating to the sample size and the accuracy of methods.Study Appraisal and Synthesis Methods
A diagnostic 2×2 table was extracted from included studies to calculate the pooled sensitivity, specificity and overall accuracy parameters (Diagnostic Odds Ratio and Summary Receiver-Operating curve). The analyses were performed separately for each method and different characteristics of the studies. The quality of the studies and heterogeneity were also evaluated.Results
Seventy five studies met the inclusion criteria from the 434 articles initially identified. The search of the grey or non-published literature did not identify any further studies. In general, the analysis demonstrated that the fluorescence-based method tend to have similar accuracy for all types of teeth, dental surfaces or settings. There was a trend of better performance of fluorescence methods in detecting more advanced caries lesions. We also observed moderate to high heterogeneity and evidenced publication bias.Conclusions
Fluorescence-based devices have similar overall performance; however, better accuracy in detecting more advanced caries lesions has been observed. 相似文献17.
Background
The Committee for Evidence-based Medicine (EBM) of the Japan Society for Oriental Medicine started compiling Evidence Reports of Kampo Treatment (EKAT) in 2007. EKAT is a compilation of structured abstracts of randomized controlled trials (RCTs), along with comments by a third party reviewer. As of 31 December, 2012, there were 378 RCTs of Kampo medicines in Japan. The primary research question of this study is “How frequently is Kampo diagnosis used in RCTs of Kampo medicines?” The secondary research question is “When is Kampo diagnosis used in RCTs?”Materials and Methods
The structured abstract (SA) of each RCT article was reviewed to examine how Kampo diagnosis was used in RCTs, especially how Kampo diagnosis was used in the randomization process.Results
Kampo diagnosis was used before randomization in 27 RCTs (7.1%), after randomization in 31 RCTs (8.2%), and not used in 320 RCTs (84.7%). Before randomization, Kampo diagnosis was used as a criterion for inclusion in 10 RCTs, criterion for exclusion in 9 RCTs, and criteria for both inclusion and exclusion in 2 RCTs. Kampo formulas were determined according to Kampo diagnosis in 7 RCTs. After randomization, subgroup analyses according to Kampo diagnosis were done in 27 RCTs, and grade of disease severity at Kampo diagnosis was used for analysis as an endpoint in 4 RCTs.Conclusions
Kampo diagnosis was used before randomization only in approximately 15% of RCTs, and the number of RCT articles using Kampo diagnosis after randomization was almost the same as that before randomization. Further studies to determine the good RCTs conforming to CONSORT requirements and good systematic reviews conforming to PRISMA requirements are needed to clarify the significance of Kampo diagnosis. 相似文献18.
Bin Zhang Bernard Combe Nathalie Rincheval David T Felson 《Arthritis research & therapy》2012,14(3):R156-6
Introduction
In development of the American College of Rheumatology (ACR)/European League Against Rheumatism (EULAR) rheumatoid arthritis (RA) remission definitions using clinical trials data, one criterion used to compare different remission definitions was whether, compared with those not in remission, those in remission had evidence of later disease stability defined by x-ray and functional status. Validation of the RA remission criteria using observational study data is necessary before recommending their use in practice.Methods
Using data from those who met RA criteria in the ESPOIR cohort, we matched each person in remission with a person not in remission and then carried out analyses comparing later stability of x-ray and health assessment questionnaire (HAQ) between the two groups. We compared the predictive validity of the same candidate definitions of remission evaluated in the ACR/EULAR process. To minimize potential bias and produce more stable results, we used a bootstrap resampling approach to select those not in remission, repeating the sample matching analysis process 500 times.Results
Results were similar to those of clinical trials analyzed for the ACR/EULAR remission criteria. Specifically, the ACR/EULAR remission definitions using either an simple disease activity index (SDAI) ≤ 3.3, clinical disease activity index (CDAI) ≤ 2.8 or a definition of remission requiring tender joint count, swollen joint count, patient global assessment all ≤ 1 performed as well or better than other candidate definitions of remission in terms of predicting later x-ray and function stability.Conclusions
ACR/EULAR definitions of remission developed for trials are similarly valid in observational studies in RA and could be used in practice. 相似文献19.
Rasha Khatib Jon-David Schwalm Salim Yusuf R. Brian Haynes Martin McKee Maheer Khan Robby Nieuwlaat 《PloS one》2014,9(1)