Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?

Background

The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement.

Objective

We analyze the relationship between peers’ ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields.

Methods and Materials

We analyze peers’ ratings for 2333 applications. We also gathered principal investigators’ research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on the 2002-2006 period. First, we analyzed the distribution of granted and rejected proposals considering a given set of bibliometric indicators to test if there are significant differences. Then, we applied a multiple logistic regression analysis to determine if bibliometric indicators can explain by themselves the concession of grant proposals.

Results

63.4% of the applications were funded. Bibliometric indicators for accepted proposals showed a better previous performance than for those rejected; however the correlation between peer review and bibliometric indicators is very heterogeneous among most areas. The logistic regression analysis showed that the main bibliometric indicators that explain the granting of research proposals in most cases are the output (number of published articles) and the number of papers published in journals that belong to the first quartile ranking of the Journal Citations Report.

Discussion

Bibliometric indicators predict the concession of grant proposals at least as well as peer ratings. Social Sciences and Education are the only areas where no relation was found, although this may be due to the limitations of the Web of Science’s coverage. These findings encourage the use of bibliometric indicators as a complement to peer review in most of the analyzed areas.     

Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research

Background

Citation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research.

Methodology

A visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact.

Results

Visualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields: Cardiac & cardiovascular systems, Clinical neurology, and Surgery. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.

Conclusions

Popular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.     

Demographic Transition in India: An Evolutionary Interpretation of Population and Health Trends Using ‘Change-Point Analysis’

Background and Rationale

Lack of a robust analytical tool for trend analysis of population and health indicators is the basic rationale of this study. In an effort to fill this gap, this study advances ‘Change-Point analyzer’ as a new analytical tool for assessment of the progress and its pattern in population and health indicators.

Methodology/Principal Findings

The defining feature of ‘change-point analyzer’ is that, it detects subtle changes that are often missed in simple trend line plots and also quantified the volume of change that is not possible in simple trend line plots. A long-term assessment of ‘change-point analyses’ of trends in population and health indicators such as IMR, Population size, TFR, and LEB in India show multiple points of critical changes. Measured change points of demographic and health trends helps in understanding the demographic transitional shifts connecting it to contextual policy shifts. Critical change-points in population and health indicators in India are associated with the evolution of structural changes in population and health policy framework.

Conclusions

This study, therefore, adds significantly to the evolutionary interpretation of critical change-points in long-term trajectories of population and health indicators vis-a-vis population and health policy shifts in India. The results have not only helped in reassessing the historical past and the current demographic transition trajectory but also advanced a new method of assessing the population and health trends which are necessary for robust monitoring of the progress in population and health policies.     

Coverage and Quality of Antenatal Care Provided at Primary Health Care Facilities in the ‘Punjab’ Province of ‘Pakistan’

Background

Antenatal care is a very important component of maternal health services. It provides the opportunity to learn about risks associated with pregnancy and guides to plan the place of deliveries thereby preventing maternal and infant morbidity and mortality. In ‘Pakistan’ antenatal services to rural population are being provided through a network of primary health care facilities designated as ''Basic Health Units and Rural Health Centers. Pakistan is a developing country, consisting of four provinces and federally administered areas. Each province is administratively subdivided in to ‘Divisions’ and ‘Districts’. By population ‘Punjab’ is the largest province of Pakistan having 36 districts. This study was conducted to assess the coverage and quality antenatal care in the primary health care facilities in ‘Punjab’ province of ‘Pakistan’.

Methods

Quantitative and Qualitative methods were used to collect data. Using multistage sampling technique nine out of thirty six districts were selected and 19 primary health care facilities of public sector (seventeen Basic Health Units and two Rural Health Centers were randomly selected from each district. Focus group discussions and in-depth interviews were conducted with clients, providers and health managers.

Results

The overall enrollment for antenatal checkup was 55.9% and drop out was 32.9% in subsequent visits. The quality of services regarding assessment, treatment and counseling was extremely poor. The reasons for low coverage and quality were the distant location of facilities, deficiency of facility resources, indifferent attitude and non availability of the staff. Moreover, lack of client awareness about importance of antenatal care and self empowerment for decision making to seek care were also responsible for low coverage.

Conclusion

The coverage and quality of the antenatal care services in ‘Punjab’ are extremely compromised. Only half of the expected pregnancies are enrolled and out of those 1/3 drop out in follow-up visits.     

The validity of peer review in a general medicine journal

All the opinions in this article are those of the authors and should not be construed to reflect, in any way, those of the Department of Veterans Affairs.

Background

Our study purpose was to assess the predictive validity of reviewer quality ratings and editorial decisions in a general medicine journal.

Methods

Submissions to the Journal of General Internal Medicine (JGIM) between July 2004 and June 2005 were included. We abstracted JGIM peer review quality ratings, verified the publication status of all articles and calculated an impact factor for published articles (Rw) by dividing the 3-year citation rate by the average for this group of papers; an Rw>1 indicates a greater than average impact.

Results

Of 507 submissions, 128 (25%) were published in JGIM, 331 rejected (128 with review) and 48 were either not resubmitted after revision was requested or were withdrawn by the author. Of 331 rejections, 243 were published elsewhere. Articles published in JGIM had a higher citation rate than those published elsewhere (Rw: 1.6 vs. 1.1, p = 0.002). Reviewer quality ratings of article quality had good internal consistency and reviewer recommendations markedly influenced publication decisions. There was no quality rating cutpoint that accurately distinguished high from low impact articles. There was a stepwise increase in Rw for articles rejected without review, rejected after review or accepted by JGIM (Rw 0.60 vs. 0.87 vs. 1.56, p<0.0005). However, there was low agreement between reviewers for quality ratings and publication recommendations. The editorial publication decision accurately discriminated high and low impact articles in 68% of submissions. We found evidence of better accuracy with a greater number of reviewers.

Conclusions

The peer review process largely succeeds in selecting high impact articles and dispatching lower impact ones, but the process is far from perfect. While the inter-rater reliability between individual reviewers is low, the accuracy of sorting is improved with a greater number of reviewers.     

Methodological Deficits in Diagnostic Research Using ‘-Omics’ Technologies: Evaluation of the QUADOMICS Tool and Quality of Recently Published Studies

Background

QUADOMICS is an adaptation of QUADAS (a quality assessment tool for use in systematic reviews of diagnostic accuracy studies), which takes into account the particular challenges presented by ‘-omics’ based technologies. Our primary objective was to evaluate the applicability and consistency of QUADOMICS. Subsequently we evaluated and describe the methodological quality of a sample of recently published studies using the tool.

Methodology/Principal Findings

45‘-omics’- based diagnostic studies were identified by systematic search of Pubmed using suitable MeSH terms (“Genomics”, “Sensitivity and specificity”, “Diagnosis”). Three investigators independently assessed the quality of the articles using QUADOMICS and met to compare observations and generate a consensus. Consistency and applicability was assessed by comparing each reviewer''s original rating with the consensus. Methodological quality was described using the consensus rating. Agreement was above 80% for all three reviewers. Four items presented difficulties with application, mostly due to the lack of a clearly defined gold standard. Methodological quality of our sample was poor; studies met roughly half of the applied criteria (mean ± sd, 54.7±18.4%). Few studies were carried out in a population that mirrored the clinical situation in which the test would be used in practice, (6, 13.3%); none described patient recruitment sufficiently; and less than half described clinical and physiological factors that might influence the biomarker profile (20, 44.4%).

Conclusions

The QUADOMICS tool can consistently be applied to diagnostic ‘-omics’ studies presently published in biomedical journals. A substantial proportion of reports in this research field fail to address design issues that are fundamental to make inferences relevant for patient care.     

Patient Reported Outcomes (PROs) in Clinical Trials: Is ‘In-Trial’ Guidance Lacking? A Systematic Review

Background

Patient reported outcomes (PROs) are increasingly assessed in clinical trials, and guidelines are available to inform the design and reporting of such trials. However, researchers involved in PRO data collection report that specific guidance on ‘in-trial’ activity (recruitment, data collection and data inputting) and the management of ‘concerning’ PRO data (i.e., data which raises concern for the well-being of the trial participant) appears to be lacking. The purpose of this review was to determine the extent and nature of published guidelines addressing these areas.

Methods and Findings

Systematic review of 1,362 articles identified 18 eligible papers containing ‘in-trial’ guidelines. Two independent authors undertook a qualitative content analysis of the selected papers. Guidelines presented in each of the articles were coded according to an a priori defined coding frame, which demonstrated reliability (pooled Kappa 0.86–0.97), and validity (<2% residual category coding). The majority of guidelines present were concerned with ‘pre-trial’ activities (72%), for example, outcome measure selection and study design issues, or ‘post-trial’ activities (16%) such as data analysis, reporting and interpretation. ‘In-trial’ guidelines represented 9.2% of all guidance across the papers reviewed, with content primarily focused on compliance, quality control, proxy assessment and reporting of data collection. There were no guidelines surrounding the management of concerning PRO data.

Conclusions

The findings highlight there are minimal in-trial guidelines in publication regarding PRO data collection and management in clinical trials. No guidance appears to exist for researchers involved with the handling of concerning PRO data. Guidelines are needed, which support researchers to manage all PRO data appropriately and which facilitate unbiased data collection.     

Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size

Background

The p value obtained from a significance test provides no information about the magnitude or importance of the underlying phenomenon. Therefore, additional reporting of effect size is often recommended. Effect sizes are theoretically independent from sample size. Yet this may not hold true empirically: non-independence could indicate publication bias.

Methods

We investigate whether effect size is independent from sample size in psychological research. We randomly sampled 1,000 psychological articles from all areas of psychological research. We extracted p values, effect sizes, and sample sizes of all empirical papers, and calculated the correlation between effect size and sample size, and investigated the distribution of p values.

Results

We found a negative correlation of r = −.45 [95% CI: −.53; −.35] between effect size and sample size. In addition, we found an inordinately high number of p values just passing the boundary of significance. Additional data showed that neither implicit nor explicit power analysis could account for this pattern of findings.

Conclusion

The negative correlation between effect size and samples size, and the biased distribution of p values indicate pervasive publication bias in the entire field of psychology.     

Development and inter-rater reliability of the Liverpool adverse drug reaction causality assessment tool

Aim

To develop and test a new adverse drug reaction (ADR) causality assessment tool (CAT).

Methods

A comparison between seven assessors of a new CAT, formulated by an expert focus group, compared with the Naranjo CAT in 80 cases from a prospective observational study and 37 published ADR case reports (819 causality assessments in total).

Main Outcome Measures

Utilisation of causality categories, measure of disagreements, inter-rater reliability (IRR).

Results

The Liverpool ADR CAT, using 40 cases from an observational study, showed causality categories of 1 unlikely, 62 possible, 92 probable and 125 definite (1, 62, 92, 125) and ‘moderate’ IRR (kappa 0.48), compared to Naranjo (0, 100, 172, 8) with ‘moderate’ IRR (kappa 0.45). In a further 40 cases, the Liverpool tool (0, 66, 81, 133) showed ‘good’ IRR (kappa 0.6) while Naranjo (1, 90, 185, 4) remained ‘moderate’.

Conclusion

The Liverpool tool assigns the full range of causality categories and shows good IRR. Further assessment by different investigators in different settings is needed to fully assess the utility of this tool.     

Systematic Differences in Impact across Publication Tracks at PNAS

Background

Citation data can be used to evaluate the editorial policies and procedures of scientific journals. Here we investigate citation counts for the three different publication tracks of the Proceedings of the National Academy of Sciences of the United States of America (PNAS). This analysis explores the consequences of differences in editor and referee selection, while controlling for the prestige of the journal in which the papers appear.

Methodology/Principal Findings

We find that papers authored and “Contributed” by NAS members (Track III) are on average cited less often than papers that are “Communicated” for others by NAS members (Track I) or submitted directly via the standard peer review process (Track II). However, we also find that the variance in the citation count of Contributed papers, and to a lesser extent Communicated papers, is larger than for direct submissions. Therefore when examining the 10% most-cited papers from each track, Contributed papers receive the most citations, followed by Communicated papers, while Direct submissions receive the least citations.

Conclusion/Significance

Our findings suggest that PNAS “Contributed” papers, in which NAS–member authors select their own reviewers, balance an overall lower impact with an increased probability of publishing exceptional papers. This analysis demonstrates that different editorial procedures are associated with different levels of impact, even within the same prominent journal, and raises interesting questions about the most appropriate metrics for judging an editorial policy''s success.     

Quality of Pharmaceutical Care in Surgical Patients

Background

Surgical patients are at risk for preventable adverse drug events (ADEs) during hospitalization. Usually, preventable ADEs are measured as an outcome parameter of quality of pharmaceutical care. However, process measures such as QIs are more efficient to assess the quality of care and provide more information about potential quality improvements.

Objective

To assess the quality of pharmaceutical care of medication-related processes in surgical wards with quality indicators, in order to detect targets for quality improvements.

Methods

For this observational cohort study, quality indicators were composed, validated, tested, and applied on a surgical cohort. Three surgical wards of an academic hospital in the Netherlands (Academic Medical Centre, Amsterdam) participated. Consecutive elective surgical patients with a hospital stay longer than 48 hours were included from April until June 2009. To assess the quality of pharmaceutical care, the set of quality indicators was applied to 252 medical records of surgical patients.

Results

Thirty-four quality indicators were composed and tested on acceptability and content- and face-validity. The selected 28 candidate quality indicators were tested for feasibility and ‘sensitivity to change’. This resulted in a final set of 27 quality indicators, of which inter-rater agreements were calculated (kappa 0.92 for eligibility, 0.74 for pass-rate). The quality of pharmaceutical care was assessed in 252 surgical patients. Nearly half of the surgical patients passed the quality indicators for pharmaceutical care (overall pass rate 49.8%). Improvements should be predominantly targeted to medication care related processes in surgical patients with gastro-intestinal problems (domain pass rate 29.4%).

Conclusions

This quality indicator set can be used to measure quality of pharmaceutical care and detect targets for quality improvements. With these results medication safety in surgical patients can be enhanced.     

Impact of GDP,Spending on R&D,Number of Universities and Scientific Journals on Research Publications among Asian Countries

Objectives

This study aimed to compare the impact of Gross Domestic Product (GDP) per capita, spending on Research and Development (R&D), number of universities, and Indexed Scientific Journals on total number of research documents (papers), citations per document and Hirsch index (H-index) in various science and social science subjects among Asian countries.

Materials and Methods

In this study, 40 Asian countries were included. The information regarding Asian countries, their GDP per capita, spending on R&D, total number of universities and indexed scientific journals were collected. We recorded the bibliometric indicators, including total number of research documents, citations per document and H-index in various science and social sciences subjects during the period 1996–2011. The main sources for information were World Bank, SCI-mago/Scopus and Web of Science; Thomson Reuters.

Results

The mean per capita GDP for all the Asian countries is 14448.31±2854.40 US$, yearly per capita spending on R&D 0.64±0.16 US$, number of universities 72.37±18.32 and mean number of ISI indexed journal per country is 17.97±7.35. The mean of research documents published in various science and social science subjects among all the Asian countries during the period 1996–2011 is 158086.92±69204.09; citations per document 8.67±0.48; and H-index 122.8±19.21. Spending on R&D, number of universities and indexed journals have a positive correlation with number of published documents, citations per document and H-index in various science and social science subjects. However, there was no association between the per capita GDP and research outcomes.

Conclusion

The Asian countries who spend more on R&D have a large number of universities and scientific indexed journals produced more in research outcomes including total number of research publication, citations per documents and H-index in various science and social science subjects.     

Graduate entry medicine: selection criteria and student performance

Background

Graduate entry medicine raises new questions about the suitability of students with different backgrounds. We examine this, and the broader issue of effectiveness of selection and assessment procedures.

Methods

The data included background characteristics, academic record, interview score and performance in pre-clinical modular assessment for two years intake of graduate entry medical students. Exploratory factor analysis is a powerful method for reducing a large number of measures to a smaller group of underlying factors. It was used here to identify patterns within and between the selection and performance data.

Principal Findings

Basic background characteristics were of little importance in predicting exam success. However, easily interpreted components were detected within variables comprising the ‘selection’ and ‘assessment’ criteria. Three selection components were identified (‘Academic’, ‘GAMSAT’, ‘Interview’) and four assessment components (‘General Exam’, ‘Oncology’, ‘OSCE’, ‘Family Case Study’). There was a striking lack of relationships between most selection and performance factors. Only ‘General Exam’ and ‘Academic’ showed a correlation (Pearson''s r = 0.55, p<0.001).

Conclusions

This study raises questions about methods of student selection and their effectiveness in predicting performance and assessing suitability for a medical career. Admissions tests and most exams only confirmed previous academic achievement, while interview scores were not correlated with any consequent assessment.     

Using Quality Measures for Quality Improvement: The Perspective of Hospital Staff

Research objective

This study examines the perspectives of a range of key hospital staff on the use, importance, scientific background, availability of data, feasibility of data collection, cost benefit aspects and availability of professional personnel for measurement of quality indicators among Iranian hospitals. The study aims to facilitate the use of quality indicators to improve quality of care in hospitals.

Study design

A cross-sectional study was conducted over the period 2009 to 2010. Staff at Iranian hospitals completed a self-administered questionnaire eliciting their views on organizational, clinical process, and outcome (clinical effectiveness, patient safety and patient centeredness) indicators.

Population studied

93 hospital frontline staff including hospital/nursing managers, medical doctors, nurses, and quality improvement/medical records officers in 48 general and specialized hospitals in Iran.

Principal findings

On average, only 69% of respondents reported using quality indicators in practice at their affiliated hospitals. Respondents varied significantly in their reported use of organizational, clinical process and outcome quality indicators. Overall, clinical process and effectiveness indicators were reported to be least used. The reported use of indicators corresponded with their perceived level of importance. Quality indicators were reported to be used among clinical staff significantly more than among managerial staff. In total, 74% of the respondents reported to use obligatory indicators, while this was 68% for voluntary indicators (p<0.05).

Conclusions

There is a general awareness of the importance and usability of quality indicators among hospital staff in Iran, but their use is currently mostly directed towards external accountability purposes. To increase the formative use of quality indicators, creation of a common culture and feeling of shared ownership, alongside an increased uptake of clinical process and effectiveness indicators is needed to support internal quality improvement processes at hospital level.     

Chinese Proprietary Herbal Medicine Listed in ‘China National Essential Drug List’ for Common Cold: A Systematic Literature Review

Objective

Chinese proprietary herbal medicines (CPHMs) have long history in China for the treatment of common cold, and lots of them have been listed in the ‘China national essential drug list’ by the Chinese Ministry of Health. The aim of this review is to provide a well-round clinical evidence assessment on the potential benefits and harms of CPHMs for common cold based on a systematic literature search to justify their clinical use and recommendation.

Methods

We searched CENTRAL, MEDLINE, EMBASE, SinoMed, CNKI, VIP, China Important Conference Papers Database, China Dissertation Database, and online clinical trial registry websites from their inception to 31 March 2013 for clinical studies of CPHMs listed in the ‘China national essential drug list’ for common cold. There was no restriction on study design.

Results

A total of 33 CPHMs were listed in ‘China national essential drug list 2012’ for the treatment of common cold but only 7 had supportive clinical evidences. A total of 6 randomised controlled trials (RCTs) and 7 case series (CSs) were included; no other study design was identified. All studies were conducted in China and published in Chinese between 1995 and 2012. All included studies had poor study design and methodological quality, and were graded as very low quality.

Conclusions

The use of CPHMs for common cold is not supported by robust evidence. Further rigorous well designed placebo-controlled, randomized trials are needed to substantiate the clinical claims made for CPHMs.     

Beyond a Dichotomous View of the Concepts of ‘Sex’ and ‘Gender’ Focus Group Discussions among Gender Researchers at a Medical Faculty

Introduction

The concepts of ‘sex’ and ‘gender’ are both of vital importance in medicine and health sciences. However, the meaning of these concepts has seldom been discussed in the medical literature. The aim of this study was to explore what the concepts of ‘sex’ and ‘gender’ meant for gender researchers based in a medical faculty.

Methods

Sixteen researchers took part in focus group discussions. The analysis was performed in several steps. The participating researchers read the text and discussed ideas for analysis in national and international workshops. The data were analysed using qualitative content analysis. The authors performed independent preliminary analyses, which were further developed and intensively discussed between the authors.

Results

The analysis of meanings of the concepts of ‘sex’ and ‘gender’ for gender researchers based in a medical faculty resulted in three categories; “Sex as more than biology”, with the subcategories ‘sex’ is not simply biological, ‘sex’ as classification, and ‘sex’ as fluid and changeable; ”Gender as a multiplicity of power-related constructions”, with the subcategories: ‘gender’ as constructions, ‘gender’ power dimensions, and ‘gender’ as doing femininities and masculinities; “Sex and gender as interwoven”, with the subcategories: ‘sex’ and ‘gender’ as inseparable and embodying ‘sex’ and ‘gender’.

Conclusions

Gender researchers within medicine pointed out the importance of looking beyond a dichotomous view of the concepts of ‘sex’ and ‘gender’. The perception of the concepts was that ‘sex’ and ‘gender’ were intertwined. Further research is needed to explore how ‘sex’ and ‘gender’ interact.     

Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices

Background

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

Methods and Findings

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

Conclusions

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.     

A Scoping Review of Strategies for the Prevention of Hip Fracture in Elderly Nursing Home Residents

Background

Elderly nursing home residents are at increased risk of hip fracture; however, the efficacy of fracture prevention strategies in this population is unclear.

Objective

We performed a scoping review of randomized controlled trials of interventions tested in the long-term care (LTC) setting, examining hip fracture outcomes.

Methods

We searched for citations in 6 respective electronic searches, supplemented by hand searches. Two reviewers independently reviewed all citations and full-text papers; consensus was achieved on final inclusion. Data was abstracted in duplicate.

Findings

We reviewed 22,349 abstracts or citations and 949 full-text papers. Data from 20 trials were included: 7 - vitamin D (n = 12,875 participants), 2 - sunlight exposure (n = 522), 1 - alendronate (n = 327), 1 - fluoride (n = 460), 4 – exercise or multimodal interventions (n = 8,165), and 5 - hip protectors (n = 2,594). Vitamin D, particularly vitamin D₃ ≥800 IU orally daily, reduced hip fracture risk. Hip protectors reduced hip fractures in included studies, although a recent large study not meeting inclusion criteria was negative. Fluoride and sunlight exposure did not significantly reduce hip fractures. Falls were reduced in three studies of exercise or multimodal interventions, with one study suggesting reduced hip fractures in a secondary analysis. A staff education and risk assessment strategy did not significantly reduce falls or hip fractures. In a study underpowered for fracture outcomes, alendronate did not significantly reduce hip fractures in LTC.

Conclusions

The intervention with the strongest evidence for reduction of hip fractures in LTC is Vitamin D supplementation; more research on other interventions is needed.     

Non-Surgical Interventions for Adolescents with Idiopathic Scoliosis: An Overview of Systematic Reviews

Background

Non-surgical interventions for adolescents with idiopathic scoliosis remain highly controversial. Despite the publication of numerous reviews no explicit methodological evaluation of papers labeled as, or having a layout of, a systematic review, addressing this subject matter, is available.

Objectives

Analysis and comparison of the content, methodology, and evidence-base from systematic reviews regarding non-surgical interventions for adolescents with idiopathic scoliosis.

Design

Systematic overview of systematic reviews.

Methods

Articles meeting the minimal criteria for a systematic review, regarding any non-surgical intervention for adolescent idiopathic scoliosis, with any outcomes measured, were included. Multiple general and systematic review specific databases, guideline registries, reference lists and websites of institutions were searched. The AMSTAR tool was used to critically appraise the methodology, and the Oxford Centre for Evidence Based Medicine and the Joanna Briggs Institute’s hierarchies were applied to analyze the levels of evidence from included reviews.

Results

From 469 citations, twenty one papers were included for analysis. Five reviews assessed the effectiveness of scoliosis-specific exercise treatments, four assessed manual therapies, five evaluated bracing, four assessed different combinations of interventions, and one evaluated usual physical activity. Two reviews addressed the adverse effects of bracing. Two papers were high quality Cochrane reviews, Three were of moderate, and the remaining sixteen were of low or very low methodological quality. The level of evidence of these reviews ranged from 1 or 1+ to 4, and in some reviews, due to their low methodological quality and/or poor reporting, this could not be established.

Conclusions

Higher quality reviews indicate that generally there is insufficient evidence to make a judgment on whether non-surgical interventions in adolescent idiopathic scoliosis are effective. Papers labeled as systematic reviews need to be considered in terms of their methodological rigor; otherwise they may be mistakenly regarded as high quality sources of evidence.

Protocol registry number

CRD42013003538, PROSPERO     

Comparison of Alternative Evidence Summary and Presentation Formats in Clinical Guideline Development: A Mixed-Method Study

Background

Best formats for summarising and presenting evidence for use in clinical guideline development remain less well defined. We aimed to assess the effectiveness of different evidence summary formats to address this gap.

Methods

Healthcare professionals attending a one-week Kenyan, national guideline development workshop were randomly allocated to receive evidence packaged in three different formats: systematic reviews (SRs) alone, systematic reviews with summary-of-findings tables, and ‘graded-entry’ formats (a ‘front-end’ summary and a contextually framed narrative report plus the SR). The influence of format on the proportion of correct responses to key clinical questions, the primary outcome, was assessed using a written test. The secondary outcome was a composite endpoint, measured on a 5-point scale, of the clarity of presentation and ease of locating the quality of evidence for critical neonatal outcomes. Interviews conducted within two months following completion of trial data collection explored panel members’ views on the evidence summary formats and experiences with appraisal and use of research information.

Results

65 (93%) of 70 participants completed questions on the prespecified outcome measures. There were no differences between groups in the odds of correct responses to key clinical questions. ‘Graded-entry’ formats were associated with a higher mean composite score for clarity and accessibility of information about the quality of evidence for critical neonatal outcomes compared to systematic reviews alone (adjusted mean difference 0.52, 95% CI 0.06 to 0.99). There was no difference in the mean composite score between SR with SoF tables and SR alone. Findings from interviews with 16 panelists indicated that short narrative evidence reports were preferred for the improved clarity of information presentation and ease of use.

Conclusions

Our findings suggest that ‘graded-entry’ evidence summary formats may improve clarity and accessibility of research evidence in clinical guideline development.

Trial Registration

Controlled-Trials.com ISRCTN05154264