Establishing a minimum dataset for prospective registration of systematic reviews: an international consultation

Background

In response to growing recognition of the value of prospective registration of systematic review protocols, we planned to develop a web-based open access international register. In order for the register to fulfil its aims of reducing unplanned duplication, reducing publication bias, and providing greater transparency, it was important to ensure the appropriate data were collected. We therefore undertook a consultation process with experts in the field to identify a minimum dataset for registration.

Methods and Findings

A two-round electronic modified Delphi survey design was used. The international panel surveyed included experts from areas relevant to systematic review including commissioners, clinical and academic researchers, methodologists, statisticians, information specialists, journal editors and users of systematic reviews. Direct invitations to participate were sent out to 315 people in the first round and 322 in the second round. Responses to an open invitation to participate were collected separately. There were 194 (143 invited and 51 open) respondents with a 100% completion rate in the first round and 209 (169 invited and 40 open) respondents with a 91% completion rate in the second round. In the second round, 113 (54%) of the participants reported having previously taken part in the first round. Participants were asked to indicate whether a series of potential items should be designated as optional or required registration items, or should not be included in the register. After the second round, a 70% or greater agreement was reached on the designation of 30 of 36 items.

Conclusions

The results of the Delphi exercise have established a dataset of 22 required items for the prospective registration of systematic reviews, and 18 optional items. The dataset captures the key attributes of review design as well as the administrative details necessary for registration.     

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews

Background

Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.

Methods/Principal Findings

We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0–19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI −8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.

Conclusion/Significance

Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.     

Epidemiology and reporting characteristics of preclinical systematic reviews

In an effort to better utilize published evidence obtained from animal experiments, systematic reviews of preclinical studies are increasingly more common—along with the methods and tools to appraise them (e.g., SYstematic Review Center for Laboratory animal Experimentation [SYRCLE’s] risk of bias tool). We performed a cross-sectional study of a sample of recent preclinical systematic reviews (2015–2018) and examined a range of epidemiological characteristics and used a 46-item checklist to assess reporting details. We identified 442 reviews published across 43 countries in 23 different disease domains that used 26 animal species. Reporting of key details to ensure transparency and reproducibility was inconsistent across reviews and within article sections. Items were most completely reported in the title, introduction, and results sections of the reviews, while least reported in the methods and discussion sections. Less than half of reviews reported that a risk of bias assessment for internal and external validity was undertaken, and none reported methods for evaluating construct validity. Our results demonstrate that a considerable number of preclinical systematic reviews investigating diverse topics have been conducted; however, their quality of reporting is inconsistent. Our study provides the justification and evidence to inform the development of guidelines for conducting and reporting preclinical systematic reviews.

A cross sectional study of a sample of recent preclinical systematic reviews reveals deficiencies in reporting and provides the justification and evidence to inform the development of specific guidelines for conducting and reporting preclinical systematic reviews.     

Cochrane systematic reviews of Chinese herbal medicines: an overview

Objectives

Our study had two objectives: a) to systematically identify all existing systematic reviews of Chinese herbal medicines (CHM) published in Cochrane Library; b) to assess the methodological quality of included reviews.

Methodology/Principal Findings

We performed a systematic search of the Cochrane Database of Systematic Reviews (CDSR, Issue 5, 2010) to identify all reviews of CHM. A total of fifty-eight reviews were eligible for our study. Twenty-one of the included reviews had at least one Traditional Chinese Medicine (TCM) practitioner as its co-author. 7 reviews didn''t include any primary study, the remaining reviews (n = 51) included a median of 9 studies and 936 participants. 50% of reviews were last assessed as up-to-date prior to 2008. The questions addressed by 39 reviews were broad in scope, in which 9 reviews combined studies with different herbal medicines. For OQAQ, the mean of overall quality score (item 10) was 5.05 (95% CI; 4.58-5.52). All reviews assessed the methodological quality of primary studies, 16% of included primary studies used adequate sequence generation and 7% used adequate allocation concealment. Of the 51 nonempty reviews, 23 reviews were reported as being inconclusive, while 27 concluded that there might be benefit of CHM, which was limited by the poor quality or inadequate quantity of included studies. 58 reviews reported searching a median of seven electronic databases, while 10 reviews did not search any Chinese database.

Conclusions

Now CDSR has included large numbers of CHM reviews, our study identified some areas which could be improved, such as almost half of included reviews did not have the participation of TCM practitioners and were not up-to-date according to Cochrane criteria, some reviews pooled the results of different herbal medicines and ignored the searching of Chinese databases.     

Completeness of chronic disease registration in general practice.

The completeness of chronic disease registration in four general practices was assessed by referring to the Hospital Activity Analysis. Overall, just over half (56%) of the 695 patients who had been discharged from hospital with a diagnosis of diabetes, cancer, myocardial infarction, epilepsy, hypertension, or thyroid disease were identified on the practice disease registers. Patients with diabetes were most likely to be identified (72%), and those who had had a myocardial infarct least likely (43%). If the standard of registration is to be improved general practitioners must be convinced of its value. The Hospital Activity Analysis might be used widely to audit and improve practice registers.     

The need for systematic reviews of reasons

There are many ethical decisions in the practice of health research and care, and in the creation of policy and guidelines. We argue that those charged with making such decisions need a new genre of review. The new genre is an application of the systematic review, which was developed over decades to inform medical decision-makers about what the totality of studies that investigate links between smoking and cancer, for example, implies about whether smoking causes cancer. We argue that there is a need for similarly inclusive and rigorous reviews of reason-based bioethics, which uses reasoning to address ethical questions. After presenting a brief history of the systematic review, we reject the only existing model for writing a systematic review of reason-based bioethics, which holds that such a review should address an ethical question. We argue that such a systematic review may mislead decision-makers when a literature is incomplete, or when there are mutually incompatible but individually reasonable answers to the ethical question. Furthermore, such a review can be written without identifying all the reasons given when the ethical questions are discussed, their alleged implications for the ethical question, and the attitudes taken to the reasons. The reviews we propose address instead the empirical question of which reasons have been given when addressing a specified ethical question, and present such detailed information on the reasons. We argue that this information is likely to improve decision-making, both directly and indirectly, and also the academic literature. We explain the limitations of our alternative model for systematic reviews.     

Guidelines for performing systematic reviews in sports science

Most of the reviews carried out in sports science have used the general items suggested by Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA). Due to the specific requirements of each knowledge area, several modifications of the PRISMA are necessary to optimize the process of the systematic reviews and, in consequence, the quality of the conclusions provided in this type of study. Therefore, this work aimed to adapt PRISMA to provide specific guidelines to carry out systematic reviews in sports science. The methodology criteria (search strategy, databases, and eligibility) and the results section (flow diagrams and study contents) were adapted based on previous studies, and several new considerations were added to design the new guidelines. We compiled 28 items suggested by sports science researchers and included two new items: (i) population/problem (i.e., age, level, and country) and (ii) the entire training process, which is monitored and compared between groups (e.g., total training load). To maximize the benefit of this document, we encourage people to read it in conjunction with the PRISMA statement. The main differences between PRISMA and the PRISMA adapted to sports science were related to registration, search strategy, flow diagrams, and results. Application of the new guidelines could improve the information provided to readers and make it easier to generalize and compare the results in sports science.