Formalized synthesis opportunities for ecology: systematic reviews and meta‐analyses

Narrative reviews are dead. Long live systematic reviews (and meta‐analyses). Synthesis in many forms is now a driving force in ecology. Advances in open data for ecology and new tools provide vastly improved capacity for novel, emergent knowledge synthesis in our discipline. Systematic reviews and meta‐analyses are two formal synthesis opportunities for ecologists that are now accepted as traditional publications, but the scope of validated syntheses will continue to expand. To date, systematic reviews are rarely used whilst the rate of meta‐analyses published in ecological journals is increasing exponentially. Systematic reviews provide an overview of the literature landscape for a topic, and meta‐analyses examine the strength of evidence integrated across different studies. Effective synthesis benefits from both approaches, but better data reporting and additional advances in the culture of sharing data, code, analytics, workflows, methods and also ideas will further energize these efforts. At this junction, synthetic efforts that include systematic reviews and meta‐analyses should continue as stand‐alone publications. This is a necessary step in the evolution of synthesis in our discipline. Nonetheless, they are still evolving tools, and meta‐analyses in particular are simply an extended set of statistical tests. Admittedly, understanding the statistics and assumptions influence how we conduct synthesis much as statistical choices often shape experimental design, i.e. ANOVA versus regression‐based experiments, but statistics do not make the paper. Current steps – primary research articles need to more effectively report evidence, sharing scientific products should expand, systematic reviews should be used to identify research gaps/delineate literature landscapes, and meta‐analyses should be used to examine evidence patterns to further predictive ecology.     

Systematic reviews: a primer for plastic surgery research

Clinicians rely on review articles to keep current with the rapid accumulation of medical and surgical literature. Traditional expert reviews, however, often suffer from inherent personal biases and may not reflect a true synthesis of the existing literature on a particular subject. Systematic reviews are structured, scientific articles that address the shortcomings of traditional reviews by adhering to strict, reproducible methods and recommended guidelines. The methods are designed to eliminate possible sources of bias, ensure as complete a review of the existing literature as possible, and present the results in a way that is useful for its intended audience. Systematic reviews may at times include a quantitative synthesis of the available data in the form of a meta-analysis. Meta-analysis is a statistical tool for combining the numerical results of separate studies to obtain a summary outcome with increased precision due to the larger, combined number of patients. Meta-analyses may be particularly helpful when individual study results are conflicting and the existing literature is inconclusive. The validity of meta-analysis, however, is highly dependent on the quality of data available in the literature. In its strictest form, meta-analysis is used to combine data from only randomized controlled clinical trials. Because randomized controlled clinical trials are infrequently performed in plastic surgery research, this article will focus on systematic reviews to provide the readers with a useful guide in performing this field of study.     

In the era of systematic reviews, does the size of an individual trial still matter

BACKGROUND TO THE DEBATE: Systematic reviews that combine high-quality evidence from several trials are now widely considered to be at the top of the hierarchy of clinical evidence. Given the primacy of systematic reviews-and the fact that individual clinical trials rarely provide definitive answers to a clinical research question-some commentators question whether the sample size calculation for an individual trial still matters. Others point out that small trials can still be potentially misleading.     

Does Consideration and Assessment of Effects on Health Equity Affect the Conclusions of Systematic Reviews? A Methodology Study

Introduction

Tackling health inequities both within and between countries remains high on the agenda of international organizations including the World Health Organization and local, regional and national governments. Systematic reviews can be a useful tool to assess effects on equity in health status because they include studies conducted in a variety of settings and populations. This study aims to describe the extent to which the impacts of health interventions on equity in health status are considered in systematic reviews, describe methods used, and assess the implications of their equity related findings for policy, practice and research.

Methods

We conducted a methodology study of equity assessment in systematic reviews. Two independent reviewers extracted information on the reporting and analysis of impacts of health interventions on equity in health status in a group of 300 systematic reviews collected from all systematic reviews indexed in one month of MEDLINE, using a pre-tested data collection form. Any differences in data extraction were resolved by discussion.

Results

Of the 300 systematic reviews, 224 assessed the effectiveness of interventions on health outcomes. Of these 224 reviews, 29 systematic reviews assessed effects on equity in health status using subgroup analysis or targeted analyses of vulnerable populations. Of these, seven conducted subgroup analyses related to health equity which were reported in insufficient detail to judge their credibility. Of these 29 reviews, 18 described implications for policy and practice based on assessment of effects on health equity.

Conclusion

The quality and completeness of reporting should be enhanced as a priority, because without this policymakers and practitioners will continue lack the evidence base they need to inform decision-making about health inequity. Furthermore, there is a need to develop methods to systematically consider impacts on equity in health status that is currently lacking in systematic reviews.     

A guide to interpreting discordant systematic reviews

Systematic reviews are becoming prominent tools to guide health care decisions. As the number of published systematic reviews increases, it is common to find more than 1 systematic review addressing the same or a very similar therapeutic question. Despite the promise for systematic reviews to resolve conflicting results of primary studies, conflicts among reviews are now emerging. Such conflicts produce difficulties for decision-makers (including clinicians, policy-makers, researchers and patients) who rely on these reviews to help them make choices among alternative interventions when experts and the results of trials disagree. The authors provide an adjunct decision tool--a decision algorithm--to help decision-makers select from among discordant reviews.     

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

Systematic reviews and meta-analyses are essential to summarize evidence relating to efficacy and safety of health care interventions accurately and reliably. The clarity and transparency of these reports, however, is not optimal. Poor reporting of systematic reviews diminishes their value to clinicians, policy makers, and other users.Since the development of the QUOROM (QUality Of Reporting Of Meta-analysis) Statement—a reporting guideline published in 1999—there have been several conceptual, methodological, and practical advances regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these issues, an international group that included experienced authors and methodologists developed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) as an evolution of the original QUOROM guideline for systematic reviews and meta-analyses of evaluations of health care interventions.The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Explanation and Elaboration document, we explain the meaning and rationale for each checklist item. For each item, we include an example of good reporting and, where possible, references to relevant empirical studies and methodological literature. The PRISMA Statement, this document, and the associated Web site (http://www.prisma-statement.org/) should be helpful resources to improve reporting of systematic reviews and meta-analyses.     

Dissemination Bias in Systematic Reviews of Animal Research: A Systematic Review

Background

Systematic reviews of preclinical studies, in vivo animal experiments in particular, can influence clinical research and thus even clinical care. Dissemination bias, selective dissemination of positive or significant results, is one of the major threats to validity in systematic reviews also in the realm of animal studies. We conducted a systematic review to determine the number of published systematic reviews of animal studies until present, to investigate their methodological features especially with respect to assessment of dissemination bias, and to investigate the citation of preclinical systematic reviews on clinical research.

Methods

Eligible studies for this systematic review constitute systematic reviews that summarize in vivo animal experiments whose results could be interpreted as applicable to clinical care. We systematically searched Ovid Medline, Embase, ToxNet, and ScienceDirect from 1^st January 2009 to 9^th January 2013 for eligible systematic reviews without language restrictions. Furthermore we included articles from two previous systematic reviews by Peters et al. and Korevaar et al.

Results

The literature search and screening process resulted in 512 included full text articles. We found an increasing number of published preclinical systematic reviews over time. The methodological quality of preclinical systematic reviews was low. The majority of preclinical systematic reviews did not assess methodological quality of the included studies (71%), nor did they assess heterogeneity (81%) or dissemination bias (87%). Statistics quantifying the importance of clinical research citing systematic reviews of animal studies showed that clinical studies referred to the preclinical research mainly to justify their study or a future study (76%).

Discussion

Preclinical systematic reviews may have an influence on clinical research but their methodological quality frequently remains low. Therefore, systematic reviews of animal research should be critically appraised before translating them to a clinical context.     

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews

Background

Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.

Methods/Principal Findings

We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0–19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI −8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.

Conclusion/Significance

Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.     

Maximizing the value of systematic reviews in ecology when data or resources are limited

Systematic reviews provide a rigorous, repeatable and quantitative method for assessing and synthesizing all available empirical evidence to evaluate a specific research, management, or policy question. They are particularly well suited for evaluating the effectiveness of environmental management actions, and thus for underpinning evidence‐based adaptive natural resource management. However, their current utility may be limited in countries like Australia, where both the amount of research relative to land area and of well‐monitored, active land management for environmental purposes are relatively low. Based on our experience conducting two of the first ecological systematic reviews in Australia, we have developed a number of recommendations for conducting systematic reviews in situations where resources and/or primary research data are limited. We discuss potential modification or augmentation of most aspects of the systematic review process including selection of a review team, question formulation, search strategy, data analysis, and the communication of results, as well as the inherent tradeoffs between systematic thoroughness and available resources that are involved in these changes. We hope that our recommendations will encourage more ecologists to undertake systematic reviews even if primary research and resources to conduct the review appear to be limited, as even a modified systematic review can provide more defensible evidence‐based guidelines for management of natural resources.     

Bias,Discrimination, and Obesity

This article reviews information on discriminatory attitudes and behaviors against obese individuals, integrates this to show whether systematic discrimination occurs and why, and discusses needed work in the field. Clear and consistent stigmatization, and in some cases discrimination, can be documented in three important areas of living: employment, education, and health care. Among the findings are that 28% of teachers in one study said that becoming obese is the worst thing that can happen to a person; 24% of nurses said that they are “repulsed” by obese persons; and, controlling for income and grades, parents provide less college support for their overweight than for their thin children. There are also suggestions but not yet documentation of discrimination occurring in adoption proceedings, jury selection, housing, and other areas. Given the vast numbers of people potentially affected, it is important to consider the research‐related, educational, and social policy implications of these findings.     

Using Multiple Types of Studies in Systematic Reviews of Health Care Interventions – A Systematic Review

Background

A systematic review may evaluate different aspects of a health care intervention. To accommodate the evaluation of various research questions, the inclusion of more than one study design may be necessary. One aim of this study is to find and describe articles on methodological issues concerning the incorporation of multiple types of study designs in systematic reviews on health care interventions. Another aim is to evaluate methods studies that have assessed whether reported effects differ by study types.

Methods and Findings

We searched PubMed, the Cochrane Database of Systematic Reviews, and the Cochrane Methodology Register on 31 March 2012 and identified 42 articles that reported on the integration of single or multiple study designs in systematic reviews. We summarized the contents of the articles qualitatively and assessed theoretical and empirical evidence. We found that many examples of reviews incorporating multiple types of studies exist and that every study design can serve a specific purpose. The clinical questions of a systematic review determine the types of design that are necessary or sufficient to provide the best possible answers. In a second independent search, we identified 49 studies, 31 systematic reviews and 18 trials that compared the effect sizes between randomized and nonrandomized controlled trials, which were statistically different in 35%, and not different in 53%. Twelve percent of studies reported both, different and non-different effect sizes.

Conclusions

Different study designs addressing the same question yielded varying results, with differences in about half of all examples. The risk of presenting uncertain results without knowing for sure the direction and magnitude of the effect holds true for both nonrandomized and randomized controlled trials. The integration of multiple study designs in systematic reviews is required if patients should be informed on the many facets of patient relevant issues of health care interventions.     

Research questions and priorities for tuberculosis: a survey of published systematic reviews and meta-analyses

Background

Systematic reviews are increasingly informing policies in tuberculosis (TB) care and control. They may also be a source of questions for future research. As part of the process of developing the International Roadmap for TB Research, we did a systematic review of published systematic reviews on TB, to identify research priorities that are most frequently suggested in reviews.

Methodology/Principal Findings

We searched EMBASE, MEDLINE, Web of Science, and the Cochrane Library for systematic reviews and meta-analyses on any aspect of TB published between 2005 and 2010. One reviewer extracted data and a second reviewer independently extracted data from a random subset of included studies. In total, 137 systematic reviews, with 141 research questions, were included in this review. We used the UK Health Research Classification System (HRCS) to help us classify the research questions and priorities. The three most common research topics were in the area of detection, screening and diagnosis of TB (32.6%), development and evaluation of treatments and therapeutic interventions (23.4%), and TB aetiology and risk factors (19.9%). The research priorities determined were mainly focused on the discovery and evaluation of bacteriological TB tests and drug-resistant TB tests and immunological tests. Other important topics of future research were genetic susceptibility linked to TB and disease determinants attributed to HIV/TB. Evaluation of drug treatments for TB, drug-resistant TB and HIV/TB were also frequently proposed research topics.

Conclusions

Systematic reviews are a good source of key research priorities. Findings from our survey have informed the development of the International Roadmap for TB Research by the TB Research Movement.     

The need for systematic reviews of reasons

There are many ethical decisions in the practice of health research and care, and in the creation of policy and guidelines. We argue that those charged with making such decisions need a new genre of review. The new genre is an application of the systematic review, which was developed over decades to inform medical decision-makers about what the totality of studies that investigate links between smoking and cancer, for example, implies about whether smoking causes cancer. We argue that there is a need for similarly inclusive and rigorous reviews of reason-based bioethics, which uses reasoning to address ethical questions. After presenting a brief history of the systematic review, we reject the only existing model for writing a systematic review of reason-based bioethics, which holds that such a review should address an ethical question. We argue that such a systematic review may mislead decision-makers when a literature is incomplete, or when there are mutually incompatible but individually reasonable answers to the ethical question. Furthermore, such a review can be written without identifying all the reasons given when the ethical questions are discussed, their alleged implications for the ethical question, and the attitudes taken to the reasons. The reviews we propose address instead the empirical question of which reasons have been given when addressing a specified ethical question, and present such detailed information on the reasons. We argue that this information is likely to improve decision-making, both directly and indirectly, and also the academic literature. We explain the limitations of our alternative model for systematic reviews.     

The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review

Background

Systematic reviews are important for decision makers. They offer many potential benefits but are often written in technical language, are too long, and do not contain contextual details which make them hard to use for decision-making. There are many organizations that develop and disseminate derivative products, such as evidence summaries, from systematic reviews for different populations or subsets of decision makers. This systematic review aimed to (1) assess the effectiveness of evidence summaries on policymakers’ use of the evidence and (2) identify the most effective summary components for increasing policymakers’ use of the evidence. We present an overview of the available evidence on systematic review derivative products.

Methods

We included studies of policymakers at all levels as well as health system managers. We included studies examining any type of “evidence summary,” “policy brief,” or other products derived from systematic reviews that presented evidence in a summarized form. The primary outcomes were the (1) use of systematic review summaries in decision-making (e.g., self-reported use of the evidence in policymaking and decision-making) and (2) policymakers’ understanding, knowledge, and/or beliefs (e.g., changes in knowledge scores about the topic included in the summary). We also assessed perceived relevance, credibility, usefulness, understandability, and desirability (e.g., format) of the summaries.

Results

Our database search combined with our gray literature search yielded 10,113 references after removal of duplicates. From these, 54 were reviewed in full text, and we included six studies (reported in seven papers) as well as protocols from two ongoing studies. Two studies assessed the use of evidence summaries in decision-making and found little to no difference in effect. There was also little to no difference in effect for knowledge, understanding or beliefs (four studies), and perceived usefulness or usability (three studies). Summary of findings tables and graded entry summaries were perceived as slightly easier to understand compared to complete systematic reviews. Two studies assessed formatting changes and found that for summary of findings tables, certain elements, such as reporting study event rates and absolute differences, were preferred as well as avoiding the use of footnotes.

Conclusions

Evidence summaries are likely easier to understand than complete systematic reviews. However, their ability to increase the use of systematic review evidence in policymaking is unclear.

Trial registration

The protocol was published in the journal Systematic Reviews (2015;4:122)

     

Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool

BackgroundSystematic reviews of the effects of healthcare interventions frequently include non-randomized studies. These are subject to confounding and a range of other biases that are seldom considered in detail when synthesizing and interpreting the results. Our aims were to assess the reliability and usability of a new Cochrane risk of bias (RoB) tool for non-randomized studies of interventions and to determine whether restricting analysis to studies with low or moderate RoB made a material difference to the results of the reviews.ConclusionsThe Cochrane RoB tool highlighted a wide range of risks of bias in studies included in two widely cited reviews and had the potential to change the conclusions of the reviews. Systematic reviews that incorporate non-randomized studies of medical interventions should include a detailed assessment of RoB for each included study.     

Methodological Quality of Systematic Reviews in Subfertility: A Comparison of Two Different Approaches

Background

Systematic reviews are used widely to guide health care decisions. Several tools have been created to assess systematic review quality. The measurement tool for assessing the methodological quality of systematic reviews known as the AMSTAR tool applies a yes/no score to eleven relevant domains of review methodology. This tool has been reworked so that each domain is scored based on a four point scale, producing R-AMSTAR.

Methods and Findings

We aimed to compare the AMSTAR and R-AMSTAR tools in assessing systematic reviews in the field of assisted reproduction for subfertility. All published systematic reviews on assisted reproductive technology, with the latest search for studies taking place from 2007–2011, were considered. Reviews that contained no included studies or considered diagnostic outcomes were excluded. Thirty each of Cochrane and non-Cochrane reviews were randomly selected from a search of relevant databases. Both tools were then applied to all sixty reviews. The results were converted to percentage scores and all reviews graded and ranked based on this. AMSTAR produced a much wider variation in percentage scores and achieved higher inter-rater reliability than R-AMSTAR according to kappa statistics. The average rating for Cochrane reviews was consistent between the two tools (88.3% for R-AMSTAR versus 83.6% for AMSTAR) but inconsistent for non-Cochrane reviews (63.9% R-AMSTAR vs. 38.5% AMSTAR). In comparing the rankings generated between the two tools Cochrane reviews changed an average of 4.2 places, compared to 2.9 for non-Cochrane.

Conclusion

R-AMSTAR provided greater guidance in the assessment of domains and produced quantitative results. However, there were many problems with the construction of its criteria and AMSTAR was much easier to apply consistently. We recommend that AMSTAR incorporates the findings of this study and produces additional guidance for its application in order to improve its reliability and usefulness.     

Non-Surgical Interventions for Adolescents with Idiopathic Scoliosis: An Overview of Systematic Reviews

Background

Non-surgical interventions for adolescents with idiopathic scoliosis remain highly controversial. Despite the publication of numerous reviews no explicit methodological evaluation of papers labeled as, or having a layout of, a systematic review, addressing this subject matter, is available.

Objectives

Analysis and comparison of the content, methodology, and evidence-base from systematic reviews regarding non-surgical interventions for adolescents with idiopathic scoliosis.

Design

Systematic overview of systematic reviews.

Methods

Articles meeting the minimal criteria for a systematic review, regarding any non-surgical intervention for adolescent idiopathic scoliosis, with any outcomes measured, were included. Multiple general and systematic review specific databases, guideline registries, reference lists and websites of institutions were searched. The AMSTAR tool was used to critically appraise the methodology, and the Oxford Centre for Evidence Based Medicine and the Joanna Briggs Institute’s hierarchies were applied to analyze the levels of evidence from included reviews.

Results

From 469 citations, twenty one papers were included for analysis. Five reviews assessed the effectiveness of scoliosis-specific exercise treatments, four assessed manual therapies, five evaluated bracing, four assessed different combinations of interventions, and one evaluated usual physical activity. Two reviews addressed the adverse effects of bracing. Two papers were high quality Cochrane reviews, Three were of moderate, and the remaining sixteen were of low or very low methodological quality. The level of evidence of these reviews ranged from 1 or 1+ to 4, and in some reviews, due to their low methodological quality and/or poor reporting, this could not be established.

Conclusions

Higher quality reviews indicate that generally there is insufficient evidence to make a judgment on whether non-surgical interventions in adolescent idiopathic scoliosis are effective. Papers labeled as systematic reviews need to be considered in terms of their methodological rigor; otherwise they may be mistakenly regarded as high quality sources of evidence.

Protocol registry number

CRD42013003538, PROSPERO