Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool

BackgroundSystematic reviews of the effects of healthcare interventions frequently include non-randomized studies. These are subject to confounding and a range of other biases that are seldom considered in detail when synthesizing and interpreting the results. Our aims were to assess the reliability and usability of a new Cochrane risk of bias (RoB) tool for non-randomized studies of interventions and to determine whether restricting analysis to studies with low or moderate RoB made a material difference to the results of the reviews.ConclusionsThe Cochrane RoB tool highlighted a wide range of risks of bias in studies included in two widely cited reviews and had the potential to change the conclusions of the reviews. Systematic reviews that incorporate non-randomized studies of medical interventions should include a detailed assessment of RoB for each included study.     

Does Consideration and Assessment of Effects on Health Equity Affect the Conclusions of Systematic Reviews? A Methodology Study

Introduction

Tackling health inequities both within and between countries remains high on the agenda of international organizations including the World Health Organization and local, regional and national governments. Systematic reviews can be a useful tool to assess effects on equity in health status because they include studies conducted in a variety of settings and populations. This study aims to describe the extent to which the impacts of health interventions on equity in health status are considered in systematic reviews, describe methods used, and assess the implications of their equity related findings for policy, practice and research.

Methods

We conducted a methodology study of equity assessment in systematic reviews. Two independent reviewers extracted information on the reporting and analysis of impacts of health interventions on equity in health status in a group of 300 systematic reviews collected from all systematic reviews indexed in one month of MEDLINE, using a pre-tested data collection form. Any differences in data extraction were resolved by discussion.

Results

Of the 300 systematic reviews, 224 assessed the effectiveness of interventions on health outcomes. Of these 224 reviews, 29 systematic reviews assessed effects on equity in health status using subgroup analysis or targeted analyses of vulnerable populations. Of these, seven conducted subgroup analyses related to health equity which were reported in insufficient detail to judge their credibility. Of these 29 reviews, 18 described implications for policy and practice based on assessment of effects on health equity.

Conclusion

The quality and completeness of reporting should be enhanced as a priority, because without this policymakers and practitioners will continue lack the evidence base they need to inform decision-making about health inequity. Furthermore, there is a need to develop methods to systematically consider impacts on equity in health status that is currently lacking in systematic reviews.     

The Contribution of Systematic Review and Meta-Analysis Methods to Human Health Risk Assessment: Neurobehavioral Effects of Manganese

Current methods of human health risk assessment may lack transparency in respect of identification, review, and synthesis of potentially relevant human and animal evidence. The nature, degree, and source of uncertainties are often unclear. This article aims to demonstrate the contribution that systematic review and meta-analysis methods can make to providing more structured, transparent, and systematic risk assessments. We focus on disparities between five risk assessments for neurobehavioral effects of manganese, and then illustrate advantages of a systematic approach. Fifty-five human epidemiological studies and 37 animal experiments were identified. Where appropriate, meta-analysis methods demonstrated consistent adverse effects associated with manganese exposure across species. In particular, there was reduced activity in subjects exposed to manganese, although exposed rats tended to be more active than controls. Limitations of exposure measurement and reporting restricted use of more quantitative methods of evidence synthesis. From a methodological viewpoint, we conclude that systematic review and meta-analysis methods can contribute to a more systematic and transparent human health risk assessment making more efficient use of available evidence, compared to current methods of risk assessment. More complex methods could encompass further differences between relevant studies and so further improve the risk assessment process.     

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

Systematic reviews and meta-analyses are essential to summarize evidence relating to efficacy and safety of health care interventions accurately and reliably. The clarity and transparency of these reports, however, is not optimal. Poor reporting of systematic reviews diminishes their value to clinicians, policy makers, and other users.Since the development of the QUOROM (QUality Of Reporting Of Meta-analysis) Statement—a reporting guideline published in 1999—there have been several conceptual, methodological, and practical advances regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these issues, an international group that included experienced authors and methodologists developed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) as an evolution of the original QUOROM guideline for systematic reviews and meta-analyses of evaluations of health care interventions.The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Explanation and Elaboration document, we explain the meaning and rationale for each checklist item. For each item, we include an example of good reporting and, where possible, references to relevant empirical studies and methodological literature. The PRISMA Statement, this document, and the associated Web site (http://www.prisma-statement.org/) should be helpful resources to improve reporting of systematic reviews and meta-analyses.     

Research questions and priorities for tuberculosis: a survey of published systematic reviews and meta-analyses

Background

Systematic reviews are increasingly informing policies in tuberculosis (TB) care and control. They may also be a source of questions for future research. As part of the process of developing the International Roadmap for TB Research, we did a systematic review of published systematic reviews on TB, to identify research priorities that are most frequently suggested in reviews.

Methodology/Principal Findings

We searched EMBASE, MEDLINE, Web of Science, and the Cochrane Library for systematic reviews and meta-analyses on any aspect of TB published between 2005 and 2010. One reviewer extracted data and a second reviewer independently extracted data from a random subset of included studies. In total, 137 systematic reviews, with 141 research questions, were included in this review. We used the UK Health Research Classification System (HRCS) to help us classify the research questions and priorities. The three most common research topics were in the area of detection, screening and diagnosis of TB (32.6%), development and evaluation of treatments and therapeutic interventions (23.4%), and TB aetiology and risk factors (19.9%). The research priorities determined were mainly focused on the discovery and evaluation of bacteriological TB tests and drug-resistant TB tests and immunological tests. Other important topics of future research were genetic susceptibility linked to TB and disease determinants attributed to HIV/TB. Evaluation of drug treatments for TB, drug-resistant TB and HIV/TB were also frequently proposed research topics.

Conclusions

Systematic reviews are a good source of key research priorities. Findings from our survey have informed the development of the International Roadmap for TB Research by the TB Research Movement.     

Qualichem In Vivo: A Tool for Assessing the Quality of In Vivo Studies and Its Application for Bisphenol A

In regulatory toxicology, quality assessment of in vivo studies is a critical step for assessing chemical risks. It is crucial for preserving public health studies that are considered suitable for regulating chemicals are robust. Current procedures for conducting quality assessments in safety agencies are not structured, clear or consistent. This leaves room for criticism about lack of transparency, subjective influence and the potential for insufficient protection provided by resulting safety standards. We propose a tool called “Qualichem in vivo” that is designed to systematically and transparently assess the quality of in vivo studies used in chemical health risk assessment. We demonstrate its use here with 12 experts, using two controversial studies on Bisphenol A (BPA) that played an important role in BPA regulation in Europe. The results obtained with Qualichem contradict the quality assessments conducted by expert committees in safety agencies for both of these studies. Furthermore, they show that reliance on standardized guidelines to ensure scientific quality is only partially justified. Qualichem allows experts with different disciplinary backgrounds and professional experiences to express their individual and sometimes divergent views—an improvement over the current way of dealing with minority opinions. It provides a transparent framework for expressing an aggregated, multi-expert level of confidence in a study, and allows a simple graphical representation of how well the study integrates the best available scientific knowledge. Qualichem can be used to compare assessments of the same study by different health agencies, increasing transparency and trust in the work of expert committees. In addition, it may be used in systematic evaluation of in vivo studies submitted by industry in the dossiers that are required for compliance with the REACH Regulation. Qualichem provides a balanced, common framework for assessing the quality of studies that may or may not be following standardized guidelines.     

External validation of a measurement tool to assess systematic reviews (AMSTAR)

Background

Thousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews.

Methodology

AMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently.

Conclusions

The sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearson''s R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearson''s R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were −0.19±1.38. This translates to a minimum detectable difference between reviews of 0.64 ‘AMSTAR points’. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews.     

Text mining for literature review and knowledge discovery in cancer risk assessment and research

Research in biomedical text mining is starting to produce technology which can make information in biomedical literature more accessible for bio-scientists. One of the current challenges is to integrate and refine this technology to support real-life scientific tasks in biomedicine, and to evaluate its usefulness in the context of such tasks. We describe CRAB - a fully integrated text mining tool designed to support chemical health risk assessment. This task is complex and time-consuming, requiring a thorough review of existing scientific data on a particular chemical. Covering human, animal, cellular and other mechanistic data from various fields of biomedicine, this is highly varied and therefore difficult to harvest from literature databases via manual means. Our tool automates the process by extracting relevant scientific data in published literature and classifying it according to multiple qualitative dimensions. Developed in close collaboration with risk assessors, the tool allows navigating the classified dataset in various ways and sharing the data with other users. We present a direct and user-based evaluation which shows that the technology integrated in the tool is highly accurate, and report a number of case studies which demonstrate how the tool can be used to support scientific discovery in cancer risk assessment and research. Our work demonstrates the usefulness of a text mining pipeline in facilitating complex research tasks in biomedicine. We discuss further development and application of our technology to other types of chemical risk assessment in the future.     

Risk assessment strategies for transgenic plants

Advances in recombinant DNA technology have created advantages for the development of plants with high agro-economical values. Since the production of transgenic plants, some issues concerning the safe use of these plants and their products have been under debate throughout the world. In this respect, the potential risks and benefits of transgenic plants need to be evaluated objectively. Risk assessment of transgenic crops is a basic prerequisite for monitoring the possible risks that could arise upon the release and use of transgenic plants. To get a meaningful tool for decision making, risk assessment needs to be carried out in a scientific sound and transparent manner. There are specific governmental regulations in many countries for the safety assessment of genetically modified (GM) crops. Furthermore, there are some international agreements, which regulate the cultivation and commercialization of transgenic plants and their derivatives. Internationally accepted risk assessment strategies have been performed to evaluate the safe use of a large variety of GM crops. The main objectives of these regulations and risk assessment strategies are focused to protect human/animal health and the environment.     

Adjustment of systematic microarray data biases

MOTIVATION: Systematic differences due to experimental features of microarray experiments are present in most large microarray data sets. Many different experimental features can cause biases including different sources of RNA, different production lots of microarrays or different microarray platforms. These systematic effects present a substantial hurdle to the analysis of microarray data. RESULTS: We present here a new method for the identification and adjustment of systematic biases that are present within microarray data sets. Our approach is based on modern statistical discrimination methods and is shown to be very effective in removing systematic biases present in a previously published breast tumor cDNA microarray data set. The new method of 'Distance Weighted Discrimination (DWD)' is shown to be better than Support Vector Machines and Singular Value Decomposition for the adjustment of systematic microarray effects. In addition, it is shown to be of general use as a tool for the discrimination of systematic problems present in microarray data sets, including the merging of two breast tumor data sets completed on different microarray platforms. AVAILABILITY: Matlab software to perform DWD can be retrieved from https://genome.unc.edu/pubsup/dwd/     

Issues Related to Chemical Analysis,Data Reporting,and Use: Implications for Human Health Risk Assessment of PCBs and PBDEs in Fish Tissue

Recent reports in the scientific literature and the media, related to elevated levels of polychlorinated biphenyls (PCBs) and polybrominated diethyl ethers (PBDEs) in farmed and wild salmon have had significant impacts on public opinion and consumer behavior, influencing the sales of farmed salmon in North America and Europe. The assessment of contaminants in fatty fish, an important source of omega-3 fatty acids, is therefore an exercise in balancing risks and benefits. Human health risk assessors and risk managers will benefit from an understanding of the level of uncertainty that is integrated into all aspects of evaluating risk in this context. Significant variability exists in the way in which analyses are conducted, how data are reported, and how they are used in risk assessments. We conducted an analytical review of PCB and PBDE data in farmed and wild salmon, and identified critical issues having implications on human health risk assessment from fish consumption. These issues include: analytical methodologies used, quantification issues, reporting of QA/QC information, tissue sampling, nature of tissue analyzed, and laboratory competence. This article reviews and outlines these issues, discusses their implications for human health risk assessment, and recommends the consistent application of analytical fish tissue data in human health risk assessment.     

Developing Univariate Distributions from Data for Risk Analysis

The importance of fitting distributions to data for risk analysis continues to grow as regulatory agencies, like the Environmental Protection Agency (EPA), continue to shift from deterministic to probabilistic risk assessment techniques. The use of Monte Carlo simulation as a tool for propagating variability and uncertainty in risk requires specification of the risk model's inputs in the form of distributions or tables of data. Several software tools exist to support risk assessors in their efforts to develop distributions. However, users must keep in mind that these tools do not replace clear thought about judgments that must be made in characterizing the information from data. This overview introduces risk assessors to the statistical concepts and physical reasons that support important judgments about appropriate types of parametric distributions and goodness-of-fit. In the context of using data to improve risk assessment and ultimately risk management, this paper discusses issues related to the nature of the data (representativeness, quantity, and quality, correlation with space and time, and distinguishing between variability and uncertainty for a set of data), and matching data and distributions appropriately. All data analysis (whether “Frequentist” or “Bayesian” or oblivious to the distinction) requires the use of subjective judgment. The paper offers an iterative process for developing distributions using data to characterize variability and uncertainty for inputs to risk models that provides incentives for collecting better information when the value of information exceeds its cost. Risk analysts need to focus attention on characterizing the information appropriately for purposes of the risk assessment (and risk management questions at hand), not on characterization for its own sake.     

Methodological Quality of Systematic Reviews in Subfertility: A Comparison of Two Different Approaches

Background

Systematic reviews are used widely to guide health care decisions. Several tools have been created to assess systematic review quality. The measurement tool for assessing the methodological quality of systematic reviews known as the AMSTAR tool applies a yes/no score to eleven relevant domains of review methodology. This tool has been reworked so that each domain is scored based on a four point scale, producing R-AMSTAR.

Methods and Findings

We aimed to compare the AMSTAR and R-AMSTAR tools in assessing systematic reviews in the field of assisted reproduction for subfertility. All published systematic reviews on assisted reproductive technology, with the latest search for studies taking place from 2007–2011, were considered. Reviews that contained no included studies or considered diagnostic outcomes were excluded. Thirty each of Cochrane and non-Cochrane reviews were randomly selected from a search of relevant databases. Both tools were then applied to all sixty reviews. The results were converted to percentage scores and all reviews graded and ranked based on this. AMSTAR produced a much wider variation in percentage scores and achieved higher inter-rater reliability than R-AMSTAR according to kappa statistics. The average rating for Cochrane reviews was consistent between the two tools (88.3% for R-AMSTAR versus 83.6% for AMSTAR) but inconsistent for non-Cochrane reviews (63.9% R-AMSTAR vs. 38.5% AMSTAR). In comparing the rankings generated between the two tools Cochrane reviews changed an average of 4.2 places, compared to 2.9 for non-Cochrane.

Conclusion

R-AMSTAR provided greater guidance in the assessment of domains and produced quantitative results. However, there were many problems with the construction of its criteria and AMSTAR was much easier to apply consistently. We recommend that AMSTAR incorporates the findings of this study and produces additional guidance for its application in order to improve its reliability and usefulness.     

Evaluating Outcomes Used in Cardiothoracic Surgery Interventional Research: A Systematic Review of Reviews to Develop a Core Outcome Set

Background

When planning clinical trials, it is a key element to choose appropriate outcomes that ensure the comparability of effects of interventions in ways that minimise bias. We hypothesise that outcome measures in cardiothoracic surgical trials are inconsistent and without standard. Therefore, comparing the relative effectiveness of interventions across studies is problematic. We surmise that cardiothoracic research has focused habitually on the identification of risk factors and on the reduction of adverse outcomes with less consideration of factors that contribute to well being and positive health outcomes (salutogenesis).

Methods and Findings

We conducted a systematic review of reviews to determine both the type and number of outcomes reported in current cardiothoracic surgery interventional research, in order to identify a list of potential outcomes for a minimum core outcome set (COS). Special focus was placed on outcomes that emphasise salutogenesis. We interpreted salutogenic outcomes as those relating to optimum and/or positive health and well being. We searched Issue 7 (July 2014) of the Cochrane Database of Systematic Reviews. Systematic reviews of randomised trials on non-minimal-invasive off- or on-pump cardiothoracic surgery (elective and emergency, excluding transplants) investigating pre-, intra- or postsurgical interventions related to the outcome of the procedure were eligible for inclusion. We excluded protocols and withdrawn systematic reviews. Two review authors extracted outcome data independently. Unique lists of salutogenically and non-salutogenically focused outcomes were established. 15 systematic reviews involving 371 randomized trials and 58,253 patients were included in this review. Applied definitions of single and composite endpoints varied significantly, and patient-centred, salutogenically focused outcomes were seldom reported. One third of included reviews did not assess patient-centred outcomes at all; all other reviews were unable to perform meta-analyses due to an absence of data or heterogeneity in outcome measures. This compares to 36 non-salutogenically focused outcome domains representing 121 individual non-salutogenically focused outcomes, whereof 50% were assessed only once. Measures of mortality, cerebrovascular complications and hospitalisation were reported most frequently. Two reviews chose a composite endpoint as primary outcome. Pooled analysis of composite endpoints was not possible, as the required data was not reported per patient in all components.

Conclusion

In cardiothoracic surgical trials, choice and definition of non-salutogenically focused single and composite outcomes are inconsistent. There is an absence of patient centred, salutogenically focused outcome parameters in cardiac trials. We recommend the development of a core outcome set of salutogenically focused and non-salutogenically focused outcomes for cardiothoracic surgical research.     

Human exposure to endocrine disrupters: carcinogenic risk assessment

Human exposure to endocrine disrupters (EDs) is widespread and is considered to pose a growing threat to human health. Recent advances in molecular and genetic research and better understanding of mechanisms of blastic cell transformation have led to efforts to improve cancer risk assessment for populations exposed to this family of xenobiotics. In risk assessment, low dose extrapolation of cancer incidence data from both experimental animals and epidemiology studies has been largely based on models assuming linear correlation at low doses, despite existence of evidence showing otherwise. Another weakness of ED risk assessment is poor exposure data in ecological studies. Those are frequently rough estimates derived from contaminated items of local food basket surveys. Polyhalogenated hydrocarbons are treated as examples. There is growing sense of urgency to develop a biologically based dose response model of cancer risk, integrating emerging data from molecular biology and epidemiology to provide more realistic data for risk assessors, public, public health managers and environmental issues administrators.     

Comparison of Contaminated Site Human Health Risk Assessment Approaches in Canada: Application of Provincial Methods to a Hypothetical Site

Human health risk assessment, whether at the screening level or more complex phase, is not an exact science. A wide variety of advice and direction is offered by international, national, and provincial/state environmental agencies regarding the conduct of risk assessment, and different risk assessors access and rely on the available regulatory advice and direction differently. This may result in wide variability in the estimates of chemical exposure and risk. A comparison of human health risk assessment approaches practiced at the provincial level in Canada was undertaken, wherein each jurisdiction's approach was applied to a hypothetical contaminated site. Approaches were found to vary both in terms of methodological considerations, and in matters of policy. The exercise yielded results in terms of estimated exposures and predicted hazard quotients/indexes and incremental lifetime cancer risks that were in some cases quite consistent (varying by a factor of less than 1.5 times), and in other cases remarkably different (varying by orders of magnitude). This article reviews the various approaches/frameworks applied and discusses the results of the hypothetical risk assessments, in terms of both the observed variation and the source of this variability.     

Science in Risk Assessment and Policy (SciRAP): An Online Resource for Evaluating and Reporting In Vivo (Eco)Toxicity Studies

(Eco)toxicity studies conducted according to internationally standardized test guidelines are often considered reliable by default and preferred as key evidence in regulatory risk assessment. At the same time regulatory agencies emphasize the use of all relevant (eco)toxicity data in the risk assessment process, including non-standard studies. However, there is a need to facilitate the use of such studies in regulatory risk assessment. Therefore, we propose a framework that facilitates a systematic and transparent evaluation of the reliability and relevance of (eco)toxicity in vivo studies for health and environmental risk assessment. The framework includes specific criteria to guide study evaluation, as well as a color-coding tool developed to aid the application of these criteria. In addition we provide guidance intended for researchers on how to report non-standard studies to ensure that they meet regulatory requirements. The intention of the evaluating and reporting criteria is to increase the usability of all relevant data that may fill information gaps in chemical risk assessments. The framework is publically available online, free of charge, at the Science in Risk Assessment and Policy (SciRAP) website: www.scirap.org. The aim of this article is to present the framework and resources available at the SciRAP website.