Design,Analysis, and Reporting of Crossover Trials for Inclusion in a Meta-Analysis

Randomized crossover trials are clinical experiments in which participants are assigned randomly to a sequence of treatments and each participant serves as his/her own control in estimating treatment effect. We need a better understanding of the validity of their results to enable recommendations as to which crossover trials can be included in meta-analysis and for development of reporting guidelines.

Objective

To evaluate the characteristics of the design, analysis, and reporting of crossover trials for inclusion in a meta-analysis of treatment for primary open-angle glaucoma and to provide empirical evidence to inform the development of tools to assess the validity of the results from crossover trials and reporting guidelines.

Methods

We searched MEDLINE, EMBASE, and Cochrane’s CENTRAL register for randomized crossover trials for a systematic review and network meta-analysis we are conducting. Two individuals independently screened the search results for eligibility and abstracted data from each included report.

Results

We identified 83 crossover trials eligible for inclusion. Issues affecting the risk of bias in crossover trials, such as carryover, period effects and missing data, were often ignored. Some trials failed to accommodate the within-individual differences in the analysis. For a large proportion of the trials, the authors tabulated the results as if they arose from a parallel design. Precision estimates properly accounting for the paired nature of the design were often unavailable from the study reports; consequently, to include trial findings in a meta-analysis would require further manipulation and assumptions.

Conclusions

The high proportion of poorly reported analyses and results has the potential to affect whether crossover data should or can be included in a meta-analysis. There is pressing need for reporting guidelines for crossover trials.     

Comparing the generalized matching law and contingency discriminability model as accounts of concurrent schedule performance using residual meta-analysis

To compare the generalized matching law (Baum, W.M., 1974b. On two types of deviation from the matching law: bias and undermatching. J. Exp. Anal. Behav. 22, 231-242) and contingency discriminability model (Davison, M., Jenkins, P.E., 1985. Stimulus discriminability, contingency discriminability, and schedule performance. Anim. Learn. Behav. 13, 77-84) as accounts of concurrent schedule performance, we conducted a residual meta-analysis of response- and time-allocation data from 20 studies (n's=886 and 774, respectively). Both models were fitted to the individual-subject data from each study, and residuals were obtained. Polynomial regressions were then performed on the pooled residuals to determine whether systematic trends were present as a function of predicted values. For the contingency discriminability model, the cubic coefficients were positive and statistically significant for both response- and time-allocation data. By contrast, no statistically significant systematic trend was obtained in the residuals for the generalized matching law. These results suggest that the relationship between log response allocation and log reinforcer allocation does not deviate significantly from linearity over an approximate range of +1.25 to -1.25 log units, consistent with the generalized matching law. Although qualitative criteria are also important in comparing models of behavioral phenomena, residual meta-analysis provides a powerful quantitative methodology for model selection and should prove useful in future research.     

How are we to determine whether dietary interventions do more good than harm to hypertensive patients?

This paper categorizes studies of dietary interventions in hypertensive patients into five categories based on the levels of evidence: level I, randomized trials with low false-positive (alpha) and low false-negative (beta) errors (high power); level II, randomized trials with high false-positive (alpha) and (or) high false-negative (beta) errors (low power); level III, nonrandomized concurrent cohort comparisons between contemporaneous patients who did and did not receive a nutritional intervention; level IV, nonrandomized historical cohort comparisons between current patients who did receive a nutritional intervention and former patients (from the same institution or from the literature) who did not; level V, case series without controls.     

Statin Use and Risk of Lung Cancer: a Meta-Analysis of Observational Studies and Randomized Controlled Trials

Clinical studies have shown that statin use may alter the risk of lung cancer. However, these studies yielded different results. To quantify the association between statin use and risk of lung cancer, we performed a detailed meta-analysis. A literature search was carried out using MEDLINE, EMBASE and COCHRANE database between January 1966 and November 2012. Before meta-analysis, between-study heterogeneity and publication bias were assessed using adequate statistical tests. Fixed-effect and random-effect models were used to calculate the pooled relative risks (RR) and corresponding 95% confidence intervals (CIs). Subgroup analyses, sensitivity analysis and cumulative meta-analysis were also performed. A total of 20 (five randomized controlled trials, eight cohorts, and seven case–control) studies contributed to the analysis. Pooled results indicated a non-significant decrease of total lung cancer risk among all statin users (RR = 0.89, 95% CI [0.78, 1.02]). Further, long-term statin use did not significantly decrease the risk of total lung cancer (RR = 0.80, 95% CI [0.39 , 1.64]). In our subgroup analyses, the results were not substantially affected by study design, participant ethnicity, or confounder adjustment. Furthermore, sensitivity analysis confirmed the stability of results. The findings of this meta-analysis suggested that there was no significant association between statin use and risk of lung cancer. More studies, especially randomized controlled trials and high quality cohort studies are warranted to confirm this association.     

Clarifying the role of principal stratification in the paired availability design

The paired availability design for historical controls postulated four classes corresponding to the treatment (old or new) a participant would receive if arrival occurred during either of two time periods associated with different availabilities of treatment. These classes were later extended to other settings and called principal strata. Judea Pearl asks if principal stratification is a goal or a tool and lists four interpretations of principal stratification. In the case of the paired availability design, principal stratification is a tool that falls squarely into Pearl's interpretation of principal stratification as "an approximation to research questions concerning population averages." We describe the paired availability design and the important role played by principal stratification in estimating the effect of receipt of treatment in a population using data on changes in availability of treatment. We discuss the assumptions and their plausibility. We also introduce the extrapolated estimate to make the generalizability assumption more plausible. By showing why the assumptions are plausible we show why the paired availability design, which includes principal stratification as a key component, is useful for estimating the effect of receipt of treatment in a population. Thus, for our application, we answer Pearl's challenge to clearly demonstrate the value of principal stratification.     

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.

Methodology/Principal Findings

We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Therapist-supported Internet-based cognitive behaviour therapy yields similar effects as face-to-face therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis

Providing therapist-guided cognitive behaviour therapy via the Internet (ICBT) has advantages, but a central research question is to what extent similar clinical effects can be obtained as with gold-standard face-to-face cognitive behaviour therapy (CBT). In a previous meta-analysis published in this journal, which was updated in 2018, we found evidence that the pooled effects for the two formats were equivalent in the treatment of psychiatric and somatic disorders, but the number of published randomized trials was relatively low (n=20). As this is a field that moves rapidly, the aim of the current study was to conduct an update of our systematic review and meta-analysis of the clinical effects of ICBT vs. face-to-face CBT for psychiatric and somatic disorders in adults. We searched the PubMed database for relevant studies published from 2016 to 2022. The main inclusion criteria were that studies had to compare ICBT to face-to-face CBT using a randomized controlled design and targeting adult populations. Quality assessment was made using the Cochrane risk of bias criteria (Version 1), and the main outcome estimate was the pooled standardized effect size (Hedges’ g) using a random effects model. We screened 5,601 records and included 11 new randomized trials, adding them to the 20 previously identified ones (total n=31). Sixteen different clinical conditions were targeted in the included studies. Half of the trials were in the fields of depression/depressive symptoms or some form of anxiety disorder. The pooled effect size across all disorders was g=0.02 (95% CI: –0.09 to 0.14) and the quality of the included studies was acceptable. This meta-analysis further supports the notion that therapist-supported ICBT yields similar effects as face-to-face CBT.     

Testing small study effects in multivariate meta-analysis

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta-analyses. The most well-known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta-analysis have been extensively studied. However, detecting small study effects in a multivariate meta-analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta-analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.     

The effect of interpersonal psychotherapy and other psychodynamic therapies versus 'treatment as usual' in patients with major depressive disorder

Background

Major depressive disorder afflicts an estimated 17% of individuals during their lifetimes at tremendous suffering and costs. Interpersonal psychotherapy and other psychodynamic therapies may be effective interventions for major depressive disorder, but the effects have only had limited assessment in systematic reviews.

Methods/Principal Findings

Cochrane systematic review methodology with meta-analysis and trial sequential analysis of randomized trials comparing the effect of psychodynamic therapies versus ‘treatment as usual’ for major depressive disorder. To be included the participants had to be older than 17 years with a primary diagnosis of major depressive disorder. Altogether, we included six trials randomizing a total of 648 participants. Five trials assessed ‘interpersonal psychotherapy’ and only one trial assessed ‘psychodynamic psychotherapy’. All six trials had high risk of bias. Meta-analysis on all six trials showed that the psychodynamic interventions significantly reduced depressive symptoms on the 17-item Hamilton Rating Scale for Depression (mean difference −3.12 (95% confidence interval −4.39 to −1.86;P<0.00001), no heterogeneity) compared with ‘treatment as usual’. Trial sequential analysis confirmed this result.

Discussion

We did not find convincing evidence supporting or refuting the effect of interpersonal psychotherapy or psychodynamic therapy compared with ‘treatment as usual’ for patients with major depressive disorder. The potential beneficial effect seems small and effects on major outcomes are unknown. Randomized trials with low risk of systematic errors and low risk of random errors are needed.     

Quantification of overdiagnosis in randomised trials of cancer screening: an overview and re-analysis of systematic reviews

The degree of overdiagnosis in common cancer screening trials is uncertain due to inadequate design of trials, varying definition and methods used to estimate overdiagnosis. Therefore, we aimed to quantify the risk of overdiagnosis for the most widely implemented cancer screening programmes and assess the implications of design limitations and biases in cancer screening trials on the estimates of overdiagnosis by conducting an overview and re-analysis of systematic reviews of cancer screening. We searched PubMed and the Cochrane Library from their inception dates to November 29, 2021. Eligible studies included systematic reviews of randomised trials comparing cancer screening interventions to no screening, which reported cancer incidence for both trial arms. We extracted data on study characteristics, cancer incidence and assessed the risk of bias using the Cochrane Collaboration’s risk of bias tool. We included 19 trials described in 30 articles for review, reporting results for the following types of screening: mammography for breast cancer, chest X-ray or low-dose CT for lung cancer, alpha-foetoprotein and ultrasound for liver cancer, digital rectal examination, prostate-specific antigen, and transrectal ultrasound for prostate cancer, and CA-125 test and/or ultrasound for ovarian cancer. No trials on screening for melanoma were eligible. Only one trial (5%) had low risk in all bias domains, leading to a post-hoc meta-analysis, excluding trials with high risk of bias in critical domains, finding the extent of overdiagnosis ranged from 17% to 38% across cancer screening programmes. We conclude that there is a significant risk of overdiagnosis in the included randomised trials on cancer screening. We found that trials were generally not designed to estimate overdiagnosis and many trials had high risk of biases that may draw the estimates of overdiagnosis towards the null. In effect, the true extent of overdiagnosis due to cancer screening is likely underestimated.     

Addressing Dichotomous Data for Participants Excluded from Trial Analysis: A Guide for Systematic Reviewers

Introduction

Systematic reviewer authors intending to include all randomized participants in their meta-analyses need to make assumptions about the outcomes of participants with missing data.

Objective

The objective of this paper is to provide systematic reviewer authors with a relatively simple guidance for addressing dichotomous data for participants excluded from analyses of randomized trials.

Methods

This guide is based on a review of the Cochrane handbook and published methodological research. The guide deals with participants excluded from the analysis who were considered ‘non-adherent to the protocol’ but for whom data are available, and participants with missing data.

Results

Systematic reviewer authors should include data from ‘non-adherent’ participants excluded from the primary study authors'' analysis but for whom data are available. For missing, unavailable participant data, authors may conduct a complete case analysis (excluding those with missing data) as the primary analysis. Alternatively, they may conduct a primary analysis that makes plausible assumptions about the outcomes of participants with missing data. When the primary analysis suggests important benefit, sensitivity meta-analyses using relatively extreme assumptions that may vary in plausibility can inform the extent to which risk of bias impacts the confidence in the results of the primary analysis. The more plausible assumptions draw on the outcome event rates within the trial or in all trials included in the meta-analysis. The proposed guide does not take into account the uncertainty associated with assumed events.

Conclusions

This guide proposes methods for handling participants excluded from analyses of randomized trials. These methods can help in establishing the extent to which risk of bias impacts meta-analysis results.     

Effect of intravitreal anti-vascular endothelial growth factor therapy on the risk of arterial thromboembolic events: a meta-analysis

Background

Intravitreal anti-vascular endothelial growth factor (VEGF) monoclonal antibodies are used in ocular neovascular diseases. A consensus has emerged that intravenous anti-VEGF can increase the risk of arterial thromboembolic events. However, the role of intravitreal anti-VEGF in arterial thromboembolism is controversial. Therefore, we did a systematic review and meta-analysis to investigate the effects of intravitreal anti-VEGF on the risk of arterial thromboembolic events.

Methods

Electronic databases were searched to identify relevant randomized clinical trials comparing intravitreal anti-VEGF with controls. Criteria for inclusion in our meta-analysis included a study duration of no less than 12 months, the use of a randomized control group not receiving any intravitreal active agent, and the availability of outcome data for arterial thromboembolic events, myocardial infarction, cerebrovascular accidents, and vascular death. The risk ratios and 95% CIs were calculated using a fixed-effects or random-effects model, depending on the heterogeneity of the included studies.

Results

A total of 4942 patients with a variety of ocular neovascular diseases from 13 randomized controlled trials were identified and included for analysis. There was no significant difference between intravitreal anti-VEGF and control in the risk of all events, with risk ratios of 0.87 (95% CI, 0.64 to 1.19) for arterial thromboembolic events, 0.96 (95% CI, 0.55–1.68) for cerebrovascular accidents, 0.69 (95% CI 0.40–1.21) for myocardial infarctions, and 0.68 (95% CI, 0.37–1.27) for vascular death.

Conclusions

The strength evidence suggests that the intravitreal use of anti-VEGF antibodies is not associated with an increased risk of arterial thromboembolic events.     

Merging the Idiographic Filter With Dynamic Factor Analysis to Model Process

To model processes we propose merging idiographic filter measurement with dynamic factor analysis. This involves testing whether or not the same latent dynamics (concurrent and lagged factor interrelations) can describe different individuals' observed multivariate time series. The methodology allows fitting, across different individuals, dynamic factor models that are invariant with respect to the latent dynamics, but not necessarily the factor loadings (measurement model). This methodology allows the same latent process to manifest differently from one individual to another, thus recognizing that the process is general but its realization in a given person is to some degree idiosyncratic. The approach is illustrated with empirical data.     

Can Chinese Herbal Medicine Improve Outcomes of In Vitro Fertilization? A Systematic Review and Meta-Analysis of Randomized Controlled Trials

Background

A large number of infertile couples are choosing Chinese herbal medicine (CHM) as an adjuvant therapy to improve their success when undergoing in vitro fertilization (IVF). There is no systematic review to evaluate the impact of CHM on the IVF outcomes.

Objective

To evaluate the effectiveness of CHM with concurrent IVF versus IVF alone on the outcomes of IVF and its safety.

Methods

The protocol of this study is registered at PROSPERO. Eligible RCTs searched from 8 databases which compared a combination of CHM and IVF with IVF alone were included. Two authors independently selected studies, extracted data and assessed methodological quality. Meta-analysis of RCTs was conducted if there was non-significant heterogeneity (evaluated by I² test) among trials. All statistical analysis was performed using RevMan 5.1 software.

Results

Twenty trials involving 1721 women were included in the meta-analysis. Three trials were evaluated as having an unclear risk of bias. The remaining trials were evaluated as having a high risk of bias. Combination of CHM and IVF significantly increases clinical pregnancy rates (OR 2.04, 95%CI 1.67 to 2.49, p<0.00001) and ongoing pregnancy rates (OR 1.91, 95%CI 1.17 to 3.10, p = 0.009). Use of CHM after embryo transfer had no better outcome in reducing the rate of ovarian hyper stimulation syndrome (OR 0.39, 95%CI 0.14 to 1.11, p = 0.08).

Conclusions

This meta-analysis showed that combination of IVF and CHM used in the included trials improve IVF success, however due to the high risk of bias observed with the trials, the significant differences found with the meta-analysis are unlikely to be accurate. No conclusion could be drawn with respect to the reproductive toxicity of CHM. Further large randomized placebo controlled trials are warranted to confirm these findings before recommending women to take CHM to improve their IVF success.     

Comparing G: multivariate analysis of genetic variation in multiple populations

The additive genetic variance–covariance matrix (G) summarizes the multivariate genetic relationships among a set of traits. The geometry of G describes the distribution of multivariate genetic variance, and generates genetic constraints that bias the direction of evolution. Determining if and how the multivariate genetic variance evolves has been limited by a number of analytical challenges in comparing G-matrices. Current methods for the comparison of G typically share several drawbacks: metrics that lack a direct relationship to evolutionary theory, the inability to be applied in conjunction with complex experimental designs, difficulties with determining statistical confidence in inferred differences and an inherently pair-wise focus. Here, we present a cohesive and general analytical framework for the comparative analysis of G that addresses these issues, and that incorporates and extends current methods with a strong geometrical basis. We describe the application of random skewers, common subspace analysis, the 4th-order genetic covariance tensor and the decomposition of the multivariate breeders equation, all within a Bayesian framework. We illustrate these methods using data from an artificial selection experiment on eight traits in Drosophila serrata, where a multi-generational pedigree was available to estimate G in each of six populations. One method, the tensor, elegantly captures all of the variation in genetic variance among populations, and allows the identification of the trait combinations that differ most in genetic variance. The tensor approach is likely to be the most generally applicable method to the comparison of G-matrices from any sampling or experimental design.     

Graphical Tools for Network Meta-Analysis in STATA

Network meta-analysis synthesizes direct and indirect evidence in a network of trials that compare multiple interventions and has the potential to rank the competing treatments according to the studied outcome. Despite its usefulness network meta-analysis is often criticized for its complexity and for being accessible only to researchers with strong statistical and computational skills. The evaluation of the underlying model assumptions, the statistical technicalities and presentation of the results in a concise and understandable way are all challenging aspects in the network meta-analysis methodology. In this paper we aim to make the methodology accessible to non-statisticians by presenting and explaining a series of graphical tools via worked examples. To this end, we provide a set of STATA routines that can be easily employed to present the evidence base, evaluate the assumptions, fit the network meta-analysis model and interpret its results.     

Assessing the implications of publication bias for two popular estimates of between-study variance in meta-analysis

Perhaps the greatest threat to the validity of a meta-analysis is the possibility of publication bias, where studies with interesting or statistically significant results are more likely to be published. This obviously impacts on inference concerning the treatment effect but also has implications for estimates of between-study variance. Two popular and established estimation methods are considered and formulae for assessing the implications of the bias are provided in terms of a general process for selecting studies. Meta-analysts, concerned that publication bias may be present, can use these as part of a sensitivity analysis to assess the robustness of their estimates of between-study variance using any selection process that is likely to be used in practice. The procedure is illustrated using a meta-analysis of clinical trials concerning the effectiveness of endoscopic sclerotherapy for preventing death in patients with cirrhosis and oesophagogastric varices.     

Systematic reviews: a primer for plastic surgery research

Clinicians rely on review articles to keep current with the rapid accumulation of medical and surgical literature. Traditional expert reviews, however, often suffer from inherent personal biases and may not reflect a true synthesis of the existing literature on a particular subject. Systematic reviews are structured, scientific articles that address the shortcomings of traditional reviews by adhering to strict, reproducible methods and recommended guidelines. The methods are designed to eliminate possible sources of bias, ensure as complete a review of the existing literature as possible, and present the results in a way that is useful for its intended audience. Systematic reviews may at times include a quantitative synthesis of the available data in the form of a meta-analysis. Meta-analysis is a statistical tool for combining the numerical results of separate studies to obtain a summary outcome with increased precision due to the larger, combined number of patients. Meta-analyses may be particularly helpful when individual study results are conflicting and the existing literature is inconclusive. The validity of meta-analysis, however, is highly dependent on the quality of data available in the literature. In its strictest form, meta-analysis is used to combine data from only randomized controlled clinical trials. Because randomized controlled clinical trials are infrequently performed in plastic surgery research, this article will focus on systematic reviews to provide the readers with a useful guide in performing this field of study.     

Core Outcomes for Colorectal Cancer Surgery: A Consensus Study

BackgroundColorectal cancer (CRC) is a major cause of worldwide morbidity and mortality. Surgical treatment is common, and there is a great need to improve the delivery of such care. The gold standard for evaluating surgery is within well-designed randomized controlled trials (RCTs); however, the impact of RCTs is diminished by a lack of coordinated outcome measurement and reporting. A solution to these issues is to develop an agreed standard “core” set of outcomes to be measured in all trials to facilitate cross-study comparisons, meta-analysis, and minimize outcome reporting bias. This study defines a core outcome set for CRC surgery.ConclusionThis study used robust consensus methodology to develop a core outcome set for use in colorectal cancer surgical trials. It is now necessary to validate the use of this set in research practice.