Detecting and adjusting for small-study effects in meta-analysis

Publication bias and related types of small-study effects threaten the validity of systematic reviews. The existence of small-study effects has been demonstrated in empirical studies. Small-study effects are graphically diagnosed by inspection of the funnel plot. Though observed funnel plot asymmetry cannot be easily linked to a specific reason, tests based on funnel plot asymmetry have been proposed. Beyond a vast range of funnel plot tests, there exist several methods for adjusting treatment effect estimates for these biases. In this article, we consider the trim-and-fill method, the Copas selection model, and more recent regression-based approaches. The methods are exemplified using a meta-analysis from the literature and compared in a simulation study, based on binary response data. They are also applied to a large set of meta-analyses. Some fundamental differences between the approaches are discussed. An assumption common to the trim-and-fill method and the Copas selection model is that the small-study effect is caused by selection. The trim-and-fill method corresponds to an unknown implicit model generated by the symmetry assumption, whereas the Copas selection model is a parametric statistical model. However, it requires a sensitivity analysis. Regression-based approaches are easier to implement and not based on a specific selection model. Both simulations and applications suggest that in the presence of strong selection both the trim-and-fill method and the Copas selection model may not fully eliminate bias, while regression-based approaches seem to be a promising alternative.     

Confidence intervals and P-values for meta-analysis with publication bias

We study publication bias in meta-analysis by supposing there is a population (y, sigma) of studies which give treatment effect estimates y approximately N(theta, sigma(2)). A selection function describes the probability that each study is selected for review. The overall estimate of theta depends on the studies selected, and hence on the (unknown) selection function. Our previous paper, Copas and Jackson (2004, Biometrics 60, 146-153), studied the maximum bias over all possible selection functions which satisfy the weak condition that large studies (small sigma) are as likely, or more likely, to be selected than small studies (large sigma). This led to a worst-case sensitivity analysis, controlling for the overall fraction of studies selected. However, no account was taken of the effect of selection on the uncertainty in estimation. This article extends the previous work by finding corresponding confidence intervals and P-values, and hence a new sensitivity analysis for publication bias. Two examples are discussed.     

Using journal impact factors to correct for the publication bias of medical studies

Publication bias of the results of medical studies can invalidate evidence-based medicine. The existing methodology for modeling this essentially relies upon the symmetry of the funnel plot. We present a new method of modeling publication bias that uses this information plus the impact factors of the publishing journals. A simple model of the publication process enables the estimation of bias-corrected intervention effects. The procedure is illustrated using a meta-analysis of the effectiveness of single-dose oral aspirin for acute pain, and results are also obtained for five other meta-analyses. The method enables the fitting of a wide range of models and is considered more flexible than other ways of compensating for publication bias. The model also provides the basis of a statistical test for the existence of publication bias. Use of the new methodology to supplement existing methods is recommended, in the context of a sensitivity analysis.     

Adjusting for publication bias in meta-analysis via inverse probability weighting using clinical trial registries

Publication bias is a major concern in conducting systematic reviews and meta-analyses. Various sensitivity analysis or bias-correction methods have been developed based on selection models, and they have some advantages over the widely used trim-and-fill bias-correction method. However, likelihood methods based on selection models may have difficulty in obtaining precise estimates and reasonable confidence intervals, or require a rather complicated sensitivity analysis process. Herein, we develop a simple publication bias adjustment method by utilizing the information on conducted but still unpublished trials from clinical trial registries. We introduce an estimating equation for parameter estimation in the selection function by regarding the publication bias issue as a missing data problem under the missing not at random assumption. With the estimated selection function, we introduce the inverse probability weighting (IPW) method to estimate the overall mean across studies. Furthermore, the IPW versions of heterogeneity measures such as the between-study variance and the I² measure are proposed. We propose methods to construct confidence intervals based on asymptotic normal approximation as well as on parametric bootstrap. Through numerical experiments, we observed that the estimators successfully eliminated bias, and the confidence intervals had empirical coverage probabilities close to the nominal level. On the other hand, the confidence interval based on asymptotic normal approximation is much wider in some scenarios than the bootstrap confidence interval. Therefore, the latter is recommended for practical use.     

Meta-analysis,funnel plots and sensitivity analysis

Publication bias is a major problem, perhaps the major problem, in meta-analysis (or systematic reviews). Small studies are more likely to be published if their results are 'significant' than if their results are negative or inconclusive, and so the studies available for review are biased in favour of those with positive outcomes. Correcting for this bias is not possible without making untestable assumptions. In this paper, a sensitivity analysis is suggested which is based on fitting a model to the funnel plot. Some examples are discussed.     

A Critical Meta-Analysis of Lens Model Studies in Human Judgment and Decision-Making

Achieving accurate judgment (‘judgmental achievement’) is of utmost importance in daily life across multiple domains. The lens model and the lens model equation provide useful frameworks for modeling components of judgmental achievement and for creating tools to help decision makers (e.g., physicians, teachers) reach better judgments (e.g., a correct diagnosis, an accurate estimation of intelligence). Previous meta-analyses of judgment and decision-making studies have attempted to evaluate overall judgmental achievement and have provided the basis for evaluating the success of bootstrapping (i.e., replacing judges by linear models that guide decision making). However, previous meta-analyses have failed to appropriately correct for a number of study design artifacts (e.g., measurement error, dichotomization), which may have potentially biased estimations (e.g., of the variability between studies) and led to erroneous interpretations (e.g., with regards to moderator variables). In the current study we therefore conduct the first psychometric meta-analysis of judgmental achievement studies that corrects for a number of study design artifacts. We identified 31 lens model studies (N = 1,151, k = 49) that met our inclusion criteria. We evaluated overall judgmental achievement as well as whether judgmental achievement depended on decision domain (e.g., medicine, education) and/or the level of expertise (expert vs. novice). We also evaluated whether using corrected estimates affected conclusions with regards to the success of bootstrapping with psychometrically-corrected models. Further, we introduce a new psychometric trim-and-fill method to estimate the effect sizes of potentially missing studies correct psychometric meta-analyses for effects of publication bias. Comparison of the results of the psychometric meta-analysis with the results of a traditional meta-analysis (which only corrected for sampling error) indicated that artifact correction leads to a) an increase in values of the lens model components, b) reduced heterogeneity between studies, and c) increases the success of bootstrapping. We argue that psychometric meta-analysis is useful for accurately evaluating human judgment and show the success of bootstrapping.     

Helicobacter pylori infection and colorectal cancer risk: a meta-analysis

BACKGROUND: Several studies suggested an association between Helicobacter pylori infection and colorectal carcinoma or adenoma risk. However, different authors reported quite varying estimates. We carried out a systematic review and meta-analysis of published studies investigating this association and paid special attention to the possibility of publication bias and sources of heterogeneity between studies. Materials and METHODS: An extensive literature search and cross-referencing were performed to identify all published studies. Summary estimates were obtained using random-effects models. The presence of possible publication bias was assessed using different statistical approaches. RESULTS: In a meta-analysis of the 11 identified human studies, published between 1991 and 2002, a summary odds ratio of 1.4 (95% CI, 1.1-1.8) was estimated for the association between H. pylori infection and colorectal cancer risk. The graphical funnel plot appeared asymmetrical, but the formal statistical evaluations did not provide strong evidence of publication bias. The proportion of variation of study results because of heterogeneity was small (36.5%). CONCLUSIONS: The results of our meta-analysis are consistent with a possible small increase in risk of colorectal cancer because of H. pylori infection. However, the possibility of some publication bias cannot be ruled out, although it could not be statistically confirmed. Larger, better designed and better controlled studies are needed to clarify the situation.     

Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis

We study recently developed nonparametric methods for estimating the number of missing studies that might exist in a meta-analysis and the effect that these studies might have had on its outcome. These are simple rank-based data augmentation techniques, which formalize the use of funnel plots. We show that they provide effective and relatively powerful tests for evaluating the existence of such publication bias. After adjusting for missing studies, we find that the point estimate of the overall effect size is approximately correct and coverage of the effect size confidence intervals is substantially improved, in many cases recovering the nominal confidence levels entirely. We illustrate the trim and fill method on existing meta-analyses of studies in clinical trials and psychometrics.     

Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview

Background

There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

Methods and Findings

Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93–1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.

Conclusions

Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors'' Summary     

No association between IL-1β -511 C/T polymorphism and the risk of duodenal ulcer: A meta-analysis of 4667 subjects

Epidemiological studies have evaluated the association between IL-1β -511 C/T polymorphism and duodenal ulcer (DU) risk. However, the results remain conflicting. The aim of this study was to perform a meta-analysis to investigate a more authentic association between IL-1β -511 C/T polymorphism and DU. Systematic searches of electronic databases Embase, PubMed and Web of Science as well as hand searching of the references of identified articles and the meeting abstracts were performed. Study selection, data abstraction and study quality evaluation were independently conducted in duplicate. Statistical analyses were performed using software Stata 11.0. The pooled odds ratios (ORs) with 95% confidence intervals (95% CIs) were performed. Publication bias was tested by Begg's funnel plot and Egger's regression test. A total of 14 studies including 1887 cases and 2780 controls were included in our final meta-analysis. There was no evidence of significant association between IL-1β -511 C/T polymorphism and DU (for T allele vs. C allele: OR=0.93, 95% CI=0.82-1.06; for T/T vs. C/C: OR=0.83, 95% CI=0.64-1.08; for dominant model: OR=0.93, 95% CI=0.80-1.07; and for recessive model: OR=0.87, 95% CI=0.69-1.11). Significant association was found in all genetic models for the PB subgroup and sensitivity analyses. In conclusion, our meta-analysis suggests that there was no evidence of a significant association between IL-1β -511 C/T polymorphism and DU with or without Helicobacter pylori infection, whereas a significant association was found by sensitivity analyses which showed a protective effect of the T allele against DU risk.     

Testing small study effects in multivariate meta-analysis

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta-analyses. The most well-known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta-analysis have been extensively studied. However, detecting small study effects in a multivariate meta-analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta-analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.     

SnIPRE: Selection Inference Using a Poisson Random Effects Model

We present an approach for identifying genes under natural selection using polymorphism and divergence data from synonymous and non-synonymous sites within genes. A generalized linear mixed model is used to model the genome-wide variability among categories of mutations and estimate its functional consequence. We demonstrate how the model''s estimated fixed and random effects can be used to identify genes under selection. The parameter estimates from our generalized linear model can be transformed to yield population genetic parameter estimates for quantities including the average selection coefficient for new mutations at a locus, the synonymous and non-synynomous mutation rates, and species divergence times. Furthermore, our approach incorporates stochastic variation due to the evolutionary process and can be fit using standard statistical software. The model is fit in both the empirical Bayes and Bayesian settings using the lme4 package in R, and Markov chain Monte Carlo methods in WinBUGS. Using simulated data we compare our method to existing approaches for detecting genes under selection: the McDonald-Kreitman test, and two versions of the Poisson random field based method MKprf. Overall, we find our method universally outperforms existing methods for detecting genes subject to selection using polymorphism and divergence data.     

Clustering threshold gradient descent regularization: with applications to microarray studies

MOTIVATION: An important goal of microarray studies is to discover genes that are associated with clinical outcomes, such as disease status and patient survival. While a typical experiment surveys gene expressions on a global scale, there may be only a small number of genes that have significant influence on a clinical outcome. Moreover, expression data have cluster structures and the genes within a cluster have correlated expressions and coordinated functions, but the effects of individual genes in the same cluster may be different. Accordingly, we seek to build statistical models with the following properties. First, the model is sparse in the sense that only a subset of the parameter vector is non-zero. Second, the cluster structures of gene expressions are properly accounted for. RESULTS: For gene expression data without pathway information, we divide genes into clusters using commonly used methods, such as K-means or hierarchical approaches. The optimal number of clusters is determined using the Gap statistic. We propose a clustering threshold gradient descent regularization (CTGDR) method, for simultaneous cluster selection and within cluster gene selection. We apply this method to binary classification and censored survival analysis. Compared to the standard TGDR and other regularization methods, the CTGDR takes into account the cluster structure and carries out feature selection at both the cluster level and within-cluster gene level. We demonstrate the CTGDR on two studies of cancer classification and two studies correlating survival of lymphoma patients with microarray expressions. AVAILABILITY: R code is available upon request. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.     

The effect of zinc supplementation on brain derived neurotrophic factor: A meta-analysis

BackgroundZinc in one of the most abundant trace minerals in human body which is involved in numerous biological pathways and has variety of roles in the nervous system. It has been assumed that zinc exerts its role in nervous system through increasing brain derived neurotrophic factor (BDNF) concentrations.ObjectivesPresent meta-analysis was aimed to review the effect of zinc supplementation on serum concentrations of BDNF.Methods and materialsFour electronic databases (Pubmed, Scopus, Web of Science, Embase) were searched for identifying studies that examined BDNF levels prior and after zinc supplementation up to May 2020. According to the Cochrane guideline, a meta-analysis was performed to pool the effect size estimate (Hedges’ test) of serum BDNF across studies. Risk of publication bias was assessed using a funnel plot and Egger’s test.ResultsFive studies were eligible and 238 participants were included. These studies enrolled subjects with premenstrual syndrome, diabetic retinopathy, major depression disorder, overweight/obese and obese with mild to moderate depressive disorders. Zinc supplementation failed to increase blood BDNF concentrations with effect size of 0.30 (95 % CI: -0.08, 0.67, P = 0.119). Funnel plot did not suggest publication bias.ConclusionZinc supplementation may not significantly increase BDNF levels. However, the small number of included articles and significant heterogeneity between them can increase the risk of a false negative result; therefore, the results should be interpreted with caution.     

Systemic Hypertension as a Risk Factor for Open-Angle Glaucoma: A Meta-Analysis of Population-Based Studies

Background/Aims

Systemic hypertension is thought to increase the risk for developing open-angle glaucoma (OAG) through several mechanisms. However, previous epidemiological studies have shown conflicting results regarding this potential association. We systematically evaluated this issue by conducting a meta-analysis of population-based studies.

Methods

A comprehensive search for articles published before 31 March 2014 was performed using PubMed, Embase, and reference lists. The pooled odds ratio (OR) was calculated using the fixed- and random-effects models, and meta-regression was performed according to age. Subgroup analyses were also conducted, and publication bias was assessed using a funnel plot and Egger’s regression test.

Results

This meta-analysis included 16 studies involving 60,084 individuals, with substantial homogeneity among the studies. The pooled OR for OAG was 1.22 (95% confidence interval, CI: 1.09–1.36) using the fixed-effects model and 1.22 (95% CI: 1.08–1.37) using the random-effects model in all included studies. For subgroup analyses, the pooled OR for high-tension glaucoma (HTG) was higher than that for normal-tension glaucoma (NTG) (OR = 1.92 and 0.94, respectively). No significant difference was detected between Asian and Western populations, and no publication bias was detected in either analysis.

Conclusions

Systemic hypertension increases the risk for developing OAG, especially in those with HTG.     

Random-effects model aimed at discovering associations in meta-analysis of genome-wide association studies

Meta-analysis is an increasingly popular tool for combining multiple different genome-wide association studies (GWASs) in a single aggregate analysis in order to identify associations with very small effect sizes. Because the data of a meta-analysis can be heterogeneous, referring to the differences in effect sizes between the collected studies, what is often done in the literature is to apply both the fixed-effects model (FE) under an assumption of the same effect size between studies and the random-effects model (RE) under an assumption of varying effect size between studies. However, surprisingly, RE gives less significant p values than FE at variants that actually show varying effect sizes between studies. This is ironic because RE is designed specifically for the case in which there is heterogeneity. As a result, usually, RE does not discover any associations that FE did not discover. In this paper, we show that the underlying reason for this phenomenon is that RE implicitly assumes a markedly conservative null-hypothesis model, and we present a new random-effects model that relaxes the conservative assumption. Unlike the traditional RE, the new method is shown to achieve higher statistical power than FE when there is heterogeneity, indicating that the new method has practical utility for discovering associations in the meta-analysis of GWASs.     

Lack of Association between Bax Promoter (-248G>A) Single Nucleotide Polymorphism and Susceptibility towards Cancer: Evidence from a Meta-Analysis

Background

The Bcl-2-associated X protein (Bax) is a proapoptotic member of the Bcl-2 family known to be activated and upregulated during apoptosis. Single nucleotide polymorphisms (SNPs) in Bax promoter may participate in the process of carcinogenesis by altering its own expression and the cancer related genes. Bax-248G>A polymorphism has been implicated to alter the risk of cancer, but the listed results are inconsistent and inconclusive. In the present study, we performed a meta-analysis to systematically summarize the possible association of this polymorphism with the risk of cancer.

Methodology

We conducted a search of case-control studies on the associations of Bax-248G>A polymorphism with susceptibility to cancer in Pub Med, Science Direct, Wiley Online Library and hand search. Data from all eligible studies based on some key search terms, inclusion and exclusion criteria were extracted for this meta-analysis. Hardy-Weinberg equilibrium (HWE) in controls, power calculation, heterogeneity analysis, Begg’s funnel plot, Egger’s linear regression test, forest plot and sensitivity analysis were performed in the present study.

Results

Cancer risk associated with Bax-248G>A polymorphism was estimated by pooled odds ratios (ORs) and 95% confidence intervals (95% CIs). The pooled ORs were calculated in allele contrast, homozygous comparison, heterozygous comparison, dominant and recessive model. Statistical significance was checked through Z and p-value in forest plot. A total of seven independent studies including 1772 cases and 1708 controls were included in our meta-analysis. Our results showed that neither allele frequency nor genotype distributions of this polymorphism were associated with risk for cancer in any of the genetic model. Furthermore, Egger’s test did not show any substantial evidence of publication bias.

Conclusions/Significance

This meta-analysis suggests that the Bax-248G>A polymorphism is not an important cancer risk factor. Nevertheless, additional well-designed studies with larger sample size focusing on different ethnicities and cancer types are required to further validate the results.     

Presenilin-2 polymorphisms and risk of sporadic AD: Evidence from a meta-analysis

Association studies of presenilin-2 (PSEN2) polymorphisms and sporadic Alzheimer's disease (AD) have yielded inconsistent results, possibly because single studies often lack sufficient statistical power. In this study, we performed a meta-analysis to evaluate the association of the two most extensively studied PSEN2 polymorphisms, rs8383 and 5′indel, with the risk of sporadic AD. We systematically reviewed relevant studies retrieved by Medline, Pubmed, Embase, AlzGene, and CNKI. Data were analyzed using the Stata (v11.0) software package. The fixed effects model or random-effects model were applied depending on between-study heterogeneity. Publication bias was evaluated using Egger's test and Begg's funnel plots. Overall, the meta-analysis included 6 case–control studies for each polymorphism with 2186 confirmed AD cases and 2507 healthy controls in total. Analysis suggested a significant association between SNP rs8383 polymorphism and AD risk with no evidence of between-study heterogeneity or publication bias. In contrast, we found no evidence for an association between the 5′indel polymorphism and AD risk. Further stratified analyses by apolipoprotein ε4 status or ethnicity also failed to reveal a statistically significant association between the 5′indel polymorphism of PSEN2 and AD risk. Our analysis supports the hypothesis that the PSEN2 rs8383 polymorphism is associated with an enlarged risk of sporadic AD. However, larger scale association studies are necessary to further validate the association of PSEN2 polymorphisms with sporadic AD risk and to define potential gene–gene interactions.     

Variable selection in nonlinear function-on-scalar regression

We develop a new method for variable selection in a nonlinear additive function-on-scalar regression (FOSR) model. Existing methods for variable selection in FOSR have focused on the linear effects of scalar predictors, which can be a restrictive assumption in the presence of multiple continuously measured covariates. We propose a computationally efficient approach for variable selection in existing linear FOSR using functional principal component scores of the functional response and extend this framework to a nonlinear additive function-on-scalar model. The proposed method provides a unified and flexible framework for variable selection in FOSR, allowing nonlinear effects of the covariates. Numerical analysis using simulation study illustrates the advantages of the proposed method over existing variable selection methods in FOSR even when the underlying covariate effects are all linear. The proposed procedure is demonstrated on accelerometer data from the 2003–2004 cohorts of the National Health and Nutrition Examination Survey (NHANES) in understanding the association between diurnal patterns of physical activity and demographic, lifestyle, and health characteristics of the participants.     

Multilocus models in the infinite island model of population structure

Different methods have been developed to consider the effects of statistical associations among genes that arise in population genetics models: kin selection models deal with associations among genes present in different interacting individuals, while multilocus models deal with associations among genes at different loci. It was pointed out recently that these two types of models are very similar in essence. In this paper, we present a method to construct multilocus models in the infinite island model of population structure (where deme size may be arbitrarily small). This method allows one to compute recursions on allele frequencies, and different types of genetic associations (including associations between different individuals from the same deme), and incorporates selection. Recursions can be simplified using quasi-equilibrium approximations; however, we show that quasi-equilibrium calculations for associations that are different from zero under neutrality must include a term that has not been previously considered. The method is illustrated using simple examples.