Dissemination biases in ecology: effect sizes matter more than quality

Publication and citation decisions in ecology are likely influenced by many factors, potentially including journal impact factors, direction and magnitude of reported effects, and year of publication. Dissemination bias exists when publication or citation of a study depends on any of these factors. We defined several dissemination biases and determined their prevalence across many sub‐disciplines in ecology, then determined whether or not data quality also affected these biases. We identified dissemination biases in ecology by conducting a meta‐analysis of citation trends for 3867 studies included in 52 meta‐analyses. We correlated effect size, year of publication, impact factor and citation rate within each meta‐analysis. In addition, we explored how data quality as defined in meta‐analyses (sample size or variance) influenced each form of bias. We also explored how the direction of the predicted or observed effect, and the research field, influenced any biases. Year of publication did not influence citation rates. The first papers published in an area reported the strongest effects, and high impact factor journals published the most extreme effects. Effect size was more important than data quality for many publication and citation trends. Dissemination biases appear common in ecology, and although their magnitude was generally small many were associated with theory tenacity, evidenced as tendencies to cite papers that most strongly support our ideas. The consequences of this behavior are amplified by the fact that papers reporting strong effects were often of lower data quality than papers reporting much weaker effects. Furthermore, high impact factor journals published the strongest effects, generally in the absence of any correlation with data quality. Increasing awareness of the prevalence of theory tenacity, confirmation bias, and the inattention to data quality among ecologists is a first step towards reducing the impact of these biases on research in our field.     

A population‐averaged approach to diagnostic test meta‐analysis

The meta‐analysis of diagnostic accuracy studies is often of interest in screening programs for many diseases. The typical summary statistics for studies chosen for a diagnostic accuracy meta‐analysis are often two dimensional: sensitivities and specificities. The common statistical analysis approach for the meta‐analysis of diagnostic studies is based on the bivariate generalized linear‐mixed model (BGLMM), which has study‐specific interpretations. In this article, we present a population‐averaged (PA) model using generalized estimating equations (GEE) for making inference on mean specificity and sensitivity of a diagnostic test in the population represented by the meta‐analytic studies. We also derive the marginalized counterparts of the regression parameters from the BGLMM. We illustrate the proposed PA approach through two dataset examples and compare performance of estimators of the marginal regression parameters from the PA model with those of the marginalized regression parameters from the BGLMM through Monte Carlo simulation studies. Overall, both marginalized BGLMM and GEE with sandwich standard errors maintained nominal 95% confidence interval coverage levels for mean specificity and mean sensitivity in meta‐analysis of 25 of more studies even under misspecification of the covariance structure of the bivariate positive test counts for diseased and nondiseased subjects.     

Ignoring non‐English‐language studies may bias ecological meta‐analyses

Meta‐analysis plays a crucial role in syntheses of quantitative evidence in ecology and biodiversity conservation. The reliability of estimates in meta‐analyses strongly depends on unbiased sampling of primary studies. Although earlier studies have explored potential biases in ecological meta‐analyses, biases in reported statistical results and associated study characteristics published in different languages have never been tested in environmental sciences. We address this knowledge gap by systematically searching published meta‐analyses and comparing effect‐size estimates between English‐ and Japanese‐language studies included in existing meta‐analyses. Of the 40 published ecological meta‐analysis articles authored by those affiliated to Japanese institutions, we find that three meta‐analysis articles searched for studies in the two languages and involved sufficient numbers of English‐ and Japanese‐language studies, resulting in four eligible meta‐analyses (i.e., four meta‐analyses conducted in the three meta‐analysis articles). In two of the four, effect sizes differ significantly between the English‐ and Japanese‐language studies included in the meta‐analyses, causing considerable changes in overall mean effect sizes and even their direction when Japanese‐language studies are excluded. The observed differences in effect sizes are likely attributable to systematic differences in reported statistical results and associated study characteristics, particularly taxa and ecosystems, between English‐ and Japanese‐language studies. Despite being based on a small sample size, our findings suggest that ignoring non‐English‐language studies may bias outcomes of ecological meta‐analyses, due to systematic differences in study characteristics and effect‐size estimates between English‐ and non‐English languages. We provide a list of actions that meta‐analysts could take in the future to reduce the risk of language bias.     

Inference for binomial probability based on dependent Bernoulli random variables with applications to meta‐analysis and group level studies

We study bias arising as a result of nonlinear transformations of random variables in random or mixed effects models and its effect on inference in group‐level studies or in meta‐analysis. The findings are illustrated on the example of overdispersed binomial distributions, where we demonstrate considerable biases arising from standard log‐odds and arcsine transformations of the estimated probability , both for single‐group studies and in combining results from several groups or studies in meta‐analysis. Our simulations confirm that these biases are linear in ρ, for small values of ρ, the intracluster correlation coefficient. These biases do not depend on the sample sizes or the number of studies K in a meta‐analysis and result in abysmal coverage of the combined effect for large K. We also propose bias‐correction for the arcsine transformation. Our simulations demonstrate that this bias‐correction works well for small values of the intraclass correlation. The methods are applied to two examples of meta‐analyses of prevalence.     

Null models for species co-occurrence patterns: assessing bias and minimum iteration number for the sequential swap

The analysis of co-occurrence matrices is a common practice to evaluate community structure. The observed data are compared with a "null model", a randomised co-occurrence matrix derived from the observation by using a statistic, e.g. the C-score, sensitive to the pattern investigated. The most frequently used algorithm, "sequential swap", has been criticised for not sampling with equal frequencies thereby calling into question the results of earlier analysis. The bias of the "sequential swap" algorithm when used with the C-score was assessed by analysing 291 published presence-absence matrices. In 152 cases, the true p-value differed by >5% from the p-value generated by an uncorrected "sequential swap". However, the absolute value of the difference was rather small. Out of the 291 matrices, there were only 5 cases in which an incorrect statistical decision would have been reached by using the uncorrected p-value (3 at the p<0.05 and 2 at the p<0.01 level), and in all 5 of these cases, the true p-value was close to the significance level. Our results confirm analytical studies of Miklos and Podani which show that the uncorrected swap gives slightly conservative results in tests for competitive segregation. However, the bias is very small and should not distort the ecological interpretation. We also estimated the number of iterations needed for the "sequential swap" to generate accurate p-values. While most authors do not exceed a number of 10⁴ iterations, the suggested minimum number of swaps for 29 out of the 291 tested matrices is greater than 10⁴. We recommend to use 30 000 "sequential swaps" if the required sample size is not assessed otherwise.     

Estimating the Biases Associated with Self-Perceived,Self-Reported,and Measured BMI on Mental Health

Purpose

The purpose of the study is to explore the relationship between individuals'' perceptions of their weight-status, self-reported height and weight, and measured weight status.

Methods

A national survey of 9,248 adolescents (47% male) between the ages of 11 and 27 is analyzed to determine whether inaccuracies in reporting are caused by misperception or conscious intent, and whether there tends to be a systematic bias in how individuals self-report. Self-esteem was used as an example of an important outcome variable in order to illustrate the magnitudes of the biases that may arise when using different measures of body size.

Results

Our results indicate that measured obesity status is associated with the reduction in Rosenberg Self-Esteem (RSE) of 0.30 points (p-value 0.005) among adolescents and 0.20 points (p-value 0.002) among young adults; in addition, using self-reported height and weight as opposed to measured height and weight does not result in a statistically detectable difference in the estimates.

Conclusions

Individuals'' self-reports of height and weight are not as unreliable as we might have expected. Although estimates from measured height and weight are preferred, in the absence of such measures, self-reported measures would likely be a reliable alternative. The differences in self-perception of weight status, however, imply that it is not comparable to measured weight categories.     

Bayesian Meta‐Analysis of the Accuracy of a Test for Tuberculous Pleuritis in the Absence of a Gold Standard Reference

Summary Absence of a perfect reference test is an acknowledged source of bias in diagnostic studies. In the case of tuberculous pleuritis, standard reference tests such as smear microscopy, culture and biopsy have poor sensitivity. Yet meta‐analyses of new tests for this disease have always assumed the reference standard is perfect, leading to biased estimates of the new test’s accuracy. We describe a method for joint meta‐analysis of sensitivity and specificity of the diagnostic test under evaluation, while considering the imperfect nature of the reference standard. We use a Bayesian hierarchical model that takes into account within‐ and between‐study variability. We show how to obtain pooled estimates of sensitivity and specificity, and how to plot a hierarchical summary receiver operating characteristic curve. We describe extensions of the model to situations where multiple reference tests are used, and where index and reference tests are conditionally dependent. The performance of the model is evaluated using simulations and illustrated using data from a meta‐analysis of nucleic acid amplification tests (NAATs) for tuberculous pleuritis. The estimate of NAAT specificity was higher and the sensitivity lower compared to a model that assumed that the reference test was perfect.     

A Global Sensitivity Test for Evaluating Statistical Hypotheses with Nonidentifiable Models

Summary We consider the problem of evaluating a statistical hypothesis when some model characteristics are nonidentifiable from observed data. Such a scenario is common in meta‐analysis for assessing publication bias and in longitudinal studies for evaluating a covariate effect when dropouts are likely to be nonignorable. One possible approach to this problem is to fix a minimal set of sensitivity parameters conditional upon which hypothesized parameters are identifiable. Here, we extend this idea and show how to evaluate the hypothesis of interest using an infimum statistic over the whole support of the sensitivity parameter. We characterize the limiting distribution of the statistic as a process in the sensitivity parameter, which involves a careful theoretical analysis of its behavior under model misspecification. In practice, we suggest a nonparametric bootstrap procedure to implement this infimum test as well as to construct confidence bands for simultaneous pointwise tests across all values of the sensitivity parameter, adjusting for multiple testing. The methodology's practical utility is illustrated in an analysis of a longitudinal psychiatric study.     

Statistical development and evaluation of microarray gene expression data filters.

Filtering is a common practice used to simplify the analysis of microarray data by removing from subsequent consideration probe sets believed to be unexpressed. The m/n filter, which is widely used in the analysis of Affymetrix data, removes all probe sets having fewer than m present calls among a set of n chips. The m/n filter has been widely used without considering its statistical properties. The level and power of the m/n filter are derived. Two alternative filters, the pooled p-value filter and the error-minimizing pooled p-value filter are proposed. The pooled p-value filter combines information from the present-absent p-values into a single summary p-value which is subsequently compared to a selected significance threshold. We show that pooled p-value filter is the uniformly most powerful statistical test under a reasonable beta model and that it exhibits greater power than the m/n filter in all scenarios considered in a simulation study. The error-minimizing pooled p-value filter compares the summary p-value with a threshold determined to minimize a total-error criterion based on a partition of the distribution of all probes' summary p-values. The pooled p-value and error-minimizing pooled p-value filters clearly perform better than the m/n filter in a case-study analysis. The case-study analysis also demonstrates a proposed method for estimating the number of differentially expressed probe sets excluded by filtering and subsequent impact on the final analysis. The filter impact analysis shows that the use of even the best filter may hinder, rather than enhance, the ability to discover interesting probe sets or genes. S-plus and R routines to implement the pooled p-value and error-minimizing pooled p-value filters have been developed and are available from www.stjuderesearch.org/depts/biostats/index.html.     

Formalizing the definition of meta‐analysis in Molecular Ecology

Meta‐analysis, the statistical synthesis of pertinent literature to develop evidence‐based conclusions, is relatively new to the field of molecular ecology, with the first meta‐analysis published in the journal Molecular Ecology in 2003 (Slate & Phua 2003). The goal of this article is to formalize the definition of meta‐analysis for the authors, editors, reviewers and readers of Molecular Ecology by completing a review of the meta‐analyses previously published in this journal. We also provide a brief overview of the many components required for meta‐analysis with a more specific discussion of the issues related to the field of molecular ecology, including the use and statistical considerations of Wright's F_ST and its related analogues as effect sizes in meta‐analysis. We performed a literature review to identify articles published as ‘meta‐analyses’ in Molecular Ecology, which were then evaluated by at least two reviewers. We specifically targeted Molecular Ecology publications because as a flagship journal in this field, meta‐analyses published in Molecular Ecology have the potential to set the standard for meta‐analyses in other journals. We found that while many of these reviewed articles were strong meta‐analyses, others failed to follow standard meta‐analytical techniques. One of these unsatisfactory meta‐analyses was in fact a secondary analysis. Other studies attempted meta‐analyses but lacked the fundamental statistics that are considered necessary for an effective and powerful meta‐analysis. By drawing attention to the inconsistency of studies labelled as meta‐analyses, we emphasize the importance of understanding the components of traditional meta‐analyses to fully embrace the strengths of quantitative data synthesis in the field of molecular ecology.     

Effects of sample size and intraspecific variation in phylogenetic comparative studies: a meta‐analytic review

Comparative analyses aim to explain interspecific variation in phenotype among taxa. In this context, phylogenetic approaches are generally applied to control for similarity due to common descent, because such phylogenetic relationships can produce spurious similarity in phenotypes (known as phylogenetic inertia or bias). On the other hand, these analyses largely ignore potential biases due to within‐species variation. Phylogenetic comparative studies inherently assume that species‐specific means from intraspecific samples of modest sample size are biologically meaningful. However, within‐species variation is often significant, because measurement errors, within‐ and between‐individual variation, seasonal fluctuations, and differences among populations can all reduce the repeatability of a trait. Although simulations revealed that low repeatability can increase the type I error in a phylogenetic study, researchers only exercise great care in accounting for similarity in phenotype due to common phylogenetic descent, while problems posed by intraspecific variation are usually neglected. A meta‐analysis of 194 comparative analyses all adjusting for similarity due to common phylogenetic descent revealed that only a few studies reported intraspecific repeatabilities, and hardly any considered or partially dealt with errors arising from intraspecific variation. This is intriguing, because the meta‐analytic data suggest that the effect of heterogeneous sampling can be as important as phylogenetic bias, and thus they should be equally controlled in comparative studies. We provide recommendations about how to handle such effects of heterogeneous sampling.     

Pvclust: an R package for assessing the uncertainty in hierarchical clustering

SUMMARY: Pvclust is an add-on package for a statistical software R to assess the uncertainty in hierarchical cluster analysis. Pvclust can be used easily for general statistical problems, such as DNA microarray analysis, to perform the bootstrap analysis of clustering, which has been popular in phylogenetic analysis. Pvclust calculates probability values (p-values) for each cluster using bootstrap resampling techniques. Two types of p-values are available: approximately unbiased (AU) p-value and bootstrap probability (BP) value. Multiscale bootstrap resampling is used for the calculation of AU p-value, which has superiority in bias over BP value calculated by the ordinary bootstrap resampling. In addition the computation time can be enormously decreased with parallel computing option.     

A Method of Moments Estimator for Random Effect Multivariate Meta‐Analysis

Summary Meta‐analysis is a powerful approach to combine evidence from multiple studies to make inference about one or more parameters of interest, such as regression coefficients. The validity of the fixed effect model meta‐analysis depends on the underlying assumption that all studies in the meta‐analysis share the same effect size. In the presence of heterogeneity, the fixed effect model incorrectly ignores the between‐study variance and may yield false positive results. The random effect model takes into account both within‐study and between‐study variances. It is more conservative than the fixed effect model and should be favored in the presence of heterogeneity. In this paper, we develop a noniterative method of moments estimator for the between‐study covariance matrix in the random effect model multivariate meta‐analysis. To our knowledge, it is the first such method of moments estimator in the matrix form. We show that our estimator is a multivariate extension of DerSimonian and Laird’s univariate method of moments estimator, and it is invariant to linear transformations. In the simulation study, our method performs well when compared to existing random effect model multivariate meta‐analysis approaches. We also apply our method in the analysis of a real data example.     

Estimating genetic benefits of polyandry from experimental studies: a meta‐analysis

The consequences of polyandry for female fitness are controversial. Sexual conflict studies and a meta‐analysis of mating rates in insects suggest that there is a longevity cost when females mate repeatedly. Even so, compensatory material benefits can elevate egg production and fertility, partly because polyandry ensures an adequate sperm supply. Polyandry can therefore confer direct benefits. The main controversy surrounds genetic benefits. The argument is analogous to that surrounding the evolution of conventional female mate choice, except that with polyandry it is post‐copulatory mechanisms that might bias paternity towards males with higher breeding values for fitness. Recent meta‐analyses of extra‐pair copulations in birds have cast doubt on whether detectable genetic benefits exist. By contrast, another meta‐analysis showed that polyandry elevates egg hatching success (possibly due to a fertilization bias towards sperm with paternal genes that elevate embryo survival) in insects. A detailed summary of whether polyandry elevates other components of offspring performance is lacking. Here we present a comprehensive meta‐analysis of 232 effect sizes from 46 experimental studies. These experiments were specifically designed to try to quantify the potential genetic benefits of polyandry by controlling fully for the number of matings by females assigned to monandry and polyandry treatments. The bias‐corrected 95% confidence intervals for egg hatching success (d = ?0.01 to 0.61), clutch production (d = 0.07 to 0.45) and fertility (d = 0.04 to 0.40) all suggest that polyandry has a beneficial effect (although P values from parametric tests were marginally non‐significant at P = 0.075, 0.052 and 0.058, respectively). Polyandry was not significantly beneficial for any single offspring performance trait (e.g. growth rate, survival, adult size), but the test power was low due to small sample sizes (suggesting that many more studies are still needed). We then calculated a composite effect size that provides an index of general offspring performance. Depending on the model assumptions, the mean effect of polyandry was either significantly positive or marginally non‐significant. A possible role for publication bias is discussed. The magnitude of the reported potential genetic benefits (d = 0.07 to 0.19) are larger than those from two recent meta‐analyses comparing offspring sired by social and extra‐pair mates in birds (d = 0.02 to 0.04). This difference raises the intriguing possibility that cryptic, post‐copulatory female choice might be more likely to generate ‘good gene’ or ‘compatible gene’ benefits than female choice of mates based on the expression of secondary sexual traits.     

Publication bias in ecology and evolution: an empirical assessment using the 'trim and fill' method

Recent reviews of specific topics, such as the relationship between male attractiveness to females and fluctuating asymmetry or attractiveness and the expression of secondary sexual characters, suggest that publication bias might be a problem in ecology and evolution. In these cases, there is a significant negative correlation between the sample size of published studies and the magnitude or strength of the research findings (formally the ‘effect size’). If all studies that are conducted are equally likely to be published, irrespective of their findings, there should not be a directional relationship between effect size and sample size; only a decrease in the variance in effect size as sample size increases due to a reduction in sampling error. One interpretation of these reports of negative correlations is that studies with small sample sizes and weaker findings (smaller effect sizes) are less likely to be published. If the biological literature is systematically biased this could undermine the attempts of reviewers to summarise actual biology relationships by inflating estimates of average effect sizes. But how common is this problem? And does it really effect the general conclusions of literature reviews? Here, we examine data sets of effect sizes extracted from 40 peer‐reviewed, published meta‐analyses. We estimate how many studies are missing using the newly developed ‘trim and fill’ method. This method uses asymmetry in plots of effect size against sample size (‘funnel plots’) to detect ‘missing’ studies. For random‐effect models of meta‐analysis 38% (15/40) of data sets had a significant number of ‘missing’ studies. After correcting for potential publication bias, 21% (8/38) of weighted mean effects were no longer significantly greater than zero, and 15% (5/34) were no longer statistically robust when we used random‐effects models in a weighted meta‐analysis. The mean correlation between sample size and the magnitude of standardised effect size was also significantly negative (r_s=‐0.20, P < 0‐0001). Individual correlations were significantly negative (P < 0.10) in 35% (14/40) of cases. Publication bias may therefore effect the main conclusions of at least 15–21% of meta‐analyses. We suggest that future literature reviews assess the robustness of their main conclusions by correcting for potential publication bias using the ‘trim and fill’ method.     

Meta-analysis of studies with missing data

Summary .  Consider a meta-analysis of studies with varying proportions of patient-level missing data, and assume that each primary study has made certain missing data adjustments so that the reported estimates of treatment effect size and variance are valid. These estimates of treatment effects can be combined across studies by standard meta-analytic methods, employing a random-effects model to account for heterogeneity across studies. However, we note that a meta-analysis based on the standard random-effects model will lead to biased estimates when the attrition rates of primary studies depend on the size of the underlying study-level treatment effect. Perhaps ignorable within each study, these types of missing data are in fact not ignorable in a meta-analysis. We propose three methods to correct the bias resulting from such missing data in a meta-analysis: reweighting the DerSimonian–Laird estimate by the completion rate; incorporating the completion rate into a Bayesian random-effects model; and inference based on a Bayesian shared-parameter model that includes the completion rate. We illustrate these methods through a meta-analysis of 16 published randomized trials that examined combined pharmacotherapy and psychological treatment for depression.     

A note on including time-dependent covariate in regression model for competing risks data

Recently, regression analysis of the cumulative incidence function has gained interest in competing risks data analysis, through the model proposed by Fine and Gray (JASA 1999; 94: 496-509). In this note, we point out that inclusion of time-dependent covariates in this model can lead to serious bias. We illustrate the problems arising in such a context, using bone marrow transplant data as a working example and numerical simulations. Practical advices are given, preventing the misuse of this model.     

Continued misuse of multiple testing correction methods in population genetics‐A wake‐up call?

Population geneticists often use multiple independent hypothesis tests of Hardy–Weinberg Equilibrium (HWE), Linkage Disequilibrium (LD), and population differentiation, to make broad inferences about their systems of choice. However, correcting for Family‐Wise Error Rates (FWER) that are inflated due to multiple comparisons, is sparingly reported in our current literature. In this issue of Molecular Ecology Resources, perform a meta‐analysis of 215 population genetics studies published between 2011 and 2013 to show (i) scarce use of FWER corrections across all three classes of tests, and (ii) when used, inconsistent application of correction methods with a clear bias towards less‐conservative corrections for tests of population differentiation, than for tests of HWE, and LD. Here we replicate this meta‐analysis using 205 population genetics studies published between 2013 and 2018, to show the same continued disuse, and inconsistencies. We hope that both studies serve as a wake‐up call to population geneticists, reviewers, and editors to be rigorous about consistently correcting for FWER inflation.     

Stress and Adiposity: A Meta‐Analysis of Longitudinal Studies

Psychosocial stress has been strongly implicated in the biology of adiposity but epidemiological studies have produced inconsistent results. The aim of this analysis was to bring together results from published, longitudinal, prospective studies examining associations between psychosocial stress and objectively measured adiposity in a meta‐analysis. Searches were conducted on Medline, PsycINFO, Web of Science, and PubMed (to January 2009) and reference lists from relevant articles were examined. Prospective studies relating psychosocial stress (general life stress (including caregiver stress), work stress) to BMI, body fat, body weight, waist circumference, or waist‐to‐hip ratio were included. Analyses from 14 cohorts were collated and evaluated. There was no significant heterogeneity, no evidence of publication bias, and no association between study quality and outcomes. The majority of analyses found no significant relationship between stress and adiposity (69%), but among those with significant effects, more found positive than negative associations (25 vs. 6%). Combining results in a meta‐analysis showed that stress was associated with increasing adiposity (r = 0.014; confidence interval (CI) = 0.002–0.025, P < 0.05). Effects were stronger for men than women, in analyses with longer rather than shorter follow‐ups, and in better quality studies. We conclude that psychosocial stress is a risk factor for weight gain but effects are very small. Variability across studies indicates there are moderating variables to be elucidated.