Systematic review of the empirical evidence of study publication bias and outcome reporting bias

Background

The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.

Methodology/Principal Findings

We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.

Conclusions

Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.     

Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias

Meta-analyses play an important role in synthesizing evidence from diverse studies and datasets that address similar questions. A major obstacle for meta-analyses arises from biases in reporting. In particular, it is speculated that findings which do not achieve formal statistical significance are less likely reported than statistically significant findings. Moreover, the patterns of bias can be complex and may also depend on the timing of the research results and their relationship with previously published work. In this paper, we present an approach that is specifically designed to analyze large-scale datasets on published results. Such datasets are currently emerging in diverse research fields, particularly in molecular medicine. We use our approach to investigate a dataset on Alzheimer's disease (AD) that covers 1167 results from case-control studies on 102 genetic markers. We observe that initial studies on a genetic marker tend to be substantially more biased than subsequent replications. The chances for initial, statistically non-significant results to be published are estimated to be about 44% (95% CI, 32% to 63%) relative to statistically significant results, while statistically non-significant replications have almost the same chance to be published as statistically significant replications (84%; 95% CI, 66% to 107%). Early replications tend to be biased against initial findings, an observation previously termed Proteus phenomenon: The chances for non-significant studies going in the same direction as the initial result are estimated to be lower than the chances for non-significant studies opposing the initial result (73%; 95% CI, 55% to 96%). Such dynamic patterns in bias are difficult to capture by conventional methods, where typically simple publication bias is assumed to operate. Our approach captures and corrects for complex dynamic patterns of bias, and thereby helps generating conclusions from published results that are more robust against the presence of different coexisting types of selective reporting.     

A bound for publication bias based on the fraction of unpublished studies

Publication bias in meta-analysis is usually modeled in terms of an accept/reject selection procedure in which the selected studies are the "published" studies and the rejected studies are the "unpublished" studies. One possible selection mechanism is to suppose that only studies that report an estimated treatment effect exceeding (or falling short of) some threshold are accepted. We show that, with appropriate choice of thresholds, this attains the maximum bias among all selection mechanisms in which the probability of selection increases with study size. It is impossible to estimate the selection mechanism from the observed studies alone: this result leads to a "worst-case" sensitivity analysis for publication bias, which is remarkably easy to implement in practice. The method is illustrated using data on the effectiveness of prophylactic corticosteroids.     

Weekend bias in Citizen Science data reporting: implications for phenology studies

Studies of bird phenology can help elucidate the effects of climate change on wildlife species but observations over broad spatial scales are difficult without a network of observers. Recently, networks of citizen volunteers have begun to report first arrival dates for many migratory species. Potential benefits are substantial (e.g., understanding ecological processes at broad spatial and temporal scales) if known biases of citizen data reporting are identified and addressed. One potential source of bias in bird phenology studies is the tendency for more “first” migratory arrivals to be reported on weekends than on weekdays. We investigated weekend bias in data reporting for five common bird species in North America (Baltimore Oriole, Icterus galbula; Barn Swallow, Hirundo rustica; Chimney Swift, Chaetura pelagica; Purple Martin, Progne subis; and Ruby-throated Hummingbird, Archilochus colubris), and assessed whether this bias affected mean arrival dates reported using data from historical (1880–1969; N?=?25,555) and recent (1997–2010; N?=?63,149) Citizen Science databases. We found a greater percentage of first arrivals reported on weekends and small but significant differences in mean arrival dates (approximately 0.5 days) for four of five species. Comparing time periods, this weekend bias decreased from 33.7 % and five species in the historical time period to 32 % and three species in the recent, perhaps related to changes in human activity patterns. Our results indicate that weekend bias in citizen data reporting is decreasing over time in North America and including a ‘day of week’ term in models examining changes in phenology could help make conclusions more robust.     

Impact of reporting bias in network meta-analysis of antidepressant placebo-controlled trials

Background

Indirect comparisons of competing treatments by network meta-analysis (NMA) are increasingly in use. Reporting bias has received little attention in this context. We aimed to assess the impact of such bias in NMAs.

Methods

We used data from 74 FDA-registered placebo-controlled trials of 12 antidepressants and their 51 matching publications. For each dataset, NMA was used to estimate the effect sizes for 66 possible pair-wise comparisons of these drugs, the probabilities of being the best drug and ranking the drugs. To assess the impact of reporting bias, we compared the NMA results for the 51 published trials and those for the 74 FDA-registered trials. To assess how reporting bias affecting only one drug may affect the ranking of all drugs, we performed 12 different NMAs for hypothetical analysis. For each of these NMAs, we used published data for one drug and FDA data for the 11 other drugs.

Findings

Pair-wise effect sizes for drugs derived from the NMA of published data and those from the NMA of FDA data differed in absolute value by at least 100% in 30 of 66 pair-wise comparisons (45%). Depending on the dataset used, the top 3 agents differed, in composition and order. When reporting bias hypothetically affected only one drug, the affected drug ranked first in 5 of the 12 NMAs but second (n = 2), fourth (n = 1) or eighth (n = 2) in the NMA of the complete FDA network.

Conclusions

In this particular network, reporting bias biased NMA-based estimates of treatments efficacy and modified ranking. The reporting bias effect in NMAs may differ from that in classical meta-analyses in that reporting bias affecting only one drug may affect the ranking of all drugs.     

Experimental investigation of the bound dissipation function

By means of a microcalorimeter (direct calorimetry) and a Warburg-apparatus (indirect calorimetry) that part of the dissipation of a growing culture of yeast cells which remains irreversible in the cells is determined (Ψ _u ). The course of the Ψ _u -function with time correlates with the increase of the specific cell concentration being conditioned by the growth phase of the culture but similar for fermentative and respirative metabolism.     

Bacteriophages may bias outcome of bacterial enrichment cultures

Enrichment cultures are widely used for the isolation of bacteria in clinical, biotechnological, and environmental studies. However, competition, relative growth rates, or inhibitory effects may alter the outcome of enrichment cultures, causing the phenomenon known as enrichment bias. Bacteriophages are a major component in many microbial systems, and it abounds in natural settings. This abundance means that bacteriophages are likely to be present in many laboratory enrichment cultures. Our hypothesis was that bacteriophages present in the sample might bias the enriched subpopulation, since it can infect and lyse the target bacteria during the enrichment step once the bacteria reach a given density. Here we show that the presence of bacteriophages in Salmonella and Shigella enrichment cultures produced a significant reduction (more than 1 log unit) in the number of these bacteria compared with samples in which bacteriophages had been reduced by filtration through 0.45-microm non-protein-binding membranes. Furthermore, our data indicate that the Salmonella biotypes isolated after the enrichment culture change if bacteriophages are present, thus distorting the results of the analysis.     

Understanding reporting bias in the dietary recall data of 11-year-old girls

Objective: This study describes patterns of bias in self‐reported dietary recall data of girls by examining differences among girls classified as under‐reporters, plausible reporters, and over‐reporters on weight, dietary patterns, and psychosocial characteristics. Research Methods and Procedures: Participants included 176 girls at age 11 and their parents. Girls’ weight and height were measured. Three 24‐hour dietary recalls and responses to psychosocial measures were collected. Plausibility cut‐offs for reported energy intake as a percentage of predicted energy requirements were used to divide the sample into under‐reporters, plausible reporters, and over‐reporters. Differences among these three groups on dietary and psychosocial variables were assessed to examine possible sources of bias in reporting. Results: Using a ±1 standard deviation cut‐off for energy intake plausibility, 50% of the sample was categorized as plausible reporters, 34% as under‐reporters, and 16% as over‐reporters. Weight status of under‐reporters was significantly higher than that of plausible reporters and over‐reporters. With respect to reported dietary intake, under‐reporters were no different from plausible reporters on intakes of foods with higher nutrient densities and lower energy densities and were significantly lower than plausible reporters on intakes of foods with lower nutrient densities and higher energy densities. Over‐reporters reported significantly higher intakes of all food groups and the majority of subgroups, relative to plausible reporters. Under‐reporters had significantly higher levels of weight concern and dietary restraint than both plausible reporters and over‐reporters. Discussion: Techniques to categorize plausible and implausible reporters can and should be used to provide an improved understanding of the nature of error in children's dietary intake data and account for this error in analysis and interpretation.     

Application of polylysine bound succinyl-NA to a membrane reactor

The amide bond at N-6 in succinyl-NAD was found to be more stable than in former accounts. Succinyl-NAD was coupled on polylysine to give a new polymer derivative of NAD, which retained at least 85% of the initial coenzymic activity even after dialysis for one week. The polymer derivative of NAD could be applied to a membrane reactor containing alcohol dehydrogenase and lactate dehydrogenase and lactate was continuously produced in a half-life of ten days.     

Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research

Background

The reporting of outcomes within published randomized trials has previously been shown to be incomplete, biased and inconsistent with study protocols. We sought to determine whether outcome reporting bias would be present in a cohort of government-funded trials subjected to rigorous peer review.

Methods

We compared protocols for randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998 with subsequent reports of the trials identified in journal publications. Characteristics of reported and unreported outcomes were recorded from the protocols and publications. Incompletely reported outcomes were defined as those with insufficient data provided in publications for inclusion in meta-analyses. An overall odds ratio measuring the association between completeness of reporting and statistical significance was calculated stratified by trial. Finally, primary outcomes specified in trial protocols were compared with those reported in publications.

Results

We identified 48 trials with 68 publications and 1402 outcomes. The median number of participants per trial was 299, and 44% of the trials were published in general medical journals. A median of 31% (10th–90th percentile range 5%–67%) of outcomes measured to assess the efficacy of an intervention (efficacy outcomes) and 59% (0%–100%) of those measured to assess the harm of an intervention (harm outcomes) per trial were incompletely reported. Statistically significant efficacy outcomes had a higher odds than nonsignificant efficacy outcomes of being fully reported (odds ratio 2.7; 95% confidence interval 1.5–5.0). Primary outcomes differed between protocols and publications for 40% of the trials.

Interpretation

Selective reporting of outcomes frequently occurs in publications of high-quality government-funded trials.Selective reporting of results from randomized trials can occur either at the level of end points within published studies (outcome reporting bias)¹ or at the level of entire trials that are selectively published (study publication bias).² Outcome reporting bias has previously been demonstrated in a broad cohort of published trials approved by a regional ethics committee.¹ The Canadian Institutes of Health Research (CIHR) — the primary federal funding agency, known before 2000 as the Medical Research Council of Canada (MRC) — recognized the need to address this issue and conducted an internal review process in 2002 to evaluate the reporting of results from its funded trials. The primary objectives were to determine (a) the prevalence of incomplete outcome reporting in journal publications of randomized trials; (b) the degree of association between adequate outcome reporting and statistical significance; and (c) the consistency between primary outcomes specified in trial protocols and those specified in subsequent journal publications.     

Battle of the sexes: analysis of sex bias in host use and reporting practices in parasitological experiments

Experimental approaches are among the most powerful tools available to biologists, yet in many disciplines their results have been questioned due to an underrepresentation of female animal subjects. In parasitology, experiments are crucial to understand host-parasite interactions, parasite development, host immune responses, as well as the efficacy of different control methods. However, distinguishing between species-wide and sex-specific effects requires the balanced inclusion of both male and female hosts in experiments and the reporting of results for each sex separately. Here, using data from over 3600 parasitological experiments on helminth-mammal interactions published in the past four decades, we investigate patterns of male versus female subject use and result reporting practices in experimental parasitology. We uncover multiple effects of the parasite taxon used, the type of host used (rats and mice for which subject selection is fully under researcher control versus farm animals), the research subject area and the year of publication, on whether host sex is even specified, whether one or both host sexes have been used (and if only one then which one), and whether the results are presented separately for each host sex. We discuss possible reasons for biases and unjustifiable selection of host subjects, and for poor experimental design and reporting of results. Finally, we make some simple recommendations for increased rigour in experimental design and to reset experimental approaches as a cornerstone of parasitological research.     

Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy

Objective

We assessed the adequacy of randomized controlled trial (RCT) registration, changes to registration data and reporting completeness for articles in ICMJE journals during 2.5 years after registration requirement policy.

Methods

For a set of 149 reports of 152 RCTs with ClinicalTrials.gov registration number, published from September 2005 to April 2008, we evaluated the completeness of 9 items from WHO 20-item Minimum Data Set relevant for assessing trial quality. We also assessed changes to the registration elements at the Archive site of ClinicalTrials.gov and compared published and registry data.

Results

RCTs were mostly registered before 13 September 2005 deadline (n = 101, 66.4%); 118 (77.6%) started recruitment before and 31 (20.4%) after registration. At the time of registration, 152 RCTs had a total of 224 missing registry fields, most commonly ‘Key secondary outcomes’ (44.1% RCTs) and ‘Primary outcome’ (38.8%). More RCTs with post-registration recruitment had missing Minimum Data Set items than RCTs with pre-registration recruitment: 57/118 (48.3%) vs. 24/31 (77.4%) (χ² ₁ = 7.255, P = 0.007). Major changes in the data entries were found for 31 (25.2%) RCTs. The number of RCTs with differences between registered and published data ranged from 21 (13.8%) for Study type to 118 (77.6%) for Target sample size.

Conclusions

ICMJE journals published RCTs with proper registration but the registration data were often not adequate, underwent substantial changes in the registry over time and differed in registered and published data. Editors need to establish quality control procedures in the journals so that they continue to contribute to the increased transparency of clinical trials.     

Assessing the bias linked to DNA recovery from biofiltration woodchips for microbial community investigation by fingerprinting

In this study, we explored methodological aspects of nucleic acid recovery from microbial communities involved in a gas biofilter filled with pine bark woodchips. DNA was recovered indirectly in two steps, comparing different methods: cell dispersion (crushing, shaking, and sonication) and DNA extraction (three commercial kits and a laboratory protocol). The objectives were (a) to optimize cell desorption from the packing material and (b) to compare the 12 combinations of desorption and extraction methods, according to three relevant criteria: DNA yield, DNA purity, and community structure representation by denaturing gradient gel electrophoresis (DGGE). Cell dispersion was not influenced by the operational parameters tested for shaking and blending, while it increased with time for sonication. DNA extraction by the laboratory protocol provided the highest DNA yields, whereas the best DNA purity was obtained by a commercial kit designed for DNA extraction from soil. After successful PCR amplification, the 12 methods did not generate the same bias in microbial community representation. Eight combinations led to high diversity estimation, independently of the experimental procedure. Among them, six provided highly similar DGGE profiles. Two protocols generated a significantly dissimilar community profile, with less diversity. This study highlighted the crucial importance of DNA recovery bias evaluation.     

Nuclear magnetic resonance investigation of the conformation of delta-haemolysin bound to dodecylphosphocholine micelles

delta-Haemolysin in mixed micelles with perdeuterated dodecylphosphocholine was investigated with two-dimensional proton nuclear magnetic resonance experiments at 500 MHz. A single set of resonance lines was observed for the micelle-bound polypeptide, indicating that delta-haemolysin adopts a single conformation in this environment. Nearly complete, sequence-specific assignments were obtained for the segment 5-23 of this 26-residue polypeptide chain. From the sequential connectivities and numerous medium-range nuclear Overhauser effects this central portion of the molecule was found to form an extended helix with pronounced amphipathic distribution of polar and nonpolar amino acid side-chains.