Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance

The power of language to modify the reader’s perception of interpreting biomedical results cannot be underestimated. Misreporting and misinterpretation are pressing problems in randomized controlled trials (RCT) output. This may be partially related to the statistical significance paradigm used in clinical trials centered around a P value below 0.05 cutoff. Strict use of this P value may lead to strategies of clinical researchers to describe their clinical results with P values approaching but not reaching the threshold to be “almost significant.” The question is how phrases expressing nonsignificant results have been reported in RCTs over the past 30 years. To this end, we conducted a quantitative analysis of English full texts containing 567,758 RCTs recorded in PubMed between 1990 and 2020 (81.5% of all published RCTs in PubMed). We determined the exact presence of 505 predefined phrases denoting results that approach but do not cross the line of formal statistical significance (P < 0.05). We modeled temporal trends in phrase data with Bayesian linear regression. Evidence for temporal change was obtained through Bayes factor (BF) analysis. In a randomly sampled subset, the associated P values were manually extracted. We identified 61,741 phrases in 49,134 RCTs indicating almost significant results (8.65%; 95% confidence interval (CI): 8.58% to 8.73%). The overall prevalence of these phrases remained stable over time, with the most prevalent phrases being “marginally significant” (in 7,735 RCTs), “all but significant” (7,015), “a nonsignificant trend” (3,442), “failed to reach statistical significance” (2,578), and “a strong trend” (1,700). The strongest evidence for an increased temporal prevalence was found for “a numerical trend,” “a positive trend,” “an increasing trend,” and “nominally significant.” In contrast, the phrases “all but significant,” “approaches statistical significance,” “did not quite reach statistical significance,” “difference was apparent,” “failed to reach statistical significance,” and “not quite significant” decreased over time. In a random sampled subset of 29,000 phrases, the manually identified and corresponding 11,926 P values, 68,1% ranged between 0.05 and 0.15 (CI: 67. to 69.0; median 0.06). Our results show that RCT reports regularly contain specific phrases describing marginally nonsignificant results to report P values close to but above the dominant 0.05 cutoff. The fact that the prevalence of the phrases remained stable over time indicates that this practice of broadly interpreting P values close to a predefined threshold remains prevalent. To enhance responsible and transparent interpretation of RCT results, researchers, clinicians, reviewers, and editors may reduce the focus on formal statistical significance thresholds and stimulate reporting of P values with corresponding effect sizes and CIs and focus on the clinical relevance of the statistical difference found in RCTs.

The power of language to modify the reader’s perception of interpreting biomedical results cannot be underestimated. An analysis of more than half a million randomized controlled trials reveals that researchers are using appealing phrases to describe non-significant findings as if they were below the p=0.05 significance threshold.     

An Antidote to the Imager's Fallacy,or How to Identify Brain Areas That Are in Limbo

Traditionally, fMRI data are analyzed using statistical parametric mapping approaches. Regardless of the precise thresholding procedure, these approaches ultimately divide the brain in regions that do or do not differ significantly across experimental conditions. This binary classification scheme fosters the so-called imager''s fallacy, where researchers prematurely conclude that region A is selectively involved in a certain cognitive task because activity in that region reaches statistical significance and activity in region B does not. For such a conclusion to be statistically valid, however, a test on the differences in activation across these two regions is required. Here we propose a simple GLM-based method that defines an “in-between” category of brain regions that are neither significantly active nor inactive, but rather “in limbo”. For regions that are in limbo, the activation pattern is inconclusive: it does not differ significantly from baseline, but neither does it differ significantly from regions that do show significant changes from baseline. This pattern indicates that measurement was insufficiently precise. By directly testing differences in activation, our procedure helps reduce the impact of the imager''s fallacy. The method is illustrated using concrete examples.     

Educating the future generation of researchers: A cross-disciplinary survey of trends in analysis methods

Methods for data analysis in the biomedical, life, and social (BLS) sciences are developing at a rapid pace. At the same time, there is increasing concern that education in quantitative methods is failing to adequately prepare students for contemporary research. These trends have led to calls for educational reform to undergraduate and graduate quantitative research method curricula. We argue that such reform should be based on data-driven insights into within- and cross-disciplinary use of analytic methods. Our survey of peer-reviewed literature analyzed approximately 1.3 million openly available research articles to monitor the cross-disciplinary mentions of analytic methods in the past decade. We applied data-driven text mining analyses to the “Methods” and “Results” sections of a large subset of this corpus to identify trends in analytic method mentions shared across disciplines, as well as those unique to each discipline. We found that the t test, analysis of variance (ANOVA), linear regression, chi-squared test, and other classical statistical methods have been and remain the most mentioned analytic methods in biomedical, life science, and social science research articles. However, mentions of these methods have declined as a percentage of the published literature between 2009 and 2020. On the other hand, multivariate statistical and machine learning approaches, such as artificial neural networks (ANNs), have seen a significant increase in the total share of scientific publications. We also found unique groupings of analytic methods associated with each BLS science discipline, such as the use of structural equation modeling (SEM) in psychology, survival models in oncology, and manifold learning in ecology. We discuss the implications of these findings for education in statistics and research methods, as well as within- and cross-disciplinary collaboration.

A quantitative survey of >1 million published research articles reveals that while classical statistical methods remain in widespread use, multivariate statistical and machine-learning approaches have seen a significant increase; statistics curricula should be revised to take full advantage of these new analytical tools.     

Genome Wide Association Study Identifies New Loci Associated with Undesired Coat Color Phenotypes in Saanen Goats

This paper reports a quantitative genetics and genomic analysis of undesirable coat color patterns in goats. Two undesirable coat colors have routinely been recorded for the past 15 years in French Saanen goats. One fifth of Saanen females have been phenotyped “pink” (8.0%) or “pink neck” (11.5%) and consequently have not been included in the breeding program as elite animals. Heritability of the binary “pink” and “pink neck” phenotype, estimated from 103,443 females was 0.26 for “pink” and 0.21 for “pink neck”. Genome wide association studies (using haplotypes or single SNPs) were implemented using a daughter design of 810 Saanen goats sired by 9 Artificial Insemination bucks genotyped with the goatSNP50 chip. A highly significant signal (-log10pvalue = 10.2) was associated with the “pink neck” phenotype on chromosome 11, suggesting the presence of a major gene. Highly significant signals for the “pink” phenotype were found on chromosomes 5 and 13 (-log10p values of 7.2 and, 7.7 respectively). The most significant SNP on chromosome 13 was in the ASIP gene region, well known for its association with coat color phenotypes. Nine significant signals were also found for both traits. The highest signal for each trait was detected by both single SNP and haplotype approaches, whereas the smaller signals were not consistently detected by the two methods. Altogether these results demonstrated a strong genetic control of the “pink” and “pink neck” phenotypes in French Saanen goats suggesting that SNP information could be used to identify and remove undesired colored animals from the breeding program.     

Statistical Analysis of Individual Participant Data Meta-Analyses: A Comparison of Methods and Recommendations for Practice

Background

Individual participant data (IPD) meta-analyses that obtain “raw” data from studies rather than summary data typically adopt a “two-stage” approach to analysis whereby IPD within trials generate summary measures, which are combined using standard meta-analytical methods. Recently, a range of “one-stage” approaches which combine all individual participant data in a single meta-analysis have been suggested as providing a more powerful and flexible approach. However, they are more complex to implement and require statistical support. This study uses a dataset to compare “two-stage” and “one-stage” models of varying complexity, to ascertain whether results obtained from the approaches differ in a clinically meaningful way.

Methods and Findings

We included data from 24 randomised controlled trials, evaluating antiplatelet agents, for the prevention of pre-eclampsia in pregnancy. We performed two-stage and one-stage IPD meta-analyses to estimate overall treatment effect and to explore potential treatment interactions whereby particular types of women and their babies might benefit differentially from receiving antiplatelets. Two-stage and one-stage approaches gave similar results, showing a benefit of using anti-platelets (Relative risk 0.90, 95% CI 0.84 to 0.97). Neither approach suggested that any particular type of women benefited more or less from antiplatelets. There were no material differences in results between different types of one-stage model.

Conclusions

For these data, two-stage and one-stage approaches to analysis produce similar results. Although one-stage models offer a flexible environment for exploring model structure and are useful where across study patterns relating to types of participant, intervention and outcome mask similar relationships within trials, the additional insights provided by their usage may not outweigh the costs of statistical support for routine application in syntheses of randomised controlled trials. Researchers considering undertaking an IPD meta-analysis should not necessarily be deterred by a perceived need for sophisticated statistical methods when combining information from large randomised trials.     

Open Versus Arthroscopic Surgical Management for Recalcitrant Trochanteric Bursitis: A Systematic Review

BackgroundWhile excision of the trochanteric bursae to treat lateral hip pain has increased in popularity, no comparison exists between the surgical outcomes and complications of the open and arthroscopic techniques involving trochanteric bursectomy. The purpose of this study was to determine the efficacies and complication rates of arthroscopic and open techniques for procedures involving trochanteric bursectomy.MethodsThe terms “trochanteric,” “bursectomy,” “arthroscopic,” “open,” “outcomes,” and “hip” were searched in five electronic databases. Fifteen studies from 120 initial results were included. Patient-reported outcomes (PRO), pain, satisfaction, and complications were included for analysis.ResultsFive hundred-two hips in 474 total patients (77.7% female) were included in this study. The average age was 54. The fourteen distinct PRO scores that were reported by the included studies improved significantly from baseline to final mean follow-up (12-70.8 months for open; 12-42 months for arthroscopic) for both approaches, demonstrating statistically significant patient benefit in a variety of hip arthroscopy settings (P > 0.05). The complication rates of all procedures ranged from 0%-33% and failure to improve pain ranged from 0%-8%. Patient satisfaction with surgery was high at 95% and 82% reported a willingness to undergo the same surgery again. No significant mean differences were found between the open and arthroscopic techniques.ConclusionThe open and arthroscopic approaches for trochanteric bursectomy are both safe and effective procedures in treating refractory lateral hip pain. No significant differences in PROs, pain, total complications, severity of complications, and total failures were seen between technique outcomes.Level of Evidence: IV     

META-GSA: Combining Findings from Gene-Set Analyses across Several Genome-Wide Association Studies

Introduction

Gene-set analysis (GSA) methods are used as complementary approaches to genome-wide association studies (GWASs). The single marker association estimates of a predefined set of genes are either contrasted with those of all remaining genes or with a null non-associated background. To pool the p-values from several GSAs, it is important to take into account the concordance of the observed patterns resulting from single marker association point estimates across any given gene set. Here we propose an enhanced version of Fisher’s inverse χ²-method META-GSA, however weighting each study to account for imperfect correlation between association patterns.

Simulation and Power

We investigated the performance of META-GSA by simulating GWASs with 500 cases and 500 controls at 100 diallelic markers in 20 different scenarios, simulating different relative risks between 1 and 1.5 in gene sets of 10 genes. Wilcoxon’s rank sum test was applied as GSA for each study. We found that META-GSA has greater power to discover truly associated gene sets than simple pooling of the p-values, by e.g. 59% versus 37%, when the true relative risk for 5 of 10 genes was assume to be 1.5. Under the null hypothesis of no difference in the true association pattern between the gene set of interest and the set of remaining genes, the results of both approaches are almost uncorrelated. We recommend not relying on p-values alone when combining the results of independent GSAs.

Application

We applied META-GSA to pool the results of four case-control GWASs of lung cancer risk (Central European Study and Toronto/Lunenfeld-Tanenbaum Research Institute Study; German Lung Cancer Study and MD Anderson Cancer Center Study), which had already been analyzed separately with four different GSA methods (EASE; SLAT, mSUMSTAT and GenGen). This application revealed the pathway GO0015291 “transmembrane transporter activity” as significantly enriched with associated genes (GSA-method: EASE, p = 0.0315 corrected for multiple testing). Similar results were found for GO0015464 “acetylcholine receptor activity” but only when not corrected for multiple testing (all GSA-methods applied; p≈0.02).     

Impact of specific training and competition on myocardial structure and function in different age ranges of male handball players

Handball activity involves cardiac changes and demands a mixture of both eccentric and concentric remodeling within the heart. This study seeks to explore heart performance and cardiac remodeling likely to define cardiac parameters which influence specific performance in male handball players across different age ranges. Forty three players, with a regular training and competitive background in handball separated into three groups aged on average 11.78±0.41 for youth players aka “schools”, “elite juniors” 15.99±0.81 and “elite adults” 24.46±2.63 years, underwent echocardiography and ECG examinations. Incremental ergocycle and specific field (SFT) tests have also been conducted. With age and regular training and competition, myocardial remodeling in different age ranges exhibit significant differences in dilatation’s parameters between “schools” and “juniors” players, such as the end-diastolic diameter (LVEDD) and the end-systolic diameter of the left ventricle (LVESD), the root of aorta (Ao) and left atrial (LA), while significant increase is observed between “juniors” and “adults” players in the interventricular septum (IVS), the posterior wall thicknesses (PWT) and LV mass index. ECG changes are also noted but NS differences were observed in studied parameters. For incremental maximal test, players demonstrate a significant increase in duration and total work between “schools” and “juniors” and, in total work only, between “juniors” and “seniors”. The SFT shows improvement in performance which ranged between 26.17±1.83 sec to 31.23±2.34 sec respectively from “seniors” to “schools”. The cross-sectional approach used to compare groups with prior hypothesis that there would be differences in exercise performance and cardiac parameters depending on duration of prior handball practice, leads to point out the early cardiac remodeling within the heart as adaptive change. Prevalence of cardiac chamber dilation with less hypertrophy remodeling was found from “schools” to “juniors” while a prevalence of cardiac hypertrophy with less pronounced chamber dilation remodeling was noted later.     

Confidence Intervals for fMRI Activation Maps

Neuroimaging activation maps typically color voxels to indicate whether the blood oxygen level-dependent (BOLD) signals measured among two or more experimental conditions differ significantly at that location. This data presentation, however, omits information critical for interpretation of experimental results. First, no information is represented about trends at voxels that do not pass the statistical test. Second, no information is given about the range of probable effect sizes at voxels that do pass the statistical test. This leads to a fundamental error in interpreting activation maps by naïve viewers, where it is assumed that colored, “active” voxels are reliably different from uncolored “inactive” voxels. In other domains, confidence intervals have been added to data graphics to reduce such errors. Here, we first document the prevalence of the fundamental error of interpretation, and then present a method for solving it by depicting confidence intervals in fMRI activation maps. Presenting images where the bounds of confidence intervals at each voxel are coded as color allows readers to visually test for differences between “active” and “inactive” voxels, and permits for more proper interpretation of neuroimaging data. Our specific graphical methods are intended as initial proposals to spur broader discussion of how to present confidence intervals for fMRI data.     

Revertant Fibers in the mdx Murine Model of Duchenne Muscular Dystrophy: An Age- and Muscle-Related Reappraisal

Muscles in Duchenne dystrophy patients are characterized by the absence of dystrophin, yet transverse sections show a small percentage of fibers (termed “revertant fibers”) positive for dystrophin expression. This phenomenon, whose biological bases have not been fully elucidated, is present also in the murine and canine models of DMD and can confound the evaluation of therapeutic approaches. We analyzed 11 different muscles in a cohort of 40 mdx mice, the most commonly model used in pre-clinical studies, belonging to four age groups; such number of animals allowed us to perform solid ANOVA statistical analysis. We assessed the average number of dystrophin-positive fibers, both absolute and normalized for muscle size, and the correlation between their formation and the ageing process. Our results indicate that various muscles develop different numbers of revertant fibers, with different time trends; besides, they suggest that the biological mechanism(s) behind dystrophin re-expression might not be limited to the early development phases but could actually continue during adulthood. Importantly, such finding was seen also in cardiac muscle, a fact that does not fit into the current hypothesis of the clonal origin of “revertant” myonuclei from satellite cells. This work represents the largest, statistically significant analysis of revertant fibers in mdx mice so far, which can now be used as a reference point for improving the evaluation of therapeutic approaches for DMD. At the same time, it provides new clues about the formation of revertant fibers/cardiomyocytes in dystrophic skeletal and cardiac muscle.     

COUMARIN THERAPY—Prothrombin Activity after Termination of Treatment

Twelve patients receiving coumarin type hypoprothrombinemic agents were studied before, during and after termination of therapy, the prothrombin proconvertin method having been used to assay the prothrombin activity complex.In no instance was post treatment “rebound” demonstrated.Prothrombin activity levels returned to pretreatment values only after ten days following termination of coumarin or Dicumarol administration.If a reactivation of thrombotic tendency occurs following discontinuance of anticoagulant therapy, it would not appear to be related to a “rebound” of prothrombin activity above that which is “normal” for the individual patient.Patients tend to return to the same level of prothrombin activity present before initiation of coumarin therapy.     

Genome Wide Association for Addiction: Replicated Results and Comparisons of Two Analytic Approaches

Background

Vulnerabilities to dependence on addictive substances are substantially heritable complex disorders whose underlying genetic architecture is likely to be polygenic, with modest contributions from variants in many individual genes. “Nontemplate” genome wide association (GWA) approaches can identity groups of chromosomal regions and genes that, taken together, are much more likely to contain allelic variants that alter vulnerability to substance dependence than expected by chance.

Methodology/Principal Findings

We report pooled “nontemplate” genome-wide association studies of two independent samples of substance dependent vs control research volunteers (n = 1620), one European-American and the other African-American using 1 million SNP (single nucleotide polymorphism) Affymetrix genotyping arrays. We assess convergence between results from these two samples using two related methods that seek clustering of nominally-positive results and assess significance levels with Monte Carlo and permutation approaches. Both “converge then cluster” and “cluster then converge” analyses document convergence between the results obtained from these two independent datasets in ways that are virtually never found by chance. The genes identified in this fashion are also identified by individually-genotyped dbGAP data that compare allele frequencies in cocaine dependent vs control individuals.

Conclusions/Significance

These overlapping results identify small chromosomal regions that are also identified by genome wide data from studies of other relevant samples to extents much greater than chance. These chromosomal regions contain more genes related to “cell adhesion” processes than expected by chance. They also contain a number of genes that encode potential targets for anti-addiction pharmacotherapeutics. “Nontemplate” GWA approaches that seek chromosomal regions in which nominally-positive associations are found in multiple independent samples are likely to complement classical, “template” GWA approaches in which “genome wide” levels of significance are sought for SNP data from single case vs control comparisons.     

The Brain’s Response to the Human Voice Depends on the Incidence of Autistic Traits in the General Population

Optimal brain sensitivity to the fundamental frequency (F0) contour changes in the human voice is important for understanding a speaker’s intonation, and consequently, the speaker’s attitude. However, whether sensitivity in the brain’s response to a human voice F0 contour change varies with an interaction between an individual’s traits (i.e., autistic traits) and a human voice element (i.e., presence or absence of communicative action such as calling) has not been investigated. In the present study, we investigated the neural processes involved in the perception of F0 contour changes in the Japanese monosyllables “ne” and “nu.” “Ne” is an interjection that means “hi” or “hey” in English; pronunciation of “ne” with a high falling F0 contour is used when the speaker wants to attract a listener’s attention (i.e., social intonation). Meanwhile, the Japanese concrete noun “nu” has no communicative meaning. We applied an adaptive spatial filtering method to the neuromagnetic time course recorded by whole-head magnetoencephalography (MEG) and estimated the spatiotemporal frequency dynamics of event-related cerebral oscillatory changes in beta band during the oddball paradigm. During the perception of the F0 contour change when “ne” was presented, there was event-related de-synchronization (ERD) in the right temporal lobe. In contrast, during the perception of the F0 contour change when “nu” was presented, ERD occurred in the left temporal lobe and in the bilateral occipital lobes. ERD that occurred during the social stimulus “ne” in the right hemisphere was significantly correlated with a greater number of autistic traits measured according to the Autism Spectrum Quotient (AQ), suggesting that the differences in human voice processing are associated with higher autistic traits, even in non-clinical subjects.     

Unusual Non-Occupational Exposure to Metals

Exposure to metals at workplaces is well known and in many cases occupational studies led to an adoption of limit values. For airborne concentrations of substances as metals refer to the “Maximaleo Arbeitsplatz-Konzentration” (MAK) in Germany or the “Threshold Limit Value” (TLV) in USA. Biological monitoring consists of an assessment of overall exposure to chemicals at the workplace and in the environment. The “Biologischer Arbeitsstoff Toleranzwert” (BAT) in Germany and the “Biological Exposure Index” in the USA serve as reference values. Besides these occupational limit values, reference values exist in Germany for the background exposure of the non occupationally exposed general population. In some cases the reference values are exceeded without any occupational exposure. Several cases of unusual environmental exposure to cobalt, mercury and manganese are reported. In such cases, it is often difficult to evaluate the measured concentration. In Germany, therefore, the “Human-Biomonitoring-Werte” (HBMValues) have been adopted in order to evaluate such high background exposures. The HBM-concept is presented. Environmental exposure to metals is usual within some limits. Reference values are helpful for an assessment. Unusual exposure occurs and the physician should be alert to symptoms of poisoning.     

Body composition adaptations to lower-body plyometric training: a systematic review and meta-analysis

The aim of this meta-analysis was to explore the effects of plyometric jump training (PJT) on body composition parameters among males. Relevant articles were searched in the electronic databases PubMed, MEDLINE, WOS, and SCOPUS, using the key words “ballistic”, “complex”, “explosive”, “force-velocity”, “plyometric”, “stretch-shortening cycle”, “jump”, “training”, and “body composition”. We included randomized controlled trials (RCTs) that investigating the effects of PJT in healthy male’s body composition (e.g., muscle mass; body fat), irrespective of age. From database searching 21 RCTs were included (separate experimental groups = 28; pooled number of participants = 594). Compared to control, PJT produced significant increases in total leg muscle volume (small ES = 0.55, p = 0.009), thigh muscle volume (small ES = 0.38, p = 0.043), thigh girth (large ES = 1.78, p = 0.011), calf girth (large ES = 1.89, p = 0.022), and muscle pennation angle (small ES = 0.53, p = 0.040). However, we did not find significant difference between PJT and control for muscle cross-sectional area, body fat, and skinfold thickness. Heterogeneity remained low-to-moderate for most analyses, and using the Egger’s test publication bias was not found in any of the analyses (p = 0.300–0.900). No injuries were reported among the included studies. PJT seems to be an effective and safe mode of exercise for increasing leg muscle volume, thigh muscle volume, thigh and calf girth, and muscle pennation angle. Therefore, PJT may be effective to improve muscle size and architecture, with potential implications in several clinical and sport-related contexts.     

Gene Order Phylogeny and the Evolution of Methanogens

Methanogens are a phylogenetically diverse group belonging to Euryarchaeota. Previously, phylogenetic approaches using large datasets revealed that methanogens can be grouped into two classes, “Class I” and “Class II”. However, some deep relationships were not resolved. For instance, the monophyly of “Class I” methanogens, which consist of Methanopyrales, Methanobacteriales and Methanococcales, is disputable due to weak statistical support. In this study, we use MSOAR to identify common orthologous genes from eight methanogen species and a Thermococcale species (outgroup), and apply GRAPPA and FastME to compute distance-based gene order phylogeny. The gene order phylogeny supports two classes of methanogens, but it differs from the original classification of methanogens by placing Methanopyrales and Methanobacteriales together with Methanosarcinales in Class II rather than with Methanococcales. This study suggests a new classification scheme for methanogens. In addition, it indicates that gene order phylogeny can complement traditional sequence-based methods in addressing taxonomic questions for deep relationships.     

Selecting Essential Information for Biosurveillance—A Multi-Criteria Decision Analysis

The National Strategy for Biosurveillancedefines biosurveillance as “the process of gathering, integrating, interpreting, and communicating essential information related to all-hazards threats or disease activity affecting human, animal, or plant health to achieve early detection and warning, contribute to overall situational awareness of the health aspects of an incident, and to enable better decision-making at all levels.” However, the strategy does not specify how “essential information” is to be identified and integrated into the current biosurveillance enterprise, or what the metrics qualify information as being “essential”. Thequestion of data stream identification and selection requires a structured methodology that can systematically evaluate the tradeoffs between the many criteria that need to be taken in account. Multi-Attribute Utility Theory, a type of multi-criteria decision analysis, can provide a well-defined, structured approach that can offer solutions to this problem. While the use of Multi-Attribute Utility Theoryas a practical method to apply formal scientific decision theoretical approaches to complex, multi-criteria problems has been demonstrated in a variety of fields, this method has never been applied to decision support in biosurveillance.We have developed a formalized decision support analytic framework that can facilitate identification of “essential information” for use in biosurveillance systems or processes and we offer this framework to the global BSV community as a tool for optimizing the BSV enterprise. To demonstrate utility, we applied the framework to the problem of evaluating data streams for use in an integrated global infectious disease surveillance system.